DoF: A Diffusion Factorization Framework for Offline Multi-Agent Decision Making
multi-agent reinforcement learning; Diffusion Models;   Offline reinforcement learning
Yes
No ethics review needed.
Can the authors provide formal statements and proofs showing that IGD generalizes IGM, individual global optimal principle or risk-sensitive IGM principles?

Are there any nontrivial examples that satisfy IGD beyond the Gaussian case (or a component-wise independent Markov chain with a stationary initial distribution)?
The empirical results show their method perform slightly better than other methods.
This paper proposes an individual-global-identically distributed (IGD) principle (property) for model design in cooperative multi-agent decision-making.  The paper claims that the property generalizes the popular individual-global-max principle (IGM).  They further propose a diffusion factorization framework that 1) satisfies the IGD principle and 2) performs well on several datasets.
I am unfamiliar with this domain of literature, and my comment would mostly focus on mathematical writing.  Overall, I have strong concerns about the claim that IGD generalizes IGM, and I feel the theory about diffusion factorization framework under Gaussian assumption satisfying IGD uninteresting.  Thus, I suggest rejection.

The technical writing quality needs significant improvement. Here are some comments.
1. Does IGD property apply to the initial distribution $p_{\theta_{tot}}(x_{tot}^0)$  or the whole series $p_{\theta_{tot}}(x_{tot}^{0:K})$? Does $x_{tot}^K$ need to be sampled from Gaussian?  Does the reverse process need to be a Markov chain?
2.  The claim that IGD generalizes IGM is not well-defined.  IGM needs a value function but does not necessarily have a distribution over action or even under a diffusion model.  Please give an explicit statement on the generalization: For instance, given a value function $Q_{jt}$ satisfies IGM what is the corresponding $p_{\theta_{tot}}$ such that Eqn.(1) holds for any history $\tau_{tot}$.
3.  At the end of section 4.2, there should be formal statements of IGD vs individual global optimal principle or risk-sensitive IGM principles.

If I understand correctly, Theorem 1 basically states that each component of isotropic Gaussian is independent.  I feel this result is very limited and hardly holds in general.  For instance, it’s unclear whether this would still apply even when the Gaussian has a non-diagonal covariance.



Minor comments
- In eq (3), instead of $\theta_i\subset \theta_{tot}$, $(\theta_1,\dots, \theta_K) = \theta_{tot}$ would be better, as the order of coordinate matters.  
- $\epsilon^{\theta_1}_1$ in eq (5) is not clearly defined.
Yes
No ethics review needed.
1.	Given that QMIX is an online algorithm not designed for offline reinforcement learning, could you clarify the rationale for including it in the comparisons? What insights does this comparison provide?
2.	Can diffusion-based methods effectively handle out-of-distribution situations? Additionally, can these methods be applied to competitive or mixed-motivation scenarios?

Please respond to the aforementioned weaknesses.
1.	The paper generalizes the well-known IGM principle to the IGD principle, which broadens the applicability of diffusion models in multi-agent decision-making and improves coordination among agents. It addresses the limitations of existing cooperative MADM methods, enhancing scalability and cooperative performance.
2.	DoF introduces novel noise and data factorization methods that align agent-generated data with a centralized diffusion model while enabling decentralized execution. This factorization process addresses the complexity of multi-agent coordination and enhances resource efficiency across a range of multi-agent tasks.
3.	The extensive experiments conducted on the SMAC and MPE show that DoF outperforms competing methods in multi-agent coordination tasks. Experimental results indicate that DoF performs well with an increasing number of agents. This scalability suggests DoF’s potential for large-scale applications in complex multi-agent environments.
The paper presents Diffusion Factorization Framework (DoF), a new method for offline cooperative multi-agent decision-making (MADM) using diffusion models. DoF generalizes the traditional Individual-Global-Max (IGM) principle with a new Individual-Global-identically-Distributed (IGD) principle. This IGD principle aligns decentralized outcomes in multi-agent diffusion models with centralized outcomes, fostering effective cooperation. DoF includes two key functions: a noise factorization function that decentralizes a centralized diffusion model per the IGD principle, and a data factorization function that models inter-agent data relationships to ensure alignment with ground truth. Extensive experiments demonstrate DoF’s advantages in scalability, cooperation, and alignment with ground truth, outperforming existing MADM methods across various benchmarks.
1.	The paper lacks clarity in some important parts. It introduces DoF-Policy and DoF-Trajectory but does not specify the suitable scenarios for each. The results section also fails to indicate which algorithm was used, and Figure 6 only includes DoF-Policy results, making it difficult to assess the relative strengths of the two methods. Additionally, implementation details for the inverse dynamics model in DoF-Trajectory are missing.
2.	In Figure 2(a), there is a mislabeling of the reverse diffusion process: the output should be  x^{k-1} instead of x^k , which may mislead the understanding of the process. Additionally, the caption of Figure 2 lacks necessary descriptions, which reduces its readability.
3.	The paper's benchmarks are based on state vectors, making it unclear whether the proposed method can be applied to environments with high-dimensional observations. The Scalability Experiment section only compares GPU memory and time with MADIFF, without providing comparisons to other baseline algorithms. Since diffusion-based methods require multiple iterations to generate the final output, does this lead to significantly higher computational costs compared to traditional methods?
Yes
No ethics review needed.
- Table 6 compares decentralized methods (WConcat) with 'centralized' methods (attention), showing similar performance. To properly interpret these results:
  - How is the centralized attention method implemented during execution?
  - What observations/information does centralized attention have access to versus decentralized methods? Is centralized attention meant to serve as an upper-bound baseline (with access to information not available in real deployments) rather than a competing method?
- The authors claim in section 4.3.2 that 'the data factorization h could model more powerful data relationships than the noise factorization function f'. However, Table 14 shows minimal performance improvement when using attention (a more complex function) versus WConcat for data factorization h. Are there theoretical or empirical evidence that supports the claim that a more complex h can compensate for using simple noise factorization functions f (as is mainly used in the paper)?
- The paper provides a novel definition of the IGD principle to generalize the IGM for the use of diffusion models.
- The authors theoretically show that the proposed DoF added with particular factorization functions satisfy the IGD principle.
- Strong empirical section showing the effectiveness of the proposed method
The paper studies the use of diffusion models in offline cooperative multi-agent decision-making (MADM). The authors propose the Individual-Global-identically-Distribute (IGD) principle, a generalization of the IGM principle. IGD requires that collectively generated outcome of individual agents follows the same distribution as the generated outcome of a whole multi-agent system. The authors propose DoF, a diffusion factorization framework for offline MADM which adopts a noise factorization function to factorize a centralized diffusion model into multiple diffusion models. The authors show theoretically that the noise factorization functions in DoF satisfy the IGD principle. Experiments are used to shown the effectiveness of DoF.
Potential limitations of noise and data factorization functions - the primary options used in the paper are Concat and Wconcat, which might lack modeling power.
A multi-agent diffusion model that can be factored into multiple small diffusion models
@inproceedings{
anonymous2024dof,
title={DoF: A Diffusion Factorization Framework for Offline Multi-Agent Decision Making},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=OTFKVkxSlL},
note={under review}
}
Diffusion models have been widely adopted in image and language generation and are now being applied to decision-making. However, the application of diffusion models in offline cooperative Multi-Agent decision making (MADM) remains limited. Although some researches exist, they suffer from scalability or poor cooperation issues due to the lack of design principles for diffusion-based MADM. The Individual-Global-Max (IGM) principle is a popular design principle for cooperative MADM. Through satisfying such principles, MADM algorithms achieve remarkable performance with good scalability. In this work, we extend the IGM principle as the Individual-Global-identically-Distributed (IGD) principle. This principle stipulates that the generated outcome of a multi-agent diffusion model should be identically distributed as the collective outcomes from multiple individual-agent diffusion models. We propose DoF, a diffusion factorization framework for MADM. It uses noise factorization function to factorize a centralized diffusion model into multiple diffusion models. We theoretically show that the noise factorization functions satisfy the IGD principle. Further, DoF uses data factorization function to model the complex relationship among data generated by multiple diffusion models. Through extensive experiments, we demonstrate the effectiveness of DoF.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
multi-agent reinforcement learning; Diffusion Models;   Offline reinforcement learning
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/d27bd68eed7b4e55ca7ffb26b51166888eb335b5.pdf
reinforcement learning
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/c49387621bf2fb740c7b97513192df5e6e51d0a5.zip
DoF: A Diffusion Factorization Framework for Offline Multi-Agent Decision Making
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission