FlickerFusion: Intra-trajectory Domain Generalizing Multi-agent Reinforcement Learning
Domain Generalization;Multi-agent Reinforcement Learning;Benchmark
Yes
No ethics review needed.
See Weaknesses.

And, for this dynamic entity number problem (in MPEv2), is it possible to build a mapping that maps the dimension-varying vector observation to images (the ones you used to render the demos), so that we can simply and directly utilize a CNN to handle the issue?
- The idea of dropping out observation of some sampled entities at each step and fuse these flickers to emulate a full-view across the temporal dimension is inspiring.
- Authors extended the widely-used MARL benchmark MPE, which can be beneficial for relevant future research.
- FlickerFusion is compared with 11 baselines, providing sufficient and solid results.
- Demo videos clearly show how FlickerFusion works.
This paper addresses the MARL problem that entities like teammates or adversaries are dynamically removed or added during the inference trajectory. The key idea is to dropout observation of some sampled entities at each step, denoted as flickers, and fuse these flickers to emulate a full-view across the temporal dimension. Numerical results and demos on MPEv2 demonstrate FlickerFusion’s promising performance.
- “Dropout” is the core idea and concept of this paper, but it only appears at the end of Introduction and has not been clearly defined. It would be better if authors define and emphasize it in Abstract of the beginning of Introduction, and distinguish it with the dropout operation in the neural network literature.
- Preliminaries is a bit too complex, it would be better to simplify it and relate it to the previous text, including some practical examples for illustration.
- The importance of each entity is different, but the flicker operation treats all entities equally by sampling uniformly. A module that evaluates entities’ intention / importance / uncertainty, using techniques like opponent modeling[1,2], should be developed to better guide the sampling of the flicker operation.
- The environment dynamics would change dramatically when the number of entities change. It is not clear how FlickerFusion handles this problem and realizes generalization. A clear study on this problem is vital for the application on more complex scenarios. 
- The learning curves should demonstrate standard deviation.
- More related work about varying agent numbers[3] and intra-trajectory teammate shift[4] should be discussed.

[1] Georgios Papoudakis, et al. Agent Modelling under Partial Observability for Deep Reinforcement Learning.

[2] Kiarash Kazari, et al. Decentralized Anomaly Detection in Cooperative Multi-Agent Reinforcement Learning.

[3] Lei Yuan, et al. Multi-agent Continual Coordination via Progressive Task Contextualization.

[4] Ziqian Zhang, et al. Fast Teammate Adaptation in the Presence of Sudden Policy Change.
Yes
No ethics review needed.
- In MPE, instead of randomly omitting entities, wouldn’t it make more sense to omit those farther from the ego agent based on their distance?
- How does omitting observations differ from the practice of dropping out nodes in standard neural network training?
- This paper tackles an important generalization capability in MARL.
- FlickerFusion is straightforward and can be easily integrated into various MARL frameworks.
This paper introduces FlickerFusion, a method designed to improve how multi-agent reinforcement learning (MARL) models generalize to environments with a dynamically changing number of agents. The approach involves randomly omitting agents’ observations during training, which helps the model perform better in new, dynamically changing environments in a zero-shot manner. The authors also present a modified version of Multi Particle Environments that supports variable numbers of agents to test their method.
- MARL is generally challenging to stabilize during training. By altering observations at each timestep, this method should increase instability in principle, yet the paper does not explain how FlickerFusion manages to achieve less variability in results compared to the baseline QMIX, as shown in Table 1 and Figure 5.
- The evaluation relies solely on the Multi Particle Environments (MPE), which has a relatively simple observational setup. It remains uncertain if FlickerFusion would be effective in more complex environments such as SMAC.
- Figures 3 and 5 in the paper would be more informative if they included quantiles instead of only the means.
Yes
No ethics review needed.
1. How would the proposed method operate in situations where there are different types of entities with varying lengths and/or compositions of observation space?
2. In Figure 5, I suggest that the authors use a different line type for the proposed method to clearly distinguish FLICKERFUSION from the baselines.
1. The issue of zero-shot out-of-domain (OOD) generalization in MARL is significant in bridging the gap between laboratory MARL research and real-world deployment.
2. The concept of FLICKERFUSION, which involves dropping out entities and recovering lost information, is novel and inspiring.
3. Although the paper uses only one series of benchmarks, MPEV2, it includes several different scenarios and baselines to demonstrate the robust performance of the proposed method.
4. The presentation is clear, and the details are effectively conveyed in the text and the Appendix.
This paper addresses multi-agent reinforcement learning (MARL) with a dynamic number of entities. Unlike existing methods that introduce extra parameters, the proposed novel method, FLICKERFUSION, functions as an augmentation technique that selectively drops entities to ensure the observation space aligns with the policy network, regardless of changes in the number of entities. The experiments and ablation studies demonstrate the superior performance of the proposed method and clarify its operational mechanisms.
1. The conceptualization of the paper appears to be excessive. For example, "intra-trajectory" is discussed in detail in the introduction section but is not mentioned in the methods section, which may confuse the audience. Additionally, some key points and implementations of the proposed method are not conveyed in the introduction. For instance, the technique of dropout is missing from both the abstract and the introduction.
2. The paper seems to overlook situations where there are different types of entities with varying lengths and/or compositions of observation space. In such cases, the stochastic dropout approach may not be effective.
Yes
No ethics review needed.
1.How does FLICKERFUSION perform in environments with extremely high or low densities of entity changes? Is the dropout strategy adaptable enough to handle these scenarios effectively?

2.Could this method be extended to other benchmarks, such as SMAC, MAgent, or CityFlow?

Ref

[1] Yuan L, Zhang Z, Li L, et al. A survey of progress on cooperative multi-agent reinforcement learning in open environment[J]. arXiv preprint arXiv:2312.01058, 2023.
1.The paper addresses a critical yet under-explored problem in MARL, specifically the intra-trajectory dynamic entity composition and its impact on generalization. The proposed approach, which uses entity dropout and stochastic fusion, is original and impactful.

2.The introduction of the MPEV2 benchmark enhances the paper's contribution, allowing a rigorous assessment of the proposed method. The results across various environments are convincing and demonstrate clear performance improvements.

3.The code and benchmarks are made publicly available, supporting reproducibility and further research in this domain.

4.This paper includes many baselines to  validate its efficiency.
The paper introduces FLICKERFUSION,  a novel method aimed at enhancing out-of-domain (OOD) generalization in Multi-Agent Reinforcement Learning (MARL) environments where the composition of entities changes dynamically within a trajectory. The authors identify two significant challenges: intra-trajectory dynamic entity addition and zero-shot OOD generalization, which existing methods fail to handle effectively. FLICKERFUSION utilizes stochastic input-space dropout to maintain observation space consistency without additional parameters, achieving superior performance across a newly developed benchmark suite, MPEV2. Empirical results show that FLICKERFUSION outperforms existing approaches in both inference reward and uncertainty reduction.
1.This paper addresses some open problems in MARL. However, it lacks discussion on several related areas, such as multi-agent policy transfer [1] and sudden policy changes [2]. Including a more comprehensive analysis of these works could strengthen the paper.

2.The definition of OOD is unclear and too simple.

3.The use of the 'Flicker Fusion Phenomena' in this study is intriguing. However, the random masking approach might result in the loss of team semantic information. Alternative methods, such as learning to group[3] or applying the widely used mean-field theory in MARL[4], could potentially offer more advantages. Could the authors elaborate on the strengths and weaknesses of their approach compared to these methods?

4.Could this paper be further enhanced by incorporating Large Language Models (LLMs) like ChatGPT[5]?

Ref

[1] Qin R, Chen F, Wang T, et al. Multi-Agent Policy Transfer via Task Relationship Modeling[J]. arXiv e-prints, 2022: arXiv: 2203.04482.

[2] Zhang Z, Yuan L, Li L, et al. Fast teammate adaptation in the presence of sudden policy change[C]//Uncertainty in Artificial Intelligence. PMLR, 2023: 2465-2476.

[3] Sheng, Junjie, et al. "Learning structured communication for multi-agent reinforcement learning." Autonomous Agents and Multi-Agent Systems 36.2 (2022): 50.

[4] Yang Y, Luo R, Li M, et al. Mean field multi-agent reinforcement learning[C]//International conference on machine learning. PMLR, 2018: 5571-5580.

[5] Sun C, Huang S, Pompili D. LLM-based Multi-Agent Reinforcement Learning: Current and Future Directions[J]. arXiv preprint arXiv:2405.11106, 2024.
We present a novel, simple yet effective method for domain generalization in multi-agent reinforcement learning
@inproceedings{
anonymous2024flickerfusion,
title={FlickerFusion: Intra-trajectory Domain Generalizing Multi-agent Reinforcement Learning},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=MRYyOaNxh3},
note={under review}
}
Multi-agent reinforcement learning has demonstrated significant potential in addressing complex cooperative tasks across various real-world applications. However, existing MARL approaches often rely on the restrictive assumption that the number of entities (e.g., agents, obstacles) remains constant between training and inference. This overlooks scenarios where entities are dynamically added or removed $\textit{during}$ the inference trajectory---a common occurrence in real-world environments like search and rescue missions and dynamic combat situations. In this paper, we tackle the challenge of $\textbf{intra-trajectory dynamic entity composition}$ under zero-shot $\textbf{out-of-domain (OOD) generalization}$, where such dynamic changes cannot be anticipated beforehand. Our empirical studies reveal that existing MARL methods suffer $\textit{significant}$ performance degradation and increased uncertainty in these scenarios. In response, we propose FlickerFusion, a novel OOD generalization method that acts as a $\textit{universally}$ applicable augmentation technique for MARL backbone methods. Our results show that FlickerFusion not only achieves superior inference rewards but also $\textit{uniquely}$ reduces uncertainty vis-à-vis the backbone, compared to existing methods. For standardized evaluation, we introduce MPEv2, an enhanced version of Multi Particle Environments (MPE), consisting of 12 benchmarks. Benchmarks, implementations, and trained models are organized and open-sourced at $\texttt{\href{flickerfusion305.github.io}{flickerfusion305.github.io}}$, accompanied by ample demo video renderings.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Domain Generalization Multi-agent Reinforcement Learning Benchmark
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/6d0c9f5e231739f8d97d6fae8814638c6c1989c2.pdf
reinforcement learning
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
FlickerFusion: Intra-trajectory Domain Generalizing Multi-agent Reinforcement Learning
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission