Last Iterate Convergence in Monotone Mean Field Games
mean field game;learning in games
Yes
No ethics review needed.
Q1. In Assumption 2.1, please clarify why “It is reasonable to assume” it. For example, it is violated in many examples of the MFG literature in grid world models, where the agents cannot jump from a given state to any other given state (for any action). I think this would be true only if there is a very strong noise that can propel an agent to any other state in just one time step. It does not seem very realistic from the physical viewpoint. Please explain.

Q2. In Definition 4.2, are regularized equilibria unique? Otherwise, please explain how you can say “the regularized equilibrium” in Theorems 4.3 and 4.4. 

Q3. Algorithm 2 and Theorem 4.3: it seems to rely very strongly on the RMD of Zhang et al. (2023). How does the proof of convergence of Theorem 4.3 differs from theirs?

Q4. Theorems 4.3 and 4.4: Could you please confirm whether these results hold for any $\lambda$ and $\sigma$, simply under Assumption 4.1? I just want to make sure there is no requirement to take $\lambda$ large enough as e.g., in  Cui & Koeppl (2021).

Q5. Theorems 4.3 and 4.4: Here it seems that the convergence only holds for the regularized MFG. How far is it from the true MFG equilibrium?

Q6. Could you please explain whether the example considered in the numerical part satisfies all the assumptions required for your convergence results? Also, is it not strictly monotone (so that it does not fall in the case covered by Perolat et al. (2022))?

Q7. To really see how your method compares with existing one, please compare your algorithm with other baselines, such as the algorithms considered in Perolat et al. (2022) (in particular the mirror-descent-type one, which has last-iterate convergence, as you mentioned).

Typos:
Definition 2.4: “is mean-field” -> “is a mean-field”
Page 3: “has the full support” -> “has full support”
The paper is relatively clear and addresses an interesting question (last-iterate convergence). It seems to have a good theoretical foundation.
This paper proposes and algorithm for mean field games (MFGs) based on proximal-point-type iterations. After proving convergence of the last iterate for an ideal algorithm, the paper considers an approximate algorithm based on mirror descent updates. For regularized MFGs, a proof of convergence is provided, with a rate of convergence. A simple illustrative example is shown at the end.
The first result holds under general assumptions but the algorithm cannot be implemented. The second result holds only for regularized MFGs, which can be quite different from the original (unregularized) MFG problem and there is not much discussion about this. As a consequence, the contributions seem relatively limited.
Yes
No ethics review needed.
Please see details in weakness.
The paper is well structured and organized.
This paper studies the convergence performance in Monotone Mean Field Games. The major issue is that the focus of this paper are all well studied. Authors cannot identify the research gap and contribution in a comparison to existing study.
1. Position: Last-iterate convergence and learning regularized MFGs have been extensively studied. However, the discussion of related work in this manuscript is somewhat superficial. The authors should conduct a thorough literature review and clearly position their work within the existing literature. This will help avoid giving the audience the impression that their contribution is marginal, merely by applying a proximal point iteration to MFGs.

2. Exponential convergence of mirror descent: Using mirror descent to learn regularized mean field games (MFGs) is not a novel approach. To the best of my knowledge, previous studies have established a standard convergence rate of $T^{-1/2}$. In contrast, this work claims an exponential convergence rate of $e^{-T}$, which represents a significant improvement. The authors should clearly explain the specific challenges they addressed to achieve this enhanced convergence rate that previous work did not tackle. Additionally, providing numerical demonstrations of such exponential convergence is essential, given the magnitude of this improvement.

3. Presentation: the presentation of the paper is not yet suitable for scientific publication. Please avoid excessive use of colors, shapes, and unconventional layouts. Figure 1 lacks informative content. Additionally, some language and structural transitions are too casual and do not provide sufficient discussion to support the arguments presented convincingly.

4. The numerical experiments are limited. More results regarding the algorithm performance in a comparison to baselines (OMD, FP, Multi-scale, double loop, single loop) for MFGs should be provided.
Yes
No ethics review needed.
Can you elaborate on the technical contributions of this work? Specifically, why wouldn't a straightforward application of the analysis proposed in [1] work in regularized monotone mean field games?
The paper is well-written and the proof is clear. The result is the first last-iterate convergence in regularized MFG to my best knowledge.
The paper studied regularized MFG and proposed a mirror descent algorithm that can converge in $O(\log(1/\epsilon))$ iterations.
My main concern is that, with the monotonicity condition and the regularization $\lambda$, the results can be straightforwardly obtained given the existing proof of exponentially fast convergence of MD in monotone games. See [1] for details. 

Specially, the relationship between $D_{\mu^\ast} (\pi^\ast, \pi^{t+1})$ and $D_{\mu^\ast} (\pi^\ast, \pi^{t})$ has already been derived in Zhang et al 2023. To obtain the exponentially fast convergence rate, you would only need to get the $1-\lambda \eta$ factor out of $D_{\mu^\ast} (\pi^\ast, \pi^{t})$. One common way to achieve this is to use the monotonicity condition. From what I understand, this paper uses the same technique that has been used in papers such as [1]. 


[1] Cen, Shicong, Yuting Wei, and Yuejie Chi. "Fast policy extragradient methods for competitive games with entropy regularization." Advances in Neural Information Processing Systems 34 (2021): 27952-27964.
We establish a novel algorithm with guaranteed last-iterate convergence in monotone MFG.
@inproceedings{
anonymous2024last,
title={Last Iterate Convergence in Monotone Mean Field Games},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=cUnqwFu5OO},
note={under review}
}
Mean Field Game (MFG) is a framework utilized to model and approximate the behavior of a large number of agents, and the computation of equilibria in MFG has been a subject of interest. Despite the proposal of methods to approximate the equilibria, algorithms that can achieve equilibrium with the most recent policy of the algorithm, namely the last-iterate policy, have been limited. 
We propose the use of a simple, proximal-point-type algorithm to compute strategies for MFGs. Subsequently, we provide the first last-iterate convergence guarantee under the Lasry--Lions-type monotonicity condition. 
We further employ the Mirror Descent algorithm for the regularized MFG to efficiently approximate the update rules of the proximal point method for MFGs. 
We demonstrate that the last-iterate strategy of Mirror Descent converges exponentially fast: we provide the guarantee of computing the $\varepsilon$ approximation in $O(\log(1/\varepsilon))$ iterations. This research offers a tractable approach for large-scale and large-population games.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
mean field game learning in games
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/8c27f069f61d01ba3b80231be45836b0fc9bd66f.pdf
optimization
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Last Iterate Convergence in Monotone Mean Field Games
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission