Mean Field Multi-Agent Reinforcement Learning
Yaodong Yang * 1 Rui Luo * 1 Minne Li 1 Ming Zhou 2 Weinan Zhang 2 Jun Wang 1

arXiv:1802.05438v1 [cs.MA] 15 Feb 2018

Abstract
Existing multi-agent reinforcement learning methods are limited typically to a small number of
agents. When the agent number increases largely,
the learning becomes intractable due to the curse
of the dimensionality and the exponential growth
of user interactions. In this paper, we present
Mean Field Reinforcement Learning where the
interactions within the population of agents are
approximated by those between a single agent
and the average effect from the overall population or neighboring agents; the interplay between
the two entities is mutually reinforced: the learning of the individual agent’s optimal policy depends on the dynamics of the population, while
the dynamics of the population change according
to the collective patterns of the individual policies.
We develop practical mean field Q-learning and
mean field Actor-Critic algorithms and analyze
the convergence of the solution. Experiments on
resource allocation, Ising model estimation, and
battle game tasks verify the learning effectiveness
of our mean field approaches in handling manyagent interactions in population.

1. Introduction
Multi-agent reinforcement learning (MARL) is concerned
with a set of autonomous agents that share a common environment (Busoniu & De Schutter). Learning in MARL is
fundamentally difficult since agents not only interact with
the environment but also with each other. Independent Qlearning (Tan, 1993) that considers other agents as a part
of the environment quite often fails as the multi-agent environment breaks the theoretical convergence guarantees and
makes the learning unstable: the changes in strategy of one
agent would affect the strategies of other agents and vice
versa (Matignon et al., 2012).
Instead, accounting for the extra information from conjecturing the policies of other agents would be beneficial to
*
Equal contribution 1 University College London, London,
United Kingdom 2 Shanghai Jiao Tong University, Shanghai, China.
Correspondence to: Jun Wang <j.wang@cs.ucl.ac.uk>.

each single learner (Foerster et al., 2017; Lowe et al., 2017a).
Studies have shown that an agent who learns the effect of
joint actions has better performance than those who does
not, in a number of scenarios including cooperative games
(Panait & Luke, 2005), zero-sum stochastic games (Littman,
1994), and general-sum stochastic games (Littman, 2001;
Hu & Wellman, 2003).
The existing equilibrium based approaches, although principled, are only capable of solving a handful of agents (Hu
& Wellman, 2003; Bowling & Veloso, 2002). The computational complexity of directly solving (Nash) equilibrium
would prevent them from applying for the situations there
are a large body of agents or even a population of agents.
Yet, in practice, many cases do require strategic interactions
among a large number of agents, ranging from gaming bots
in MMORPG (Massively Multiplayer Online Role-Playing
Game) (Jeong et al., 2015) to trading agents in stock markets (Troy, 1997), and to online advertising bidding (Wang
et al., 2017).
In this paper, we tackle MARL when a large number of
agents co-exist. We consider a setting where each agent is
directly interacting with a finite set of other agents; through
a chain of direct interactions, any pair of agents is interconnected globally (Blume, 1993). The scalability is solved by
employing Mean Field Theory (Stanley, 1971): the interactions within the population of agents are approximated by
that of a single agent played with the average effect from the
overall (local) population (Lasry & Lions, 2007). The learning is mutually reinforced between two entities rather than
among many entities: the learning of the individual agent’s
optimal policy is based on the dynamics of the agent population, while at the same time, the dynamics of the population
is updated according to the individual policies. Based on
the formulation, we develop practical mean field Q-learning
and mean field Actor-Critic algorithms, and discuss the convergence of our solution under certain assumptions. Our experiment on a simple multi-agent resource allocation shows
that our mean field MARL is capable of learning over manyagent interactions when others fail. We also demonstrate
that with temporal-difference learning, mean field MARL
manages to learn and solve an Ising model without even
explicitly knowing the energy function. At last, in a mixed
cooperative-competitive battle game, we show that the mean
field MARL achieves high winning rates against other baselines previously reported for many agent systems.

Mean Field Multi-Agent Reinforcement Learning

2. Preliminary

2.2. Nash Q-learning

MARL intersects between reinforcement learning and game
theory. The marriage of the two gives rise to the general
framework of Stochastic Game (van der Wal, 1980).

In MARL, the goal is to learn an optimal strategy for each
agent. In stochastic games, the notion of Nash equilibrium
is thus of great importance (Hu & Wellman, 2003); it is
represented as a set of N policies π∗ = {π∗1, . . . , π∗N } such
that ∀π j ∈ Ω(A j ), it satisfies

2.1. Stochastic Game
An N-agent (or, player) stochastic game Γ is expressed
by a tuple hS, A1, . . . , A N , r 1, . . . , r N , p, γi, where S denotes the state space and A j is the action space of agent
j ( j = 1, . . . , N). At each time step, all agents choose
action simultaneously, which leads to their rewards. r j :
S × A1 × · · · × A N → R is the reward function for agent
j. γ denotes the reward discount factor across the time.
p : S × A1 × · · · × A N → Ω(S) is the transition probability map of the environment, with Ω(S) indicating the
collection of probability distributions over the state space.
Each stage of a stochastic game, depending on the state of
the current game is in, is called a stage game.
The policy (or, strategy) for agent j is π j : S → Ω(A j )
where Ω(A j ) is the collection of probability distribution
over agent j’s action space A j . π = {π 1, . . . , π N } denotes
the joint policy of all N agents. π is usually assumed independent of time, also called stationary strategy. Given initial
state s and certain policy π j , agent j aims to maximize the
value function, namely:
vπj (s) = v j (s; π) =

∞
X
t=0



γ t Eπ, p rtj |s0 = s, π .

(1)

We can define the Q-function in the framework of N-agent
game based on the value function. The Bellman equation
for Q-function of agent j under the joint policy π is Qπj :
S × A1 × · · · × A N → R such that
Qπj (s, a) = r j (s, a) + γEs0 ∼p [vπj (s0 )].

(2)

And the corresponding value function vπj for agent j under
the joint policy π can be expressed in terms of Q-function


vπj (s) = Ea∼π Qπj (s, a) .

(3)

Note that the Q-function for N-agent game extends the conventional formulation for single-agent game by including
actions taken by all N agents a = {a1, . . . , a N }, and by
taking the expectation over the actions of all agents.
We model the MARL problem by the stochastic game in a
discrete-time and non-cooperative setting (no explicit coalitions are considered). We assume the game has incomplete
but perfect information (Littman, 1994), i.e., agents do not
know the game dynamics or the reward functions of others,
but they can observe other agents’ immediate reward and
actions taken previously.

v j (s; π∗ ) = v j (s; π∗j , π∗−j ) ≥ v j (s; π j , π∗−j ),

(4)

where we adopt compact notations for the joint policy of all
players except j as π∗−j = {π∗1, . . . , π∗j−1, π∗j+1, . . . , π∗N }.

In a Nash equilibrium, each agent acts as the best response
π∗j to others, provided that all other agents follow the policy
π∗−j . It has been shown, for every N-agent stochastic game,
there is at least one Nash equilibrium in stationary strategies
(Fink et al., 1964). Given a set of Nash strategy π∗ , the
Nash value function v Nash (s) = (vπ1∗ (s), . . . , vπN∗ (s)) stands
as the expected sum of discounted rewards if all agents
follow these Nash policies from state s onward.

Nash Q-learning (Hu & Wellman, 2003) is an iterative procedure of calculating the Nash policy: 1) solving a Nash
equilibrium through the Lemke-Howson algorithm (Lemke
& Howson, 1964) for the current stage game {Q tj }, 2) improving the estimation of the Nash Q-function using the
new Nash policy. Nash Q keeps iterating the above two
steps until the Q-value converges. It is shown that under certain assumptions, the Nash Q operator HtNash Q =
(HtNash Q1, . . . , HtNash Q N ) defined as


(HtNash Q)(s, a) ≡ Es0 ∼p r(s, a) + γvtNash (s0 )
(5)
is a contraction mapping given that π is an appropriate Nash
equilibrium solver for stage game (Q1 (s0 ), . . . , Q N (s0 )) (s0
is the state at time t + 1); that is, the Q-function will eventually converge to a Nash equilibrium value of the entire
game.

3. Mean Field MARL
As the number of agents increases, the joint action space a
becomes high dimensional. All agents act strategically and
simultaneously learn their values over others actions, which
make the learning of the standard Q function Q j (s, a) in the
previous section infeasible. To handle this, we parametrize
the Q-function by only the pairwise local interactions:
Q j (s, a) ≡

1 X j
Q (s, a j , ak ),
Nj
j

(6)

k∈K

where K j is the set of neighbor agents for agent j. Its size
N j = #K j grows proportionally to the number of agents,
and the ratio depends on applications. It is worth noting that
the pairwise approximation, while significantly reducing the
complexity of the interactions among agents, still preserves
global interactions between any pair of agents implicitly

Mean Field Multi-Agent Reinforcement Learning

(Blume, 1993). Similar approaches in machine learning can
be found such as in factorization machine (Rendle, 2012)
and learning to rank (Cao et al., 2007).

As shown in Eq. (10), we see that under the mean field
approximation, the MARL problem is converted into the
computation of a single agent’s best response with respect
to the action distribution of other agents.

3.1. Mean Field Approximation

To discriminate from the Nash value function v Nash (s), we
refer the mean field value function defined in Eq. (10) as

We consider discrete action space, whereas a continuous
version can be analogically derived. Specially, action a j
is a discrete categorical variable, which is represented as a
one-hot encoding with each component indicating one of
j
the M possible actions: a j ≡ (a1j , . . . , a M
). The pairwise
j
j k
interaction Q (s, a , a ) can be approximated by the mean
field theory (Stanley, 1971). Conveniently the one-hot action of each agent can be represented as a small fluctuation
δak from the mean action within the neighborhood:
ak = ā + δak ,

where ā =

1 X k
a .
Nj
j

(7)

k∈K

Note that the mean action ā ≡ (ā1, . . . , ā M ) is equivalent
to the multinomial distribution of actions from neighboring
agents. With the first-order Taylor expansion, the pairwise
Q-function is approximated as
1 X j
Q (s, a j , ak )
Nj
k∈K j


1 X
≈ j
Q j (s, a j , ā) + ∇ā Q j (s, a j , ā)δak
N
k∈K j
1 X k
= Q j (s, a j , ā) + ∇ā Q j (s, a j , ā) j
δa
N
j

Q j (s, a) =

k∈K

= Q j (s, a j , ā).

(8)

Note that in the second line of Eq. (8) we utilize the firstorder approximation of Q j , where the accuracy improves
when the number of neighbours grows large, since the mean
value of the higher order terms approach zero. The multipleagent interactions have been converted into those of twoagent interactions: a single agent and the mean agent from
the neighbors.
The update rule for the mean field Q-function is defined as
j
Q t+1
(s, a j , ā)

h
i
= (1 − αt )Q tj (s, a j , ā) + αt rtj + γvtj (s0 )

(9)

where αt denotes the learning rate and the mean field value
function is as
X
etj (s, a j ),
vtj (s) =
π j (a j |s)Q
(10)
aj

in which



1 X k
j
j
j
j
e
Q t (s, a ) = Ea− j ∼π− j Q t (s, a , ā = j
a ) . (11)
N
k6= j

v MF (s) = (vπ1 (s), . . . , vπN (s)).
Based on the mean field value function, we finally define the
mean field Q operator as HtMF Q = (HtMF Q1, . . . , HtMF Q N ),
and


(HtMF Q)(s, a) ≡ Es0 ∼p r(s, a) + γvtMF (s0 ) .
(12)
In fact, we can prove that HMF is a contraction mapping; that
is, by updating the Q-function via the mean field operator
HMF , the mean field Q-learning will eventually converge to
the Nash equilibrium under certain assumptions.
3.2. Implementation
In this section we describe the implementation logic of our
proposed algorithm. We parameterize the Q-function and
policy π by φ and θ respectively, and update parameter φ according to Eq.(9). Once Q-function is updated, we improve
the current policy π. For on-policy reinforcement learning,
we use Boltzmann strategy and actor-critic method (Konda
& Tsitsiklis, 2000) to improve policy, presenting MF-Q
(Algorithm 1 in Appendix A) and MF-AC (Algorithm 2 in
Appendix A) respectively.
For agent i, MF-Q can be trained by minimizing the loss
function
L(φ) = (y − Q φj (s, a j , ā))2,

(13)

where y = r + γ Q φj (s0, (a j )0, ā0 ) is the target. Differentiating L(φ) w.r.t the weights produces the following gradient,

∇φ L(φ) = y − Q φj (s, a j , ā) ∇φ Q φj (s, a j , ā).
(14)

The Critic of MF-AC follows the same update rule as above.
The Actor can be trained by using the sampled policy gradient:
∇θ J(θ) ≈ ∇θ log πθj (s)Q φ (s, a j , ā)|a j =π j (s),
θ

(15)

Note that we require the policies of other agents to apply
an update in Eq. 14. For off-policy learning, we can use
DPG (Silver et al., 2014) for deterministic environment or
standard Q-learning (Watkins & Dayan, 1992) for a discrete
case. We iteratively execute parameter update for φ and π
until the Q-function converges.
3.3. Proof of Convergence
We prove the convergence of Q t = (Q1t , . . . , Q tN ) under the
MF-Q iterations to Nash equilibrium Q ∗ = (Q1∗, . . . , Q∗N )

Mean Field Multi-Agent Reinforcement Learning

for learning agents. The proof is conducted through showing
that under the main assumptions below, the mean field Q
operator HMF of Eq. (12) is in fact a contraction mapping,
and its fixed point is Q ∗ .
We start by introducing the main assumptions:
Assumption 1. During training, each action-value pair
will be visited infinitely often, and the reward is bounded by
some constant K.
Assumption 2. Agent’s policy is Greedy in the Limit with
Infinite Exploration (GLIE). In the case of Boltzmann softmax policy, the temperature decays to the zero so that the
policy becomes greedy on the Q-function in the limit.
We make further assumptions on the structure of the game:
Assumption 3. For every stage game (Q1t (s), ..., Q tN (s))
encountered during training, for all t, s and j ∈ {1, . . . , N},
only one of the two scenarios could happen: the Nash
equilibrium π = (π 1, . . . , π N ) is either (I) a global
optimal point,
i.e. Eπ [Q tj (s)] ≥ Eπ̂ [Q tj (s)], ∀π̂ ∈
Q k
j
Ω
k A , or, (II) a saddle point, i.e. Eπ j Eπ − j [Q t (s)] ≥
Eπ̂ j Eπ− j [Q tj (s)], ∀π̂ j ∈ Ω A j ; Eπ j Eπ− j [Q tj (s)] ≤

Q
k
Eπ j Eπ̂− j [Q tj (s)], ∀π̂ −j ∈ Ω
k6= j A .
Note that Assumption 3 asks for a strong requirement on
every individual stage game encountered during training. In
practice, however, we find this is not necessary to hold in
order for the learning algorithm to converge. This is in line
with the empirical findings in Hu & Wellman (2003).

Our proof depends on the following two Lemmas:
Lemma 1. Under the Assumption 3, the Nash Q operator
defined in Eq. (5) is a rigorous contraction mapping on the
complete metric space from Q to Q, its fixed point is the
Nash Q-value of the entire game, i.e., HtNash Q ∗ = Q ∗ .

In Lemma 2, by subtracting Q ∗ (s, a) at both sides in Eq. (9)
and comparing with Eq. (16), we can find in fact
∆t (x) = Q t (s, a) − Q ∗ (s, a),

Ft (x) = rt + γvtMF (st+1 ) − Q∗ (st , at ),

where x = {st , at }. α(t) can be regarded as the learning
rate, and agent will only update the Q-function with the corresponding state st and actions a visited, i.e., αt (s, a) = 0
for (s, a) 6= (st , at ). Note that Lemma 2 suggests the convergence of ∆t (x) to zero, that means, if it holds, the current
Q-function will asymptotically approximate the optimal Q ∗ .
One last piece before the main theorem is the below:
Proposition 1. P
Let the metric space be R N , and the metric
be M(a, b) = j |a j − b j |, ∀ a = {a j }1N , b = {b j }1N . If
the Q function is Lipschitz continuous w.r.t. the input of a j ,
then for sufficiently small temperature β, the operator

exp − βQ(s, a j , ā)
j

B(a ) = P
exp − βQ(s, ak , ā)
a k ∈A j

is a contraction mapping. The proof is in Appendix due to
the space limit.
Theorem 1. In finite-state stochastic game, the Q t values
computed by the MF-Q rule in Eq. (9) converges to the Nash
Q-value Q ∗ = (Q1∗, . . . , Q∗N ), if the Assumptions 1, & 2, 3,
and condition 1 & 2 of Lemma 2 are met.
Proof. We denote the σ-field generated by the random variables in the experience of the stochastic game
{st , αt , at , rt−1, ..., s1, α1, a1, Q 0 } by Ft . Given the fact that
vt , ..., v0 are Ft -measurable, thus both ∆t and Ft−1 are Ft measurable, which satisfies the measurability condition of
Lemma 2.

Proof. See Theorem 17 in Hu & Wellman (2003).

We further need to prove HMF operator satisfies conditions 3
& 4 to employ Lemma 2. We start from condition 3.

Lemma 2. The random process {∆t } in R defined as

Ft (st , at ) = rt + γvtMF (st+1 ) − Q ∗ (st , at )

∆t+1 (x) = (1 − αt (x))∆t (x) + αt (x)Ft (x),

(16)

which converges to zero with probability 1 (w.p.1) under the
following assumptions:
P
P
1. 0 ≤ αt (x) ≤ 1, t αt (x) = ∞, t αt2 (x) < ∞

2. x ∈ X , the set of possible states is finite.

3. kE[Ft (x)|Ft ]kW ≤ γk∆t kW + ct , where γ ∈ [0, 1), ct
converges to zero w.p.1.
2
4. var[Ft (x)|Ft ] ≤ K(1 + k∆t kW
), for constant K > 0.

Here Ft is the filtration of an increasing sequences of σfields that include the history of the process. We assume that
αt , ∆t , Ft ∈ Ft , and k · kW is a weighted maximum norm.
Proof. This Lemma is proved by Jaakkola et al. (1994)
(Theorem 1) and Szepesvári & Littman (1999) (Corollary
5). We include it here to stay self-contained.

= rt + γvtNash (st+1 ) − Q ∗ (st , at )

+ γ[vtMF (st+1 ) − vtNash (st+1 )]


= rt + γvtNash (st+1 ) − Q ∗ (st , at ) + Ct (st , at )
= FtNash (st , at ) + Ct (st , at )

(17)

In fact, we find FtNash (st , at ) is the corresponding Ft in
Lemma 2 as if the convergent proof was about the Nash Q
learning algorithm. Based on Lemma 1, it is easy to know
that FtNash (·, ·) is a contraction mapping where the norm k·k
is the maximum norm over a, thus we have, for all t,
kE[FtNash (·, ·)|Ft ]k∞ ≤ γkQ t − Q ∗ k∞ = γk∆t k∞ .

In meeting the condition 3 of Lemma 2 now we have

kE[Ft (st , at )|Ft ]k ≤ kFtNash (st , at )|Ft k + kCt (st , at )|Ft k
≤ γk∆t k + kCt (st , at )|Ft k.
(18)

Mean Field Multi-Agent Reinforcement Learning

We are left to prove ct = kCt (st , at )|Ft k converges to zero
w.p.1. In fact, under Assumption 3, for each stage game,
all the global optimal equilibrium point(s) share the same
equilibria value, so does any saddle-point equilibrium.
Case I: In each global equilibrium, according to the definition, all players get the same unique maximum value.
Case II: Assume the stage game {Q tj }1N has two saddle-point
equilibrium: π and ρ, then for agent j, we have
Eπ j Eπ− j [Q tj (s)] ≥ Eρ j Eπ− j [Q tj (s)],

Eρ j Eρ− j [Q tj (s)] ≤ Eρ j Eπ− j [Q tj (s)].

Combing them we have
Eπ j Eπ− j [Q tj (s)] ≥ Eρ j Eρ− j [Q tj (s)]
By the definition of saddle point, reversing the order of π
and ρ, the formula still holds; thus the equilibria to agent
i under both saddle-point equilibrium must be the same:
Eπ j Eπ− j [Q tj (s)] = Eρ j Eρ− j [Q tj (s)].
Given the Proposition 1 that the policy of mean field Qfunction is a contraction mapping, and that all global/saddle
points share the same Nash value in each stage game, v MF
value will therefore converge to v Nash value asymptotically.
Condition 3 is thus satisfied.
For condition 4, based on the condition 3 just proved, HMF
is a contraction mapping, i.e., HMF Q∗ = Q ∗ , thus we have:
var[Ft (st , at )|Ft ] = E[(rt + γvtMF (st+1 ) − Q ∗ (st , at ))2 ]
= E[(rt + γvtMF (st+1 ) − HMF (Q ∗ ))2 ]

= var[rt + γvtMF (st+1 )|Ft ]
2
≤ K(1 + k∆t kW
)

(19)

In the last step, we apply the Assumption 1 that the reward
is always bounded by some constant. Finally, by meeting
both condition 3 & 4, according to Lemma 2, ∆t converges
to zero w.p.1, i.e., Q t converges to Q ∗ w.p.1.
Apart from convergence, we also discuss the Rationality
(Bowling & Veloso, 2001; 2002) of the MF-Q algorithm,
due to the space limit, we leave it in the Appendix.

4. Related Work
We continue our discussion on related work started in Section 1 and make comparisons with existing techniques in a
greater scope. Our work follows the same direction as from
(Littman, 1994; Hu & Wellman, 2003; Bowling & Veloso,
2002) in adapting a Stochastic Game (van der Wal, 1980)
in the MARL formulation. Specifically, Littman (1994)
addressed two-player zero-sum stochastic games by introducing a “minimax” operator in Q-learning, whereas Hu
& Wellman (2003) extended it to the general-sum case by

learning a Nash equilibrium in each stage game and considering a mixed strategy. Nash-Q learning is guaranteed to
converge to Nash strategies under the (strong) assumption
that there exists equilibrium for every stage game, defined
by the Q-functions. In the situation where agents can be
identified as either "friends" or "foes", one can simply the
solution by alternating between fully cooperative and zerosum learning (Littman, 2001). Moreover, Littman & Stone
(2005) and de Cote & Littman (2008) draw on the folk
theorem and acquired a polynomial-time Nash equilibrium
algorithm for repeated stochastic games. Bowling & Veloso
(2002) varied the learning rate to improve the convergence.
The recent treatment of MARL was using deep neural networks as a function approximator. For instance, in addressing the non-stationary issue in MARL, various solutions
have been proposed including neural-based opponent modeling (He & Boyd-Graber, 2016), policy parameters sharing
(Gupta et al., 2017) etc. Researchers have also adopted
the paradigm of centralized training with decentralized execution for multi-agent policy-gradient learning: BICNET
(Peng et al., 2017), COMA (Foerster et al., 2016) and MADDPG (Lowe et al., 2017a), which allow the centralized critic
Q-function augmented with actions of other agents, and
the actor needs only local observation history to optimize
agent’s policy.
The above MARL approaches limit their studies mostly to
a few agents. As the number of agents grows, not only the
input space of Q grows linearly, but most critically the accumulated noises by the exploratory actions of other agents
make the Q-function learning no longer feasible. Our work
addresses the issue by making use of mean field approximation (Stanley, 1971) over the joint action space. The
parameters of the Q-function is independent of the number
of agents as it transforms multiple agents interactions into
two entities interactions (single agent vs. a distribution of
other agents). This would also effectively alleviate the exploratory noise (Colby et al., 2015) caused by many other
agents and allow each agent to effectively determine which
actions are beneficial or not, as we have demonstrated from
one of our experiments.
Our work is also closely related to the recent development
of mean field game (MFG) (Lasry & Lions, 2007; Huang
et al., 2006; Weintraub et al., 2006). MFG studies population behaviors resulting from the aggregations of decisions
taken from individuals. Mathematically, the dynamics are
governed by a set of two stochastic differential equations
that model the backward dynamics of individual’s value
function, and the forward dynamics of the aggregate distribution of agent population. Despite that the backward
equation equivalently describes what Bellman equation indicates in MDP, the primarily goal for MFG is rather for a
model-based planning/control and to infer the movements
of the individual density through time. We also employ the
mean field approximation (Stanley, 1971) in physics, but our

1.0

0.8

0.8

0.6

IL
FMQ
Rec-FMQ
MF-Q
MAAC
MF-AC

0.4
0.2
0.0

0

200

400

600

Timestep

(a) N = 100

800

1000

1.0

IL
FMQ
Rec-FMQ
MF-Q
MAAC
MF-AC

0.6
0.4

Performance

1.0

Performance

Performance

Mean Field Multi-Agent Reinforcement Learning

0.2
0.0

0

200

400

600

Timestep

800

1000

(b) N = 500

IL
FMQ
Rec-FMQ
MF-Q
MAAC
MF-AC

0.8
0.6
0.4
0.2
0.0

0

200

400

600

Timestep

800

1000

(c) N = 1000

Figure 1: Learning over N agents in the GS environment with µ = 400 and σ = 200.
work is different in that we focus on a model-free solution of
learning optimal actions when the dynamics of the system
(environment) and the reward function are unknown. Very
recently, Yang et al. (2017) built a connection between MFG
and RL. Their focus is, however, on inverse RL in order to
learn both the reward function and the forward dynamics of
the MFG from the policy data, whereas our goal is to form
a computable Q-function under the framework of temporal
difference learning.

5. Experiments
We analyze and evaluate our algorithms by three different
scenarios, including two stage games (Busoniu & De Schutter), namely Gaussian Squeeze and Ising Model, and a mixed
cooperative-competitive battle game.
5.1. Gaussian Squeeze
5.1.1. E NVIRONMENT
In the Gaussian Squeeze problem (GS) (HolmesParker et al.,
2014), N homogeneous agents determine their individual
PN
j
actions a j to jointly optimize the sum x =
j=1 a for
resource allocation. Each agent has 10 action choices, i.e.,
integer value from 0 to 9 as the quantity of resource utilized
by this agent. This problem is similar to traffic congestion
domains, where each agent is a traffic controller of a single
intersection and wants to send a j vehicles into a shared
main route. Controllers are expected to coordinate with
each other to make the full use of the main route while
avoiding too much traffic. The system objective is defined as
−(x−µ)2

G(x) = xe σ 2 , where µ and σ is the predefined mean
and standard deviation of the system objective’s Gaussian
respectively. The goal of agents is to learn to allocate system
resources efficiently, avoiding over- or under-utilization.
This GS problem is ideal for an ablation study on the impact
of multi-agent exploratory noises toward the learning (Colby
et al., 2015).
5.1.2. M ODEL S ETTINGS
We implement MF-Q and MF-AC following the framework
of centralized training (shared Q) with decentralized execution (independent policy). We compare them with 4

baseline models: (1) Independent Learner (IL) (Watkins &
Dayan, 1992), a decentralized Q-Learning algorithm that
agents only observe local actions and update their Q-values
without regard to the actions performed by other agents; (2)
Frequency Maximum Q-value (FMQ) (Kapetanakis & Kudenko, 2002), a modified IL which increases the Q-values
of actions that frequently produced good rewards in the past;
(3) Recursive Frequency Maximum Q-value (Rec-FMQ)
(Matignon et al., 2012), an improved and robust version
of FMQ which recursively computes the occurrence frequency using a linear interpolation to evaluate and choose
actions; and (4) Multi-agent Actor-Critic (MAAC), a variant of MADDPG architecture (Lowe et al., 2017b) for the
discrete action space in the task. All models use the multilayer perception (MLP) as function approximator. The
full descriptions of implementation details are provided in
Appendix B.1.
5.1.3. R ESULTS
Fig. 1 illustrates our results for different number of agents
in the GS environment with µ = 400 and σ = 200 with
three numbers of agents, i.e., N = 100, 500, 1000, for 3
levels of congestion settings respectively. Fig. 1a shows
that all models perform well in the small GS domain. As
the number of agents becomes larger, Figs. 1b and 1c show
MF-Q and MF-AC are still capable of learning the optimal
allocation effectively after a few iterations, whereas all the
baselines, IL, FMQ, Rec-FMQ and MAAC, fail to learn
at all. This advantage is largely owing to the mean field
methods awareness of other agents’ actions, while still keeping the interactions among agents relatively small, which
reduces the effect of exploratory behaviors of the other
agents. MF-Q converges quicker than MF-AC as it takes
time for the Actor to learn an optimal policy. Both FMQ and
Rec-FMQ fail to reach good performance because agents
cannot distinguish rewards received for the same action due
to various behaviors of other agents and stochastic rewards,
and are thus unable to update their own Q-values w.r.t. the
actual contributions. It is worth noticing that MAAC (all
others actions as independent inputs in the Q function) is
inefficient in learning when the number of agents becomes
large. It fails to handle the non-aggregated noises from the
exploration of many other agents.

Order Parameter

Mean Field Multi-Agent Reinforcement Learning
1.0
0.8

= 1.2

5

5

5

10

10

0.4

15

15

15

0

0.2
0.5

1.0

1.5

2.0

Temperature

2.5

MSE
OP

0.8

0.8

0.6

0.6

0.4

0.4

0.2

0.2
5000

10000

Timestep

15000

200000.0

(a) τ = 0.8

1.0

1.0
MSE
OP

0.8

0.8

0.6

0.6

0.4

0.4

0.2

0.2

0.0 0

5000

10000

Timestep

15000

200000.0

(b) τ = 1.2

Figure 3: Performance against training steps in 20x20 grid
Ising model.
5.2. MARL for Ising Model
5.2.1. E NVIRONMENT
In statistical mechanics, the Ising model is a mathematical model of ferromagnetism (Ising, 1925). It is also
proved to be successful in numerous applications, particularly in sociophysics (Galam & Walliser, 2010). The typical
mean field
model (Stanley, 1971) can iteratively solve
PIsing
j
ha j i =
a a P(a) for every site j on the condition that
the reward mapping function is given. See Eq. (23) in Appendix B.2 for details. Our mean field MARL, however, is
different as the algorithm does not need to know the reward
function, but learns it over time.
To fit our case, we turn the Ising model into a stage game
where
for each site (agent) is as r j = h j a j +
P the reward
λ
j k
j
k∈K j a a , where K is the set of nearest neighbors of
2
j
site j, h ∈ R is the external field affecting site j, and λ ∈ R
is an interaction term determines how much the sites tend
to align in the same direction. Unlike the typical setting
above, each site does not know the reward function exactly,
but aims to maximize its received reward by changing its
individual policy π j to choose the spin state: up or down.
In addition to the observed reward, the order parameter (OP)
(Stanley, 1971) is also a commonly used evaluation metric
|N −N |
for Ising model. OP is defined as ξ = ↑ N ↓ , where N↑
represents the number of sites spinning up, and vice versa.
The closer the OP is to 1, the more orderly the system.
5.2.2. M ODEL S ETTINGS
To prove the correctness of MF-Q value, we implement
MCMC methods (Binder et al., 1993) to simulate the Ising
model and provide the ground truth for our proposed method.
MCMC relies on repeated random sampling to obtain numer-

10

15

< C : = 0.9

0

Mean Squared Error

1.0

Order Parameter

Mean Squared Error

1.0

5

0

5

10

15

0

(a) MF-Q
C:

0

= 1.2

5

5

5

10

10

15

15

15

5

10

15

0

5

10

15

(b) MCMC

5

10

15

> C : = 2.0

0

10

0

> C : = 2.0

0

10

Figure 2: The OP at equilibrium w.r.t. the Ising model
temperature setting in 20x20 grid Ising model.
Order Parameter

C:

0

0.6

0.0
0.0

0.0 0

< C : = 0.9

0

MCMC
MF-Q

0

5

10

15

Figure 4: The spinning pattern of MCMC and MF-Q at equilibrium under different Ising model temperature settings.
ical results and is often used in physical and mathematical
domains. The full descriptions of MCMC and MF-Q for
Ising model are provided in Appendix B.2. Notice that we
do not know exactly the mapping function in Eq. (23), but
rather using Temporal Difference learning to approximate
ha j i during lea rning procedure. One of our learning goals
is to obtain the accurate approximation of ha j i. Once it is
accurately approximated, we should be able to converge to
the simulation result of Ising model provided by MCMC.
5.2.3. R ESULTS
Correctness of MF-Q Value. Fig. 2 illustrates the order
parameter at equilibrium w.r.t. the Ising Model temperature
setting in 20x20 grid Ising model. As shown on the graph,
MF-Q can converge to a similar OP value with MCMC
under different temperature settings, proving the correctness of our algorithm. Also, MF-Q finds a similar Curie
temperature as MCMC, which is around τ = 1.2.
Learning Ability. Fig. 3 illustrates the Mean Squared Error
between learned Q-value and reward target (mean field) in
20x20 grid Ising model w.r.t. training steps. As one can
observe, MF-Q is able to learn the target well under low temperature settings (Fig. 3a). Around the Curie temperature,
the environment will force the site to choose spin state with
too much randomness, resulting in a lower OP and higher
MSE (Fig. 3b).
Convergence Equilibrium. Fig. 4 shows some example
results at equilibrium from our simulation. Most of the
sites choose the same spinning pattern in both MCMC and
MF-Q under a low temperature (τ = 0.9). As the temperature rises, some sites start to change its spinning pattern
and some patches are formed as spontaneous magnetization
with the constraint of neighboring sites and system energy
(τ = 1.2). This phenomenon is mostly observed around
the Curie temperature. After passing the Curie temperature,
the system becomes totally disordered due to large thermal
fluctuations, resulting in a random spinning situation.

Mean Field Multi-Agent Reinforcement Learning

As demonstrated in our previous experiment (Figure 1), a
variant MAAC also fails when the agent number is large.
We thus exclude MADDPG as a baseline, while focusing on
the comparison between mean field methods (MF-Q, MFAC) and their non-mean field counterparts, i.e., independent
Q-learning (IL) and advantageous actor critic (AC).
(a) two groups agents

(b) chasing and cooperation

Figure 5: Battle Game.
1.50

vs IL
vs MF-Q

400

vs MF-AC
vs AC

Total-Reward

Win-Rate

1.25
1.00
0.75
0.50
0.25
0.00

AC

IL

MF-Q

MF-AC

vs IL
vs MF-Q

vs MF-AC
vs AC

300
200
100
0

AC

IL

MF-Q

MF-AC

(a) the rate of wining per round (b) the total reward per round

Figure 6: Performance of algorithms on wining rate, total reward. Each bar cluster shows the index of a set of
competing algorithms, and a higher index is better.
400

MF-AC

250
0

0

Total reward

Total reward

200

200
400
600

250
500
750
1000

MF-Q

1250

800
0

500

1000

Epochs

1500

2000

(a) MF-AC learning curve

0

500

1000

Epochs

1500

2000

(b) MF-Q learning curve

Figure 7: It can be seen from the above two figures that the
MF-AC converges much slower than that of MF-Q.
5.3. Mixed Cooperative-Competitive Battle Game
5.3.1. E NVIRONMENT
The Battle game in the Opensource MAgent system (Zheng
et al., 2017) is a Mixed Cooperative-Competitive scenario
with two armies fighting against each other in a grid world,
shown in Fig. 5. In our settings, each army consists of 64
homogeneous agents. In each timestep, each agent takes
actions to either move to or attack one of the 8 neighbor
grids. The goal of each army is to get more rewards by
collaborating teammates and eliminating all opponents. We
adopt the default setting, i.g., the reward is -0.005 for every
move, 0.2 for attacking an enemy agent, 5 for killing an
enemy agent, -0.1 for attacking a blank grid, and -0.1 for
being attacked or killed.
5.3.2. M ODEL S ETTINGS
Our MF-Q and MF-AC are compared with the baselines
that are successful on the MAgent platform (Zheng et al.,
2017). In the battle game there are a large number of agents,
each of which can only take in observation within its local view, thus the Q-function involves different numbers
of neighbor agents. This makes MADDPG not feasible.

5.3.3. R ESULTS
We get all models with 2000 rounds self-play training, then
use them for cross-comparative experiments. Fig. 6 shows
the result of winning rate and the total reward over 2000
rounds cross-comparative experiments. It is evidence that
on all the metrics mean field methods, i.e., MF-Q and MFAC, largely outperfom the corresponding baselines, i.e., IL
and AC respectively, which shows the effectiveness of the
mean field approximation.
In addition, IL performs much better than AC, and although
worse than its mean field counterpart MF-Q, better than MFAC. This is consistent with the findings in in the Gaussian
Squeeze game (in Figure 1) that IL converges quicker than
AC. This might imply the effectiveness of off-policy learning with shuffled buffer replay in many-agent RL toward a
more stable learning process. The Q-learning family tends
to introduce a positive bias (Hasselt, 2010) by using the maximum action value as an approximation for the maximum
expected action value; we find that such overestimation can
be beneficial for each single agent to find the best response
to others even though the environment itself is still varying.
We further test the situation when the agent number is 8 for
each army, where MF-AC achieves 58% wining rate and get
total rewards of 34.4 while AC achieves 42% wining rate
and get total rewards of 31.6. We see that (i) when the agent
number is small, MF-AC slightly outperforms AC mainly
because of the higher capacity of the Q-network; (ii) when
the agent number gets larger, MF-AC starts to significantly
outperform AC, which verifies the mean field methods work
particularly better for scenarios with a larger number of
agents.

6. Conclusions
In this paper, we have developed mean field reinforcement
learning that tackles the problem of intractable computation
and high dynamics of interactions among a large number
of agents. With mean field theory, we approximated the
Q-function by using the distribution of the actions from
other agents as inputs in order to balance the complexity of
the interactions modeled and the learning ability. It is done
by iteratively learning the best responded actions on the
condition of the distribution of other agents’ actions, which
is subsequently updated each time a new action policy is
obtained. We provided our theoretical analysis on our MFQ learning convergence. The results from three different

Mean Field Multi-Agent Reinforcement Learning

types of environments have shown the effectiveness of our
approaches when the number of agents is large.
There are two interesting future directions. The first is to
investigate the local and global communication mechanisms,
and a more appropriate network architecture for mean field
Q-functions in such a setting. The second is to construct a
Bayesian solution to embed the prior knowledge of agents
optimal policy into the distribution of the agent population.

References
Binder, Kurt, Heermann, Dieter, Roelofs, Lyle, Mallinckrodt, A John, and McKay, Susan. Monte carlo simulation
in statistical physics. Computers in Physics, 7(2):156–
157, 1993.

Foerster, Jakob N., Chen, Richard Y., Al-Shedivat, Maruan,
Whiteson, Shimon, Abbeel, Pieter, and Mordatch, Igor.
Learning with opponent-learning awareness. CoRR,
abs/1709.04326, 2017.
Galam, Serge and Walliser, Bernard. Ising model versus
normal form game. Physica A: Statistical Mechanics and
its Applications, 389(3):481–489, 2010.
Gupta, Jayesh K, Egorov, Maxim, and Kochenderfer, Mykel.
Cooperative multi-agent control using deep reinforcement
learning. In AAMAS, pp. 66–83. Springer, 2017.
Hasselt, Hado V. Double q-learning. In NIPS, pp. 2613–
2621, 2010.

Blume, Lawrence E. The statistical mechanics of strategic
interaction. Games and economic behavior, 5(3):387–
424, 1993.

He, He and Boyd-Graber, Jordan L. Opponent modeling in
deep reinforcement learning. In Balcan, Maria-Florina
and Weinberger, Kilian Q. (eds.), ICML, volume 48, pp.
1804–1813. JMLR.org, 2016.

Bowling, Michael and Veloso, Manuela. Rational and convergent learning in stochastic games. In IJCAI, volume 17,
pp. 1021–1026. Lawrence Erlbaum Associates Ltd, 2001.

HolmesParker, C, Taylor, M, Zhan, Y, and Tumer, K. Exploiting structure and agent-centric rewards to promote
coordination in large multiagent systems. In Adaptive
and Learning Agents Workshop, 2014.

Bowling, Michael and Veloso, Manuela. Multiagent learning using a variable learning rate. Artificial Intelligence,
136(2):215–250, 2002.

Hu, Junling and Wellman, Michael P. Nash q-learning
for general-sum stochastic games. Journal of machine
learning research, 4(Nov):1039–1069, 2003.

Busoniu, Lucian and De Schutter, Bart. A comprehensive
survey of multiagent reinforcement learning.

Huang, Minyi, Malhamé, Roland P, Caines, Peter E, et al.
Large population stochastic dynamic games: closed-loop
mckean-vlasov systems and the nash certainty equivalence principle. Communications in Information & Systems, 6(3):221–252, 2006.

Cao, Zhe, Qin, Tao, Liu, Tie-Yan, Tsai, Ming-Feng, and
Li, Hang. Learning to rank: from pairwise approach
to listwise approach. In Proceedings of the 24th international conference on Machine learning, pp. 129–136.
ACM, 2007.
Colby, Mitchell K., Kharaghani, Sepideh, HolmesParker,
Chris, and Tumer, Kagan. Counterfactual exploration
for improving multiagent learning. AAMAS ’15, pp.
171–179, 2015. ISBN 978-1-4503-3413-6.
de Cote, Enrique Munoz and Littman, Michael L. A
polynomial-time nash equilibrium algorithm for repeated
stochastic games. In McAllester, David A. and Myllymäki, Petri (eds.), UAI 2008, pp. 419–426. AUAI Press,
2008. ISBN 0-9749039-4-9.
Fink, Arlington M et al. Equilibrium in a stochastic n-person
game. Journal of science of the hiroshima university,
series ai (mathematics), 28(1):89–93, 1964.
Foerster, Jakob, Assael, Yannis, de Freitas, Nando, and
Whiteson, Shimon. Learning to communicate with deep
multi-agent reinforcement learning. In NIPS, pp. 2137–
2145, 2016.

Ising, Ernst. Beitrag zur theorie des ferromagnetismus.
Zeitschrift für Physik, 31(1):253–258, 1925.
Jaakkola, Tommi, Jordan, Michael I, and Singh, Satinder P.
Convergence of stochastic iterative dynamic programming algorithms. In NIPS, pp. 703–710, 1994.
Jeong, Seong Hoon, Kang, Ah Reum, and Kim, Huy Kang.
Analysis of game bot’s behavioral characteristics in social
interaction networks of mmorpg. In ACM SIGCOMM,
volume 45, pp. 99–100. ACM, 2015.
Kapetanakis, Spiros and Kudenko, Daniel. Reinforcement
learning of coordination in cooperative multi-agent systems. In NCAI, pp. 326–331, Menlo Park, CA, USA,
2002. ISBN 0-262-51129-0.
Konda, Vijay R. and Tsitsiklis, John N. Actor-critic
algorithms.
In Solla, S. A., Leen, T. K., and
Müller, K. (eds.), Advances in Neural Information
Processing Systems 12, pp. 1008–1014. MIT Press,
2000. URL http://papers.nips.cc/paper/
1786-actor-critic-algorithms.pdf.

Mean Field Multi-Agent Reinforcement Learning

Kreyszig, Erwin. Introductory functional analysis with
applications, volume 1. wiley New York, 1978.
Lasry, Jean-Michel and Lions, Pierre-Louis. Mean field
games. Japanese journal of mathematics, 2(1):229–260,
2007.
Lemke, Carlton E and Howson, Jr, Joseph T. Equilibrium
points of bimatrix games. Journal of the Society for Industrial and Applied Mathematics, 12(2):413–423, 1964.
Littman, Michael L. Markov games as a framework for
multi-agent reinforcement learning. In ICML, volume
157, pp. 157–163, 1994.
Littman, Michael L. Friend-or-foe q-learning in generalsum games. In ICML, volume 1, pp. 322–328, 2001.
Littman, Michael L and Stone, Peter. A polynomial-time
nash equilibrium algorithm for repeated games. Decision
Support Systems, 39(1):55–66, 2005.
Lowe, Ryan, Wu, Yi, Tamar, Aviv, Harb, Jean, Abbeel,
OpenAI Pieter, and Mordatch, Igor. Multi-agent actorcritic for mixed cooperative-competitive environments.
In NIPS, pp. 6382–6393, 2017a.
Lowe, Ryan, Wu, Yi, Tamar, Aviv, Harb, Jean, Abbeel,
Pieter, and Mordatch, Igor. Multi-agent actor-critic for
mixed cooperative-competitive environments. In Guyon,
Isabelle, von Luxburg, Ulrike, Bengio, Samy, Wallach,
Hanna M., Fergus, Rob, Vishwanathan, S. V. N., and
Garnett, Roman (eds.), NIPS, pp. 6382–6393, 2017b.
Matignon, Laetitia, Laurent, Guillaume J, and Le Fort-Piat,
Nadine. Independent reinforcement learners in cooperative markov games: a survey regarding coordination
problems. The Knowledge Engineering Review, 27(1):
1–31, 2012.
Panait, Liviu and Luke, Sean. Cooperative multi-agent
learning: The state of the art. AAMAS, 11(3):387–434,
2005.
Peng, Peng, Yuan, Quan, Wen, Ying, Yang, Yaodong, Tang,
Zhenkun, Long, Haitao, and Wang, Jun. Multiagent
bidirectionally-coordinated nets for learning to play starcraft combat games. arXiv preprint arXiv:1703.10069,
2017.
Rendle, Steffen. Factorization machines with libfm. ACM
Transactions on Intelligent Systems and Technology
(TIST), 3(3):57, 2012.
Silver, David, Lever, Guy, Heess, Nicolas, Degris, Thomas,
Wierstra, Daan, and Riedmiller, Martin. Deterministic
policy gradient algorithms. In ICML, pp. 387–395, 2014.
Stanley, H Eugene. Phase transitions and critical phenomena.
Clarendon, Oxford, 9, 1971.

Szepesvári, Csaba and Littman, Michael L. A unified analysis of value-function-based reinforcement-learning algorithms. Neural computation, 11(8):2017–2060, 1999.
Tan, Ming. Multi-agent reinforcement learning: Independent vs. cooperative agents. In Proceedings of the tenth
international conference on machine learning, pp. 330–
337, 1993.
Troy, Carol Alvarez. Envisioning stock trading where the
brokers are bots. New York Times, 16, 1997.
van der Wal, J. Stochastic dynamic programming: successive approximations and nearly optimal strategies for
markov decision processes and markov games. 1980.
Wang, Jun, Zhang, Weinan, Yuan, Shuai, et al. Display
advertising with real-time bidding (rtb) and behavioural
targeting. Foundations and Trends R in Information Retrieval, 11(4-5):297–435, 2017.
Watkins, Christopher JCH and Dayan, Peter. Q-learning.
Machine learning, 8(3-4):279–292, 1992.
Weintraub, Gabriel Y, Benkard, Lanier, and Van Roy, Benjamin. Oblivious equilibrium: A mean field approximation for large-scale dynamic games. In Advances in
neural information processing systems, pp. 1489–1496,
2006.
Yang, Jiachen, Ye, Xiaojing, Trivedi, Rakshit, Xu, Huan,
and Zha, Hongyuan. Deep mean field games for learning optimal behavior policy of large populations. arXiv
preprint arXiv:1711.03156, 2017.
Zheng, Lianmin, Yang, Jiacheng, Cai, Han, Zhang, Weinan,
Wang, Jun, and Yu, Yong. Magent: A many-agent reinforcement learning platform for artificial collective intelligence. arXiv preprint arXiv:1712.00600, 2017.

Mean Field Multi-Agent Reinforcement Learning

A. Detailed Mean Field Reinforcement Learning Algorithms
Here we provide the pseudo code for Mean Field Q-learning (MF-Q) in Algorithm 1 and Mean Field Actor-Critic (MF-AC)
in Algorithm 2.
Algorithm 1 Mean Field Q-learning (MF-Q)
Initialise Q φ , Q φ− , and ā
while training not finished do
For each agent j, sample action a j w.r.t. Q φ , exploration rate , and ā00 from the last time step
Take the joint action a = (a1, . . . , a N ) and observe the reward r and next state s0
Store (s, a, r, s0, ā) in replay buffer D
for agent j = 1 to N do
Sample a random minibatch of K samples (s, a, r, s0, ā) from D
Sample action (a j )0 w.r.t. Q φ− , exploration rate , and ā
Set y = r j + γ Q φ− ((s j )0, (a j )0, ā0 )
2
P
Update Q-network by minimizing the loss L(φ) = K1 K y − Q φ (s j , a j , ā)
end for
Update target network parameters for each agent j:
j
j
j
φ−
← τφ−
+ (1 − τ)φ−

end while
Algorithm 2 Mean Field Actor-Critic (MF-AC)
Initialize Q φ , Q φ− , πθj , πθj − and ā
while training not finished do
For each agent j, sample action a j = πθj (s j )
Take the joint action a = (a1, . . . , a N ) and observe the reward r and next state s0
Store (s, a, r, s0, ā) in replay buffer D
for agent j = 1 to N do
Sample a random minibatch of K samples (s, a, r, s0, ā) from D
Set y = r j + γ Q φ− ((s j )0, (a j )0, ā0 )|(a j )0 =π j ((s j )0 )
θ−
2
P
Update critic by minimizing the loss L(φ) = K1 K y − Q φ (s j , a j , ā)
Update actor using the sampled policy gradient:
∇θ j J(θ j ) ≈

1 X
∇θ j log πθj ((s j )0 )Q φ− ((s j )0, (a j )0, ā0 )|(a j )0 =π j ((s j )0 )
K K
θ−

end for
Update target network parameters for each agent j:
j
j
j
φ−
← τφ−
+ (1 − τ)φ−

end while

j
j
j
θ−
← τθ −
+ (1 − τ)θ −

B. Experiment Details
B.1. Gaussian Squeeze
IL, FMQ, Rec-FMQ and MF-Q all use a three-layer MLP to approximate Q-value. All agents share the same Q-network
for each experiment. The shared Q-network takes an agent embedding as input and computes Q-value for each candidate
action. For MF-Q, we also feed in the action approximation ā. We use the Adam optimizer with a learning rate of
0.00005 and -greedy exploration unless otherwise specified. For FMQ, we set the exponential decay rate s = 0.006, start
temperature max_temp=1000 and FMQ heuristic c = 5. For Rec-FMQ, we set the frequency learning rate α f = 0.01.

Mean Field Multi-Agent Reinforcement Learning

Algorithm 3 MCMC in Ising Model
initialize spin state a ∈ {−1, 1} N for N sites
while training not finished do
randomly choose site j ∈ I N
j
flip the spin state for site j: (a j )0 ← −aP
compute neighbor energy E(a, h) = − j (h j a j +
randomly choose  ∼ U(0, 1)
if exp((E(a j , h) − (E(a j )0, h))/τ) >  then
a j ← (a j )0
end if
end while

λ
2

P

k∈K j

a j ak ) for a j and (a j )0

MAAC and MF-AC use the Adam optimizer with a learning rate of 0.001 and 0.0001 for Critics and Actors respectively,
and τ = 0.01 for updating the target networks. We share the Critic among all agents in each experiment and feed in an agent
embedding as extra input. Actors are kept seperate. The discounted factor γ is set to be 0.95 and the mini-batch size is set to
be 200. The size of the replay buffer is 106 and we update the network parameters after every 500 samples added to the
replay buffer.
For all models, we use the performance of the joint-policy learned up to that point if learning and exploration were turned
off (i.e., take the greedy action w.r.t. the learned policy) to compare our method with the above baseline models.
B.2. Ising Model
An Ising model is defined as a stateless system with N homogeneous sites on a finite square lattice. Each site determines
their individual spin a j to interact with each other and aims to minimize the system energy for a more stable environment.
The system energy is defined as
E(a, h) = −

X
j

(h j a j +

λ X j k
a a )
2
j

(20)

k∈K

where K j is the set of nearest neighbors of site j, h j ∈ R is the external field affecting site j, and λ ∈ R is an interaction
term determines how much the sites tend to align in the same direction. The system is said to reach an equilibrium point
when the system energy is minimized, with the probability
exp (−E(a, h)/τ)
P(a) = P
,
a exp(−E(a, h)/τ)

(21)

where τ is the system temperature. When the temperature rises beyond a certain point (the Curie temperature), the system
can no longer keep a stable form and a phase transition happens. As the ground-truth is known, we would be able to evaluate
the correctness of the Q-function learning when there is a large body of agents interacted.
P
The mean field theory provides an approximate solution to ha j i = a a j P(a) through a set of self-consistent mean field
equations

P
exp −[h j a j + λ k∈K j hak i]/τ
j
.
P
ha i =
(22)
1 + exp −[h j a j + λ k∈K j hak i]/τ

which can be solved iteratively by

j (t+1)

ha i

where t represents the number of iterations.


P
exp −[h j a j + λ k∈K j hak i(t) ]/τ
,
P
=
1 + exp −[h j a j + λ k∈K j hak i(t) ]/τ

(23)

To learn an optimal joint policy π ∗ for Ising model, we use the stateless Q-learning with mean field approximation (MF-Q),
defined as
Q j (a j , ā) ← Q j (a j , ā) + α[r j − Q j (a j , ā)],

(24)

Mean Field Multi-Agent Reinforcement Learning

where the mean ā is given as the mean hai i from the last time step, and the individual reward is
λ X j k
r j = hj aj +
a a .
2
j

(25)

k∈K

To balance the trade-off between exploration and exploitation under low temperature settings, we use a policy with Boltzmann
exploration and a decayed exploring temperature. The temperature for Boltzmann exploration of MF-Q is multiplied by a
decay factor exponentially through out the training process.

Without lost of generality, we assume λ > 0, thus neighboring sites with the same action result in lower energy (observe
higher reward) and are more stable. Each site should also align with the sign of external field h j to reduce the system energy.
For simplification, we eliminate the effect of external fields and assume the model to be discrete, i.e., ∀ j ∈ N, h j = 0, a j ∈
{−1, 1}.
We simulate the Ising model using Metropolis Monte Carlo methods (MCMC). After initialization, we randomly change a
site’s spin state and calculate the energy change, select a random number between 0 and 1, and accept the state change only
(E j −(E j )0 )
τ

if the number is less than e
the more probable spin states.

. This is called the Metropolis technique, which saves computation time by selecting

B.3. Battle Game
IL and MF-Q have almost the same hyper-parameters settings. The learning rate is α = 10−4 , and with a dynamic
exploration rate linearly decays from γ = 1.0 to γ = 0.05 during the 2000 rounds training. The discounted factor γ is set to
be 0.95 and the mini-batch size is 128. The size of replay buffer is 5 × 105 .

AC and MF-AC also have almost the same hyper-parameters settings. The learning rate is α = 10−4 , the temperature of
soft-max layer in actor is τ = 0.1. And the coefficient of entropy in the total loss is 0.08, the coefficient of value in the total
loss is 0.1.

C. Details of MF-Q

P
Proposition 1. Let the metric space be R N , and the metric be M(a, b) = j |a j − b j |, ∀ a = {a j }1N , b = {b j }1N . If the
Q function is Lipschitz continuous w.r.t. the input of a j , then for sufficiently small temperature β, the operator

exp − βQ(s, a j , ā)
j

B(a ) = P
exp − βQ(s, ak , ā)
a k ∈A j
is a contraction mapping. The proof is in Appendix due to the space limit.

Proof. Following the contraction mapping theorem (Kreyszig, 1978), in order to be a contraction, the operator has to satisfy:
where 0 ≤ α < 1.

M(B(a), B(b)) ≤ αM(a, b), ∀a, b

Here we start from binary case and then adapt to the general multinomial case. We first rewrite B(a j ) as:
B(a j ) = 1 −

1
.
1 + exp (−βQ(s, a j , ā))

Then we have:
1
1
−
1 + e−βQ(s,a j , ā)
1 + e−βQ(s,b j , ā)
βe−βQ0
=
Q(s, a j , ā) − Q(s, b j , ā)
(1 + e−βQ0 )2
1
· Q(s, a j , ā) − Q(s, b j , ā)
≤
4T
1
≤
· K · a j − bj
4T

|B(a j ) − B(b j )| =

(26)

Mean Field Multi-Agent Reinforcement Learning

In the 2nd equation, we apply the mean value theorem in Calculus: ∃x0 ∈ [x1, x2 ], s.t., f (x1 ) − f (x2 ) = f 0 (x0 )(x1 − x2 ).
In the 3rd equation we use the maximum value for e−βQ0 /(1 + e−βQ0 )2 = 1/4 when Q0 = 0. In the last equation we we
apply the Lipschitz constraint in the assumption where constant K ≥ 0. Finally, we have:
M(B(a), B(b)) ≤
=

X
1
·K·
a j − bj
4T
j
K
M(a, b)
4T

(27)

In order for the contraction to hold, T > K4 . In other words, when the action space is binary for each agent, and the
temperature is sufficiently large, the mean field procedure converges.
This proposition can be easily extended to multinomial case by replacing binary variable a j by a multi-dimensional binary
indicator vector a j , on each dimension, the rest of the derivations would remain essentially the same.
C.1. Discussion on Rationality
In aligned with (Bowling & Veloso, 2001; 2002), we argue that to better evaluate a multi-agent learning algorithm, on top of
the convergence guarantee, discussion on property of Rationality is also needed.
Property 1. (also see (Bowling & Veloso, 2001; 2002)) In an N-agent stochastic game defined in this paper, given all
agents converge to stationary policies, if the learning algorithm converges to a policy that is a best response to the other
agents’ policies, then the algorithm is Rationale.
Our mean field Q-learning is rational in that Eq. (6) converts many agents interactions into two-body interactions between a
single agent and the distribution of other agents actions. When all agents follow stationary policies, their policy distribution
would be stationary too. As such the two-body stochastic game becomes an MDP, and the agent would choose a policy
(based on Assumption 2) which is the best response to the distribution of other stationary policies. As agents are symmetric
in our case, they all the best response to the distributions, therefore are rational.

