The Mechanics of n-Player Differentiable Games
David Balduzzi 1 Sébastien Racaniere 1 James Martens 1 Jakob Foerster 2 Karl Tuyls 1 Thore Graepel 1

arXiv:1802.05642v1 [cs.LG] 15 Feb 2018

Abstract
The cornerstone underpinning deep learning is the
guarantee that gradient descent on an objective
converges to local minima. Unfortunately, this
guarantee fails in settings, such as generative adversarial nets, where there are multiple interacting
losses. The behavior of gradient-based methods
in games is not well understood – and is becoming
increasingly important as adversarial and multiobjective architectures proliferate. In this paper,
we develop new techniques to understand and control the dynamics in general games. The key result
is to decompose the second-order dynamics into
two components. The first is related to potential
games, which reduce to gradient descent on an implicit function; the second relates to Hamiltonian
games, a new class of games that obey a conservation law, akin to conservation laws in classical
mechanical systems. The decomposition motivates Symplectic Gradient Adjustment (SGA), a
new algorithm for finding stable fixed points in
general games. Basic experiments show SGA is
competitive with recently proposed algorithms for
finding local Nash equilibria in GANs – whilst at
the same time being applicable to – and having
guarantees in – much more general games.

1. Introduction
Recent progress in machine learning is heavily dependent on
using gradient descent, applied to optimize the parameters
of models with respect to a (single) objectives. A basic
result is that gradient descent converges to a local minimum
of the objective under a broad range of conditions (Lee et al.,
2017). However, there is a rapidly growing set of powerful
models that do not optimize a single objective, including:
generative adversarial networks (Goodfellow et al., 2014),
proximal gradient TD learning (Liu et al., 2016), multi-level
optimization (Pfau & Vinyals, 2016), synthetic gradients
1
DeepMind 2 University of Oxford. Correspondence to: David
Balduzzi <dbalduzzi@google.com>.

(Jaderberg et al., 2017), hierarchical reinforcement learning
(Wayne & Abbott, 2014; Vezhnevets et al., 2017), curiosity
(Pathak et al., 2017), and imaginative agents (Racanière
et al., 2017). In effect, the models are trained via games
played by cooperating and competing modules.
No-regret algorithms such as gradient descent are guaranteed to converge to coarse correlated equilibria in games
(Stoltz & Lugosi, 2007). However, the dynamics do not
converge to Nash equilibria – and do not even stabilize –
in general (Mertikopoulos et al., 2018). Concretely, cyclic
behaviors emerge even in simple cases, see example 1.
This paper presents an analysis of the second-order structure
of game dynamics that allows to identify two classes of
games, potential and Hamiltonian, that are easy to solve
separately. We then derive symplectic gradient adjustment
(SGA), a method for finding stable fixed points in games.
SGA’s performance is evaluated in basic experiments.
1.1. Background and problem description
Procedures that converge to Nash equilibria have been found
for restricted game classes: potential games, 2-player zerosum games and a few others (Hu & Wellman, 2003; Hart
& Mas-Colell, 2013). Finding equilibria can be reformulated as a nonlinear complementarity problem, but these
are ‘hopelessly impractical to solve’ in general (Shoham &
Leyton-Brown, 2008) because the problem is PPAD hard
(Daskalakis et al., 2009).
Players are primarily neural nets in our setting. We therefore restrict to methods based on gradient-descent (gametheorists have considered a much broader range of techniques). The losses are not necessarily convex in any of
their parameters, so the concept of Nash equilibrium does
not apply here. Nash equilibria are only guaranteed to exist
for convex losses and constraint sets. Even leaving existence aside, finding Nash equilibria is analogous to, but
much harder than, finding global minima in neural nets –
which is not realistic with gradient-based methods.
There are (at least) three problems with gradient descent in
games. Firstly, the potential existence of cycles (recurrent
dynamics) implies there are no convergence guarantees,
see example 1 and Mertikopoulos et al. (2018). Secondly,
even when gradient descent converges, the rate may be

Differentiable Game Mechanics

too slow in practice because ‘rotational forces’ necessitate
extremely small learning rates (see figure 3). Finally, since
there is no single objective, there is no way to measure
progress. Application-specific proxies have been proposed,
for example the inception score for GANs (Salimans et al.,
2016), but these are little help during training – the inception
score is no substitute for looking at samples.
1.2. Outline and summary of main contributions
We start with the well-known case of a zero-sum bimatrix
game: example 1. It turns out that the dynamics (that is,
the dynamics under simultaneous gradient descent) can be
reformulated via Hamilton’s equations. The cyclic behavior
arises because the dynamics live on the level sets of the
Hamiltonian. More directly useful, gradient descent on the
Hamiltonian converges to a Nash equilibrium.
Lemma 1 shows that the Hessian of any game decomposes
into symmetric and antisymmetric components. There are
thus two ‘pure’ cases: when only the symmetric component is present, or only the antisymmetric. The first case,
known as potential games (Monderer & Shapley, 1996),
have been intensively studied because they are exactly the
games where gradient descent does converge.
The second case, Hamiltonian1 games, were not studied
previously, probably because they coincide with zero-sum
games in the bimatrix case (or constant-sum, depending
on the constraints). Zero-sum and Hamiltonian games differ when the losses are not bilinear or there are more than
two players. Hamiltonian games are important because
(i) they are easy to solve and (ii) general games combine
potential-like and Hamiltonian-like dynamics. The concept
of a zero-sum game is too loose to be useful when there are
many players: any n-player game can be reformulated
Pn as
a zero-sum (n + 1)-player game where `n+1 = − i=1 `i .
Zero-sum games are as complicated as general-sum games.
Theorem 3 shows that Hamiltonian games obey a conservation law – which also provides the key to solving them, by
gradient descent on the conserved quantity.
The general case, neither potential nor Hamiltonian, is more
difficult and the focus of the remainder of the paper. Section 3 proposes symplectic gradient adjustment (SGA), a
gradient-based method for finding stable fixed points in general games. Appendix B contains TensorFlow code to compute the adjustment. The algorithm computes two Hessianvector products, at a cost of two iterations of backprop. SGA
satisfies a few natural desiderata: (D1) it is compatible with
the original dynamics and (D2, D3) it is guaranteed to find
stable equilibria in potential and Hamiltonian games.
For general games, correctly picking the sign of the adjustment (whether to add or subtract) is critical since it
1

Lu (1992) defined an unrelated notion of Hamiltonian game.

determines the behavior near unstable equilibria. Lemma 2
proves a characterization of local Nash equilibria. The characterization yields a local criterion to distinguish stable from
unstable equilibria, and a local measure of which direction
improves progress, lemma 9. Correctly aligning SGA allows
higher learning rates and faster, more robust convergence,
see theorem 7 and experiments in section 4.
Section 4 investigates a basic GAN setup from Metz et al.
(2017), that tests for mode-collapse and mode hopping.
Whereas simultaneous gradient descent completely fails; the
symplectic adjustment leads to rapid convergence – slightly
improved by correctly choosing the sign of the adjustment.
Finally, section 3.5 applies the same criterion to align consensus optimization (Mescheder et al., 2017), preventing
it from converging to unstable equilibria and (slightly) improving performance, figure 9 in the appendix.
Caveat. The behavior of SGA near saddle points is not
analysed below. On the one hand, it was only recently
shown that gradient descent behaves well near saddles when
optimizing a single objective (Lee et al., 2016; 2017). On
the other hand, Newton’s method is attracted to saddles,
see analysis and proposed remedy in Dauphin et al. (2014).
Studying saddles in general games is deferred to future
work.
1.3. Related work
Convergence to Nash equilibria in two-player games was
studied in Singh et al. (2000). WoLF (Win or Learn Fast)
is guaranteed to converge to Nash equilibria in two-player
two-action games (Bowling & Veloso, 2002). Extensions
include weighted policy learning (Abdallah & Lesser, 2008)
and GIGA-WoLF (Bowling, 2004).
There has been almost no work on convergence to fixed
points in general games. Optimistic mirror descent (OMD)
exhibits approximate convergence in two-player bilinear
zero-sum games (Daskalakis et al., 2018), which are a special case of Hamiltonian games. In more general settings it
converges to coarse correlated equilibria.
There has been interesting recent work on convergence
in GANs. Heusel et al. (2017) propose a two-time scale
methods to find Nash. However, it likely scales badly with
the number of players. Nagarajan & Kolter (2017) prove
convergence for some algorithms, but under very strong
assumptions Mescheder (2018). Consensus optimization
(Mescheder et al., 2017) is discussed in section 3.
Learning with opponent-learning awareness (LOLA) infinitesimally modifies the objectives of players to take into
account their opponents’ goals (Foerster et al., 2018). However, there are no guarantees that LOLA converges and,
moreover, in general it may modify the fixed-points of the

Differentiable Game Mechanics

A

has Nash equilibrium at (x, y) = (0, 0). The simultaneous
gradient ξ(x, y) = (Ay, −A| x) rotates around the Nash
equilibrium, see figure 1.

B

The matrix A admits singular value decomposition (SVD)
1
A = U| DV. Changing to coordinates u = D 2 Ux and
1
|
v = D 2 Vy gives `1/2 (u, v) = ±u v. Introduce the
Hamiltonian

symplectic
rotation
Figure 1. (A) ξ cycles around the origin. (B) gradient descent on
H converges to Nash.

game.
Notation. Dot products are written as v| w or hv, wi. The
angle between vectors is θ(v, w). Positive definiteness is
denoted S  0. Omitted proofs are in appendix A.

2. The infinitesimal structure of games
In contrast to the classical formulation of games, we do not
constrain the parameter sets (e.g. to the probability simplex)
or require losses to be convex in the corresponding players’
parameters. Players could be interacting neural nets such as
GANs (Goodfellow et al., 2014).
Definition 1. A game is a collection of players [n] =
{1, . . . , n} and twice continuously differentiable losses {`i :
d
Rd → R}ni=1 . Parameters
Pn are w = (w1 , . . . , wn ) ∈ R
di
with wi ∈ R where i=1 di = d. .

It is sometimes convenient to write w = (wi , w−i ) where
w−i concatenates the parameters of all the players other than
the ith , which is placed out of order by abuse of notation.
Computing the gradients of the losses with respect to the
relevant parameters yields the simultaneous gradient
d

ξ(w) = (∇w1 `1 , . . . , ∇wn `n ) ∈ R .
By the dynamics of the game, we mean following the negative of the vector field ξ with infinitesimal steps. There is
no reason to expect ξ to be the gradient of a single function
in general, and therefore no reason to expect the dynamics
to converge to a fixed point.
2.1. Warmup: Hamiltonian mechanics in games
The following example illustrates the essential problem with
gradients in games and a key ingredient of our approach.
Example 1 (Conservation of energy
Pnin a zero-sum bimatrix
game). Zero-sum games, where i=1 `i ≡ 0, are a wellstudied class of games. The zero-sum game
`1 (x, y) = x| Ay

and

`2 (x, y) = −x| Ay

H=

 1
1
kuk2 + kvk2 = (x| U| DUx + y| V| DVy) .
2
2

Remarkably, the dynamics can be reformulated via Hamilton’s equations in the coordinates given by the SVD of A:


∂H ∂H
ξ(u, v) =
,−
.
∂v
∂u

The vector field ξ cycles around the equilibrium because
ξ conserves the level sets of the Hamiltonian. To see this,
check that hξ, ∇Hi = 0. However, gradient descent on
the Hamiltonian converges to the Nash equilibrium.
Papadimitriou & Piliouras (2016) recently analyzed the dynamics of Matching Pennies (essentially, the above example)
and showed with some effort that the cyclic behavior covers
the entire parameter space. The Hamiltonian reformulation
directly explains the cyclic behavior via a conservation law.
2.2. The generalized Helmholtz decomposition
The Hessian of a game is the (d × d)-matrix of second
d
α (w)
derivatives H(w) := ∇w · ξ(w)| = ∂ξ∂w
. Conβ
cretely, the Hessian can be written

∇2w1 `1
∇2w1 ,w2 `1
2
 ∇w ,w `2
∇2w2 `2
2
1

H(w) = 
..

.
∇2wn ,w1 `n

∇2wn ,w2 `n

α,β=1

···
···
···


∇2w1 ,wn `1
∇2w2 ,wn `2 

,
..

.
∇2wn `n

where ∇2wi ,wj `k is the (di × dj )-block of 2nd -order derivatives. The Hessian of a game is not necessarily symmetric.
Note: α, β run over dimensions; i, j run over players.
Lemma 1 (generalized Helmholtz decomposition). The
Hessian of any vector field decomposes uniquely into two
components H(w) = S(w) + A(w) where S ≡ S| is
symmetric and A + A| ≡ 0 is antisymmetric.
Proof. Any matrix decomposes uniquely as M = S + A
where S = 12 (M + M| ) and A = 21 (M − M| ) are symmetric and antisymmetric. Applying the decomposition to
the game Hessian yields the result.
The decomposition is preserved by orthogonal change-ofcoordinates P| MP = P| SP + P| AP since the terms
remain symmetric and antisymmetric.

Differentiable Game Mechanics

The connection to the classical Helmholtz decomposition in
calculus is sketched in appendix D. Two obvious classes of
games arise from the decomposition:
Definition 2. A game is a potential game if A(w) ≡ 0. It
is a Hamiltonian game if S(w) ≡ 0.
Potential games are well-studied and easy to solve. Hamiltonian games are a new class of games that are also easy to
solve. The general case is more delicate, see section 3.
2.3. A characterization of local Nash equilibria
Gradient-based methods can reliably find local – but not
global – optima of nonconvex objective functions (Lee et al.,
2016; 2017). Similarly, gradient-based methods cannot be
expected to find (global) Nash equilibria in games.
Definition 3. A point w∗ is a local Nash equilibrium
if, for all i, there is a neighborhood Ui of wi∗ such that
∗
∗
`i (wi0 , w−i
) ≥ `i (wi∗ , w−i
) for wi0 ∈ Ui .

Lemma 2 (Characterization of local Nash equilibria). The
point w is a local Nash equilibrium iff ξ(w) = 0 and S  0
in a neighborhood of w.
Proof. If S is positive semidefinite then so are its (di × di )submatrices Si := ∇2wi `i for all i. The result follows.

they are games where simultaneous gradient descent on the
losses is gradient descent on a single function. It follows
that descent on ξ converges to a fixed point that is a local
minimum of φ – or saddle, but see Lee et al. (2017).
2.5. Hamiltonian games
A concrete example may help understand antisymmetric matrices. Suppose n competitors play one-on-one and that the
probability of player i beating player j is pij . Then, assuming there are no draws, the probabilities
satisfy
 pij + pji = 1

and pii = 12 . The matrix A = log

pij
1−pij

n

i,j=1

of logits

is then antisymmetric. Intuitively, antisymmetry reflects a
hyperadversarial setting where all pairwise interactions between players are zero-sum. In general, Hamiltonian games
are related to – but distinct from – zero-sum games.

Example 2 (an unconstrained2 bimatrix game is zero–
sum iff it is Hamiltonian). Consider bimatrix game with
`1 (x, y) = x| Py and `2 (x, y) = x| Qy. Then ξ =
(Py, Q| x) and the Hessian components have block structure




0
P−Q
0
P+Q
2A =
2S =
(Q − P)|
0
(P + Q)|
0
The game is Hamiltonian iff S = 0 iff P + Q = 0 iff
`1 + `2 = 0.

We refer to a fixed point, with ξ(w) = 0, as stable if
S(w)  0 and unstable if S(w) ≺ 0. Saddle points, neither
stable nor unstable, are beyond the scope of the paper.

There are examples of Hamiltonian games that are not zerosum, and vice versa.

2.4. Potential games

Example 3 (Hamiltonian game that is not zero-sum). Fix
constants a and b and suppose players 1 and 2 minimize
losses

Potential games were introduced in Monderer & Shapley
(1996). A game is a potential game if there is a single
`1 (x, y) = x(y − b) and `2 (x, y) = −(x − a)y
potential function φ : Rd → R and positive numbers {αi >
0}ni=1 such that
with respect to x and y respectively.

 Example 4 (zero-sum game that is not Hamiltonian). Playφ(wi0 , w−i )−φ(wi00 , w−i ) = αi `i (wi0 , w−i )−`i (wi00 , w−i ) ers 1 and 2 minimize
for all i and all and all wi0 , wi00 , w−i . Monderer & Shapley
(1996) show a game is a potential game iff αi ∇wi `i =
∇wi φ for all i, which is equivalent to

|
αi ∇2wi wj `i = αj ∇2wi wj `j = αj ∇2wj wi `j
∀i, j.
It follows that our definition of potential game is the special
case where αi = 1 for all i, which Monderer & Shapley
(1996) refer to as an exact potential game. We use the
shorthand ‘potential game’ to refer to exact potential games
in what follows.

Potential games have been extensively studied since they
are one of the few classes of games for which Nash equilibria can be computed (Rosenthal, 1973). For our purposes,

`1 (x, y) = x2 + y 2

`2 (x, y) = −(x2 + y 2 ).

The game actually has potential function φ(x, y) = x2 − y 2 .
Hamiltonian games are quite different from potential games.
There is a Hamiltonian function H that specifies a conserved quantity. Whereas the dynamics equal ∇φ in potential games; they are orthogonal to ∇H in Hamiltonian
games. The orthogonality implies the conservation law that
underlies the cyclic behavior in example 1.
Theorem 3. Let H(w) := 12 kξ(w)k22 . If the game is Hamiltonian then (i) ∇H = A| ξ and (ii) ξ preserves the level
2
The parameters in bimatrix games are usually constrained to
the probability simplex.

Differentiable Game Mechanics

u

A

B

u
v’

v’

v

(negative)

v

(positive)

w

w

Figure 2. Infinitesimal alignment is positive (cyan) when small
positive λ either: (A) pulls u toward w, if w and u have angle
< 90◦ ; or (B) pushes u away from w if their angle is > 90◦ .

sets of H since hξ, ∇Hi = 0. If the Hessian is invertible
and limkwk→∞ H(w) = ∞ then (iii) gradient descent on
H converges to a local Nash equilibrium.
3

In fact, H is a Hamiltonian function for the game dynamics.
We use the notation H(w) = 12 kξ(w)k2 throughout the
paper. However, H is only a Hamiltonian function for ξ if
the game is Hamiltonian.
There is a precise mapping from Hamiltonian games to symplectic geometry (see Appendix D). Symplectic geometry is
the modern formulation of classical mechanics. Recall that
periodic behaviors (e.g. orbits) often arise in classical mechanics. The orbits lie on the level sets of the Hamiltonian,
which expresses the total energy of the system.

3. Algorithms
Fixed points of potential and Hamiltonian games can be
found by descent on ξ and ∇H respectively. This section
explores approaches to finding stable fixed points in general
games.
3.1. Finding fixed points
If the Hessian H(w) is invertible, then ∇H = H| ξ = 0
iff ξ = 0. Thus, gradient descent on H converges to fixed
points of ξ. Unfortunately, there is no guarantee that descent
on H will find a stable fixed point.

Algorithm 1 Symplectic Gradient Adjustment
n
n
Input:
 losses L = {`i }i=1 , weights W = {wi }i=1 
ξ ← gradient(`i , wi ) for (`i , wi ) ∈ (L, W)
A| ξ ← get sym adj(L, W)
// appendix B
if align then


1
2
∇H ← gradient(
2 kξk , w) for w ∈ W)


1
λ ← sign d1 hξ, ∇HihA| ξ, ∇Hi + 
//  = 10
else
λ←1
end if
// plug into any optimizer
Output: ξ + λ · A| ξ

Note that kξk2 = κ2 (x2 + y 2 ) and

 
x
ξ + λ · H ξ = κ(λκ − 1) ·
.
y
|

Descent on ξ + λ · H| ξ converges to the global maximum
(x, y) = (0, 0) unless λ < κ1 .
Although consensus optimization works well in some special cases; it cannot be considered a candidate algorithm for
finding stable fixed points in general games, since it fails in
the basic case of potential games.
3.2. Finding stable fixed points
There are two classes of games where we know how to find
stable fixed points: potential games where ξ converges to a
local minimum and Hamiltonian games where ∇H, which
is orthogonal to ξ, finds stable fixed points. In the general
case, the following desiderata provide a set of reasonable
properties for an adjustment ξ λ of the game dynamics:
Desiderata. To find stable fixed points, an adjustment ξ λ
to the game dynamics should satisfy
D1. compatible with game dynamics: hξ λ , ξi = α1 · kξk2 ;

Mescheder et al. (2017) propose consensus optimization, a
gradient adjustment of the form

D2. compatible with potential dynamics: if the game is a
potential game then hξ λ , ∇φi = α2 · k∇φk2 ;

ξ + λ · H| ξ = ξ + λ · ∇H.

D3. compatible with Hamiltonian dynamics: If the game is
Hamiltonian then hξ λ , ∇Hi = α3 · k∇Hk2 ;

Unfortunately, consensus optimization can converge to unstable fixed points even in simple cases where the ‘game’ is
to minimize a single function:
Example 5 (consensus optimization converges to global
maximum). Consider a potential game with losses
`1 (x, y) = `2 (x, y) = − κ2 (x2 + y 2 ) with κ  0. Then
 


x
κ 0
ξ = −κ ·
and H = −
y
0 κ
3

Notation: H for Hamiltonian; H for Hessian.

D4. attracted to local Nash equilibria: in neighborhoods
where S  0, require θ(ξ λ , ∇H) ≤ θ(ξ, ∇H);
D5. repelled by unstable equilibria: in neighborhoods
where S ≺ 0, require θ(ξ λ , ∇H) ≥ θ(ξ, ∇H);
for some α1 , α2 , α3 > 0.
Note that desideratum D1 does not guarantee that players act in their own self-interest – this requires a stronger

Differentiable Game Mechanics
learning rate

0.01

0.032

0.1

GRADIENT DESCENT

positivity condition on dot-products with subvectors of ξ,
see Balduzzi (2017). Desiderata D4 and D5 are explained
in section 3.4. The unadjusted dynamics ξ satisfies all the
desiderata except D3. Consensus optimization only satisfies
D3 and D4. Ideally, desideratum D5 would be strengthened to ‘repelled by saddle points where ξ(w) = 0 but
S(w) 6 0’.
Proposition 4. The symplectic gradient adjustment (SGA)

SGA

3.3. Symplectic gradient adjustment
ξ λ := ξ + λ · A| ξ.
satisfies D1–D3 for λ > 0, with α1 = 1 = α2 and α3 = λ.
Note that desiderata D1 and D2 are true even when λ < 0.
This will prove useful, since example 6 and theorem 5 show
it is necessary pick negative λ near S ≺ 0. Section 3.4
shows how to also satisfy desiderata D4 and D5.
Proof. First claim: λ · ξ | A| ξ = 0 by skew-symmetry of A.
Second claim: A ≡ 0 in a potential game, so ξ λ = ξ = ∇φ.
Third claim: hξ λ , ∇Hi = hξ λ , H| ξi = hξ λ , A| ξi = λ ·
ξ | AA| ξ = λ · k∇Hk2 since H = A by assumption and
since ξ | A| ξ = 0 by antisymmetry.
In the example below, almost any choice of positive λ results
in convergence to an unstable equilibrium. The problem
arises from the combination of a weak repellor with a strong
rotational force. The next section shows how to pick λ to
avoid unstable equilibria.
Example 6 (failure case for λ > 0). Suppose  > 0 is small
and


f (x, y) = − x2 − xy and g(x, y) = − y 2 + xy
2
2
with an unstable equilibrium at (0, 0). The dynamics are
   


−x
−y
0 −1
ξ =·
+
with A =
−y
x
1 0
and

 
 
x
−y
|
A ξ=
+
y
x

Finally observe that

 
 
x
−y
ξ + λ · A ξ = (λ − ) ·
+ (1 + λ) ·
y
x
|

which converges to the unstable equilibrium if λ > .
Lemma 8 in the appendix shows that, if S and A commute
and S  0, then hξ λ , ∇Hi ≥ 0 for λ > 0. The proof
of theorem 5 introduces the additive condition number to
upper-bound the worst-case noncommutativity of S, which
allows to quantify the relationship between ξ λ and ∇H.

Figure 3. SGA allows faster and more robust convergence to stable
fixed points. Note the scale of top-right panel differs from rest.

Theorem 5. Let S be a symmetric matrix with eigenvalues
σmax ≥ · · · ≥ σmin . The additive condition number4 of S is
κ = σmax −σmin . Suppose that S  0 is positive semidefinite
with additive condition number κ. Then
hξ λ , ∇Hi ≥ 0

for

0<λ<

4
.
κ

If S is negative semidefinite, then for λ in the same range
hξ −λ , ∇Hi ≤ 0.
The inequalities are strict if H is invertible.
If κ = 0, then S = σ · I commutes with all matrices. The
larger κ, the larger the potential failure of S to commute
with other matrices. The analysis is pessimistic: S and A
may approximately commute for large κ (e.g. if A ≈ 0).
3.4. How to pick sign(λ)
This section explains desiderata D4–D5 and shows how to
pick sign(λ) to speed up convergence towards stable and
away from unstable fixed points.
First, observe that hξ, ∇Hi = ξ | (S + A)| ξ = ξ | Sξ. It
follows that for ξ 6= 0:
(
if S  0 then hξ, ∇Hi ≥ 0;
(1)
if S ≺ 0 then hξ, ∇Hi < 0.
A criterion to probe the positive/negative definiteness of S
is thus to check the sign of hξ, ∇Hi. The dot product can
take any value if S is not positive nor negative definite. The
behavior near saddles is beyond the scope of the paper.
Desiderata D4 can be interpreted as follows. If ξ points at
a stable equilibrium then we require that ξ λ points more
4

The condition number of a positive definite matrix is

σmax
.
σmin

Definition 4. Let u and v be two vectors. The infinitesimal
alignment of ξ λ := u + λ · v with a third vector w is
align(ξ λ , w) :=

d  2
cos θλ
dλ

LEARNING RATE

|λ=0

for θλ := θ(ξ λ , w).

If u and w point the same way, u| w > 0, then align > 0
when v bends u further toward w, see figure 2A. Otherwise
align > 0 when v bends u away from w, figure 2B.
Proposition 6. Desiderata D4–5 are satisfied for λ such
that λ · hξ, ∇Hi · hA| ξ, ∇Hi ≥ 0.
Computing the sign of hξ, ∇Hi provides a check for stable and unstable fixed points. Computing the sign of
hA| ξ, ∇Hi checks whether the adjustment term points towards or away from the (nearby) fixed point. Putting the
two checks together yields a prescription for the sign of λ.
Alignment and convergence rates. Finally, we show that
increasing alignment helps speed convergence. Gradient
descent is also known as the method of steepest descent. In
general games, however, ξ does not follow the steepest path
to fixed points due to the ‘rotational force’, which forces
lower learning rates and slows down convergence:
Theorem 7. Suppose f is convex and Lipschitz smooth with
k∇f (x) − ∇f (y)k ≤ L · kx − yk. Let wt+1 = wt − η · v
where kvk = k∇f (wt )k. Then the optimal step size is
θ
η ∗ = cos
L where θ := θ(∇f (wt ), v), with
f (wt+1 ) ≤ f (wt ) −

OMD
SGA

LOSS AFTER 250 STEPS

towards the equilibrium (i.e. has smaller angle). Conversely
if ξ points away then the adjustment should point further
away. More formally,

STEPS TO CONVERGE

Differentiable Game Mechanics

cos2 θ
· k∇f (wt )k2 .
2L

Increasing the cosine with the steepest direction improves
convergence. The alignment computation in algorithm 1
chooses λ to be positive or negative such that ξ λ is bent
towards stable fixed points (increasing the cosine) and away
from unstable fixed points. Adding a small  > 0 to the
computation introduces a weak bias towards attractors.
3.5. Aligned consensus optimization
The stability criterion in (1) also provides a simple way to
prevent consensus optimization from converging to unstable
equilibria. Aligned consensus optimization is


ξ + |λ| · sign hξ, ∇Hi · H| ξ,
(2)

where in practice we set λ = 1. Aligned consensus satisfies
desiderata D3–D5. However, it behaves strangely in potential games. Multiplying by the Hessian is the ‘inverse’ of
Newton’s method: it increases the gap between small and

OMD
SGA

LEARNING RATE

Figure 4. Comparison of SGA with optimistic mirror descent.
Sweep over learning rates in range [0.01, 1.75]; λ = 1 throughout.
(A) iterations to convergence (up to 250). (B): average absolute
value of losses over iterations 240-250, cutoff at 5.

large eigenvalues, increasing the (usual, multiplicative) condition number and slows down convergence. Nevertheless,
consensus optimization works well in GANs (Mescheder
et al., 2017), and aligned consensus may improve performance, see figure 9 in appendix.
Finally, note that dropping the first term ξ from (2) yields
a simpler update that also satisfies D3–D5. However, the
resulting algorithm performs poorly in experiments (not
shown), perhaps because it is attracted to saddles.

4. Experiments
We compare SGA with simultaneous gradient descent, optimistic mirror descent (Daskalakis et al., 2018) and consensus optimization (Mescheder et al., 2017) in basic settings.
4.1. Learning rates and alignment
We investigate the effect of the SGA when a weak attractor
is coupled to a strong rotational force:
`1 (x, y) =

1 2
x + 10xy
2

and

`2 (x, y) =

1 2
y − 10xy
2

Gradient descent is extremely sensitive to the choice of
learning rate η, top row of figure 3. As η increases through
{0.01, 0.032, 0.1} gradient descent goes from converging
extremely slowly, to diverging slowly, to diverging rapidly.
SGA yields faster, more robust convergence. SGA converges faster with learning rates η = 0.01 and η = 0.032,
and only starts overshooting the fixed point for η = 0.1.
4.2. Basic adversarial games
Figure 4 compares SGA with optimistic mirror descent on
a zero-sum bimatrix game with `1/2 (w1 , w2 ) = ±w1| w2 .
The example is modified from Daskalakis et al. (2018) who
also consider a linear offset that makes no difference. A run
converges, panel A, if the average absolute value of losses
on the last 10 iterations is < 0.01.
Although OMD’s peak performance is better than SGA, we
find that SGA converges – and does so faster – for a wider
range of learning rates. OMD diverges for learning rates

Differentiable Game Mechanics
4000

6000

8000

SGA without ALIGNMENT

learning rate 9e-5
learning rate 9e-5

learning rate 1e-4

2000

SGA with ALIGNMENT

GRADIENT DESCENT

Iteration:

Figure 5. Top: Simultaneous gradient descent suffers from mode collapse and in later iterations (not shown) mode hopping. Middle:
vanilla SGA converges smoothly to the ground truth (figure 8 in appendix). Bottom: SGA with alignment converges slightly faster.

not in the range [0.3, 1.2]. Simultaneous gradient descent
oscillates without converging (not shown). Individual runs
are shown in the appendix. The appendix also compares the
performance of the algorithms on four-player games.

aligned SGA converges to the correct distribution; alignment
speeds up convergence slightly. Consensus optimization performs similarly, see figure 9 in the appendix. However, it
can converge to local maxima, recall example 5.

4.3. Generative adversarial networks

5. Discussion

We apply SGA to a basic setup adapted from Metz et al.
(2017). Data is sampled from a highly multimodal distribution designed to probe the tendency to collapse onto a subset
of modes during training. The distribution is a mixture of
16 Gaussians arranged in a 4 × 4 grid, see ground truth in
figure 8. The generator and discriminator networks both
have 6 ReLU layers of 384 neurons. The generator has two
output neurons; the discriminator has one.

Modern deep learning treats differentiable modules like
plug-and-play lego blocks. For this to work, at the very
least, we need to know that gradient descent will find local
minima. Unfortunately, gradient descent does not necessarily find local minima when optimizing multiple interacting
objectives. With the recent proliferation of algorithms that
optimize more than one loss, it is becoming increasingly
urgent to understand and control the dynamics of interacting
losses. Although there is interesting recent work on twoplayer adversarial games such as GANs, there is essentially
no work on finding stable fixed points in general.

Figure 5 shows results after {2000, 4000, 6000, 8000} iterations. The networks are trained under RMSProp. Learning
rates were chosen by visual inspection of grid search results at iteration 8000, see appendix. Simultaneous gradient
descent and SGA are shown in the figure. Results for consensus optimization are in the appendix.
Simultaneous gradient descent exhibits mode collapse followed by mode hopping in later iterations (not shown).
Mode hopping is analogous to the cycles in example 1. Un-

The generalized Helmholtz decomposition provides a powerful new perspective on game dynamics. A key feature
is that the analysis is indifferent to the number of players.
Instead, it is the interplay between the simultaneous gradient ξ on the losses and the symmetric and antisymmetric
matrices of second-order terms that guides algorithm design
and governs the dynamics under gradient adjustments.

Differentiable Game Mechanics

Symplectic gradient adjustment is a straightforward application of the Helmholtz decomposition. It is unlikely that
SGA is the only or best approach to finding stable fixed
points. A deeper understanding of the interaction between
the potential and Hamiltonian components will lead to more
effective algorithms. Of particular interest are pure firstorder methods that do not use Hessian-vector products.
Finally, it is worth raising a philosophical point. The goal in
this paper is to find stable fixed points (e.g. because in GANs
they yield pleasing samples). We are not concerned with
the losses of the players per se. The gradient adjustments
may lead to a player acting against its own self-interest by
increasing its loss. We consider this acceptable insofar as it
encourages convergence to a local Nash equilibrium.
Acknowledgements. We thank Guillame Desjardins and
Csaba Szepesvari for useful feedback.

Hart, Sergiu and Mas-Colell, Andreu. Simple Adaptive
Strategies: From Regret-Matching to Uncoupled Dynamics. World Scientific, 2013.
Heusel, M, Ramsauer, H, Unterthiner, T, Nessler, B, Klambauer, G, and Hochreiter, S. GANs Trained by a Two
Time-Scale Update Rule Converge to a Nash Equilibrium.
In NIPS, 2017.
Hu, Junling and Wellman, Michael P. Nash Q-Learning
for General-Sum Stochastic Games. Journal of Machine
Learning Research, 4:1039–1069, 2003.
Jaderberg, M, Czarnecki, W M, Osindero, S, Vinyals, O,
Graves, A, and Kavukcuoglu, K. Decoupled Neural Interfaces using Synthetic Gradients. In ICML, 2017.

References

Lee, Jason D, Simchowitz, Max, Jordan, Michael I, and
Recht, Benjamin. Gradient Descent Converges to Minimizers. In COLT, 2016.

Abdallah, Sherief and Lesser, Victor R. A multiagent reinforcement learning algorithm with non-linear dynamics.
J. Artif. Intell. Res., 33:521–549, 2008.

Lee, JD, Panageas, I, Piliouras, G, Simchowitz, M, Jordan,
MI, and Recht, B. First-order Methods Almost Always
Avoid Saddle Points. In arXiv:1710.07406, 2017.

Balduzzi, D. Strongly-Typed Agents are Guaranteed to
Interact Safely. In ICML, 2017.

Liu, Bo, Liu, Ji, Ghavamzadeh, Mohammad, Mahadevan,
Sridhar, and Petrik, Marek. Proximal Gradient Temporal
Difference Learning Algorithms. In IJCAI, 2016.

Bottacin, Francesco. A Marsden-Weinstein Reduction Theorem for Presymplectic Manifolds. 2005.
Bowling, Michael and Veloso, Manuela. Multiagent learning using a variable learning rate. Artificial Intelligence,
136:215–250, 2002.
Bowling, Michael H. Convergence and no-regret in multiagent learning. In NIPS, pp. 209–216, 2004.
Daskalakis, C, Goldberg, P W, and Papadimitriou, C. The
Complexity of Computing a Nash Equilibrium. SIAM J.
Computing, 39(1):195–259, 2009.

Lu, Xiaoyun. Hamiltonian games. Journal of Combinatorial
Theory, Series B, 55:18–32, 1992.
Mertikopoulos, Panayotis, Papadimitriou, Christos, and Piliouras, Georgios. Cycles in adversarial regularized learning. In SODA, 2018.
Mescheder, Lars. On the convergence properties of GAN
training. In ArXiv:1801:04406, 2018.
Mescheder, Lars, Nowozin, Sebastian, and Geiger, Andreas.
The Numerics of GANs. In NIPS. 2017.

Daskalakis, C, Ilyas, A, Syrgkanis, V, and Zeng, H. Training
GANs with Optimism. In ICLR, 2018.

Metz, L, Poole, B, Pfau, D, and Sohl-Dickstein, J. Unrolled
generative adversarial networks. In ICLR, 2017.

Dauphin, Y, Pascanu, R, Gulcehre, C, Cho, K, Ganguli, S,
and Bengio, Y. Identifying and attacking the saddle point
problem in high-dimensional non-convex optimization.
In NIPS, 2014.

Monderer, Dov and Shapley, Lloyd S. Potential Games.
Games and Economic Behavior, 14:124–143, 1996.

Foerster, J N, Chen, R Y, Al-Shedivat, M, Whiteson, S,
Abbeel, P, and Mordatch, I. Learning with OpponentLearning Awareness. In AAMAS, 2018.
Goodfellow, I J, Pouget-Abadie, J, Mirza, M, Xu, B, WardeFarley, D, Ozair, S, Courville, A, and Bengio, Y. Generative Adversarial Nets. In NIPS, 2014.

Nagarajan, Vaishnavh and Kolter, J Zico. Gradient descent
GAN optimization is locally stable. In NIPS, 2017.
Nesterov, Yurii. Introductory Lectures on Convex Optimization: A Basic Course. Kluwer, 2004.
Papadimitriou, Christos and Piliouras, Georgios. From Nash
Equilibria to Chain Recurrent Sets: Solution Concepts
and Topology. In ITCS, 2016.

Differentiable Game Mechanics

Pathak, D, Agrawal, P, Efros, A A, and Darrell, T. Curiositydriven Exploration by Self-supervised Prediction. In
ICML, 2017.
Pfau, David and Vinyals, Oriol. Connecting Generative
Adversarial Networks and Actor-Critic Methods. In
arXiv:1610.01945, 2016.
Racanière, S, Weber, T, Reichert, D P, Buesing, L, Guez,
A, Rezende, D J, Badia, A P, Vinyals, O, Heess, N, Li,
Y, Pascanu, R, Battaglia, P, Hassabis, D, Silver, D, and
Wierstra, D. Imagination-Augmented Agents for Deep
Reinforcement Learning. In NIPS, 2017.
Rosenthal, R W. A Class of Games Possessing Pure-Strategy
Nash Equilibria. Int J Game Theory, 2:65–67, 1973.
Salimans, Tim, Goodfellow, Ian, Zaremba, Wojciech, Cheung, Vicki, Radford, Alec, and Chen, Xi. Improved
Techniques for Training GANs. In NIPS, 2016.
Shoham, Yoav and Leyton-Brown, Kevin. Multiagent Systems: Algorithmic, Game-Theoretic, and Logical Foundations. Cambridge University Press, 2008.
Singh, S, Kearns, M, and Mansour, Y. Nash Convergence
of Gradient Dynamics in General-Sum Games. In UAI,
2000.
Stoltz, G and Lugosi, G. Learning correlated equilibria
in games with compact sets of strategies. Games and
Economic Behavior, 59:187–208, 2007.
Vezhnevets, A, Osindero, S, Schaul, T, Heess, N, Jaderberg,
M, Silver, D, and Kavukcuoglu, K. FeUdal Networks for
Hierarchical Reinforcement Learning. In ICML, 2017.
Wayne, G and Abbott, L F. Hierarchical Control Using
Networks Trained with Higher-Level Forward Models.
Neural Computation, (26), 2014.

APPENDIX
A. Proofs
Proof of theorem 3.
Proof. Direct computation shows ∇H = H| ξ for any
game. The first statement follows since H = A in Hamiltonian games. For the second statement, the directional
derivative is Dξ H = hξ, ∇Hi = ξ | A| ξ where ξ | A| ξ =
(ξ | A| ξ)| = ξ | Aξ = −(ξ | A| ξ) since A = −A| by
skew-symmetry. It follows that ξ | A| ξ = 0.
For the third, gradient descent on H will converge to a point
where ∇H = H| ξ(w) = 0. If the Hessian is invertible
then clearly ξ(w) = 0. The fixed-point is a local Nash

equilibrium by Lemma 2 since 0 ≡ S  0 in a Hamiltonian
game.
Proof of theorem 5. The following lemma, whilst not
used in the proof of theorem 5, nevertheless provides a
useful intuition about the role of commutativity.
Lemma 8. If S  0 is symmetric positive semidefinite and
S commutes with A then
hξ λ , ∇Hi ≥ 0 for all λ ≥ 0.
Proof. First observe that ξ | ASξ = ξ | S| A| ξ =
−ξ | SAξ, where the first equality holds since the expression is a scalar, and the second holds since S = S| and
A = −A| . It follows that ξ | ASξ = 0 if SA = AS.
Rewrite the inequality as

hξ λ · A| ξ, ∇Hi = hξ + λ · A| ξ, Sξ + A| ξi
= ξ | Sξ + λξ | AA| ξ ≥ 0

since ξ | ASξ = 0 and by positivity of S, λ and AA| .
The proof of the theorem follows.
Proof. We prove the case S  0; the case S  0 is similar.
Rewrite the inequality as
hξ + λ · A| ξ, ∇Hi = (ξ + λ · A| ξ)| · (S + A| )ξ

= ξ | Sξ + λξ | ASξ + λξ | AA| ξ

Let β = kA| ξk and S̃ = S − σmin · I, where I is the identity
matrix. Then
ξ | Sξ + λξ | ASξ + λ · β 2 ≥ ξ | S̃ξ + λξ | AS̃ξ + λ · β 2
since ξ | Sξ ≥ ξ | S̃ξ by construction and ξ | AS̃ξ =
ξ | ASξ − σmin ξ | Aξ = ξ | ASξ because ξ | Aξ = 0 by
the skew-symmetry of A. It therefore suffices to show that
the inequality holds when σmin = 0 and κ = σmax .
Since S is positive semidefinite, there exists an uppertriangular square-root matrix T such that T| T = S and so
ξ | Sξ = kTξk2 . Further,
|ξ | ASξ| ≤ kA| ξk · kT| Tξk ≤
since kTk2 =
obtains

√

√

σmax · kA| ξk · kTξk.

σmax . Putting the observations together

kTξk2 + λ(kAξk2 − hAξ, Sξi)

≥kTξk2 + λ(kAξk2 − kAξk kSξk

≥kTξk2 + λkAξk(kAξk − kSξk)
√
≥kTξk2 + λkAξk(kAξk − σmax kTξk)

Differentiable Game Mechanics

√
√
Set α = λ and η = σmax . We can continue the above
computation
=kTξk2 + α2 kAξk(kAξk − ηkTξk)

=kTξk2 + α2 kAξk2 − α2 kAξkηkTξk

=(kTξk − αkAξk)2 + 2αkAξk kTξk − α2 ηkAξk kTξk
=(kTξk − αkAξk)2 + kAξk kTξk(2α − α2 η)

A

B

w

w

Figure 6. Alignment and learning rates. The larger cos θ, the larger
the learning rate η that can be applied to unit vector ξ without
w + η · ξ leaving the unit circle.

Finally, 2α − α2 η > 0 for any α in the range (0, η2 ), which
4
is to say, for any 0 < λ < σmax
. The kernel of S and the
kernel of T coincide. If ξ is in the kernel of A, resp. T, it
cannot be in the kernel of T, resp. A and the term (kTξk −
αkAξk)2 is positive. Otherwise, the term kAξkkTξk is
positive.

Proof of theorem 7. The following lemma provides some
intuition for theorem 7. The idea is that, the smaller the
cosine between the ‘correct direction’ w and the ‘update
direction’ ξ, the smaller the learning rate needs to be for the
update to stay in a unit ball, see figure 6.

Proof of proposition 6.

Lemma 10 (alignment lemma). If w and ξ are unit vectors
with 0 < w| ξ then kw − η · ξk ≤ 1 for 0 ≤ η ≤ 2w| ξ.

Lemma 9. When ξ λ is the symplectic gradient adjustment,




sign align(ξ λ , ∇H) = sign hξ, ∇Hi · hA| ξ, ∇Hi .
Proof. Observe that


2
hξ λ , ∇Hi
cos θλ =
kξ λ k · k∇Hk
hξ, ∇Hi + 2λhξ, ∇HihA| ξ, ∇Hi + O(λ2 )

=
kξk2 + O(λ2 ) · k∇Hk2
2

where the denominator has no linear term in λ because
ξ ⊥ A| ξ. It follows that the sign of the infinitesimal
alignment is
sign



d
cos2 θλ
dλ



n
o
= sign hξ, ∇HihA| ξ, ∇Hi

as required.
The proposition follows easily.
Proof. If we are in a neighborhood of a stable fixed
point then hξ, ∇Hi ≥ 0. It follows
by lemma

 9 that
sign align(ξ λ ), ∇H) = sign hA| ξ, ∇Hi and so


choosing sign(λ) = sign hA| ξ, ∇Hi leads to the angle
between ξ λ and ∇H being smaller than the angle between
ξ and ∇H, satisfying desideratum D4.

The proof for the unstable case is similar.

Proof. Check kw − η · ξk2 = 1 + η 2 − 2η · w| ξ ≤ 1 iff
η 2 ≤ 2η · w| ξ. The result follows.
The next lemma is a standard technical result (e.g. Nesterov,
2004).
Lemma 11. Let f : Rd → R be a convex Lipschitz smooth
function satisfying k∇f (y) − ∇f (x)k ≤ L · ky − xk for
all x, y ∈ Rd . Then
|f (y) − f (x) − h∇f (x), y − xi| ≤

L
· ky − xk2
2

for all x, y ∈ Rd .
Finally the proof of the theorem is adapted from a standard
result in Nesterov (2004), where there is no cosine.
Proof. By the above lemma,
f (y) ≤ f (x) + h∇f (x), y − xi +

L
ky − xk2
2

L
· kξk2
2
L
= f (x) − η · h∇f, ξi + η 2 · k∇f k2
2
η
= f (x) − η(α − L) · k∇f k2
2
= f (x) − η · h∇f, ξi + η 2

where α := cos θ. Solve
n
η o
min ∆(η) = min −η(α − L)
η
η
2
to obtain η ∗ =

α
L

2

and ∆(η ∗ ) = − α2 L as required.

OMD
SGA

OMD
SGA

STEPS TO CONVERGE

STEPS TO CONVERGE

Differentiable Game Mechanics

WEAK
ATTRACTOR

LEARNING RATE

NO
ATTRACTOR

LEARNING RATE

Figure 7. Time to convergence of OMD and SGA on two 4player games. Times are cutoff after 5000 iterations. Left panel:
Weak attractor corresponding to potential function φ(w, x, y, z) =
1
(w2 + x2 + y 2 + z 2 ). Right panel: No potential function.
200

B. TensorFlow code to compute A| ξ:
The code requires a list of n losses, Ls, and a list of variables
for the n players, xs. The function fwd gradients is in
the module tf.contrib.kfac.utils.
def jac vec(ys, xs, vs) :
return fwd gradients(ys, xs,
grad xs = vs, stop gradients = xs)
def jac tran vec(ys, xs, vs) :
dydxs = tf.gradients(ys, xs, grad ys = vs,
stop gradients = xs)
return [tf.zeros like(x) if dydx is None
else dydx
for (x, dydx) in zip(xs, dydxs)]
def get sym adj(Ls, xs) :
xi = [tf.gradients(`, x)[0]
for (`, x) in zip(Ls, xs)]
H xi = jac vec(xi, xs, xi)
Ht xi = jac tran vec(xi, xs, xi)
At xi = [ ht−h
for (h, ht) in zip(H xi, Ht xi)]
2
return At xi

C. Further experiments
OMD and SGA on a four-player game. Figure 7 shows
time to convergence (same criterion as section 4.2 for optimistic mirror descent and SGA. The games are constructed
with four players, each of which controls one parameter.
The game was setup to have antisymmetric and symmetric
components of the Hessian as follows. The antisymmetric
component of the game Hessian is


0
−1
A=
−1
−1

1
0
−1
−1



1 1
1 1

0 1
−1 0

Figure 8. Ground truth for GAN experiments. A mixture of 16
Gaussians.

in both panels. The symmetric component is

1
1 
0
·
S=
100 0
0

0
1
0
0

0
0
1
0


0
0

0
1

in the left panel and S ≡ 0 in the right panel.

OMD converges considerably slower than SGA across the
full range of learning rates. It also diverges for learning
rates > 0.22. SGA converges more quickly and robustly.
Ground truth for GAN experiments. Figure 8 shows
the probability distribution that is sampled to train the generator and discriminator in the GAN example.
More details on GAN experiments. Figure 9 shows the
performance of consensus optimization without and with
alignment. Introducing alignment slightly improves speed
of convergence (second column) and final result (fourth
column), although intermediate results in third column are
ambiguous.

Grid search was over learning rates {1e-5, 2e-5,5e-5, 8e-5,
1e-4, 2e-4, 5e-4} and then a more refined linear search over
[8e-5, 2e-4].

D. Helmholtz and Hamilton
This section explains the mathematical connections with the
Helmholtz decomposition and symplectic geometry. The
discussion is not necessary to understand the main text.
It is also not self-contained. The details can be found in
textbooks covering differential and symplectic geometry.

Differentiable Game Mechanics
2000

4000

6000

8000

learning rate 9.25e-5

CONSENSUS OPTIMIZATION
with ALIGNMENT

learning rate 9e-5

CONSENSUS OPTIMIZATION

Iteration:

Figure 9. Top: Consensus optimization on GAN example. Bottom: Consensus optimization with alignment.

D.1. The Helmholtz decomposition

D.2. Hamiltonian mechanics

The classical Helmholtz decomposition states that any vector field ξ in 3-dimensions is the sum of curl-free (gradient)
and divergence-free (infinitesimal rotation) components:

A symplectic form ω is a closed nondegenerate differential
2-form. Given a manifold with a symplectic form, a vector
field ξ is Hamiltonian vector field if there exists a function
H : M → R satisfying

ξ=

∇φ
|{z}

gradient component

+

curl(B)
| {z }

rotational component

h

curl(•) := ∇×(•)

We explain the link between curl and the antisymmetric
component of the game Hessian. Recall that gradients of
functions are actually differential 1-forms, not vector fields.
Differential 1-forms and vector fields on a manifold are
canonically isomorphic once a Riemannian metric has been
chosen. In our case, we are implicitly using the Euclidean
metric. The antisymmetric matrix A is the differential 2form obtained by applying the exterior derivative d to the
1-form ξ.
In 3-dimensions, the Hodge star operator is an isormorphism
from differential 2-forms to vector fields, and the curl can
be reformulated as curl(•) = ∗d(•). In claiming A is
analogous to curl, we are simply dropping the Hodge-star
operator.
Finally, recall that the Lie algebra of infinitesimal rotations
in d-dimensions is given by antisymmetric matrices. When
d = 3, the Lie algebra can be represented as vectors (three
numbers specify a 3 × 3 antisymmetric matrix) with the
×-product as Lie bracket. In general, the antisymmetric
matrix A captures the infinitesimal tendency of ξ to rotate
at each point in the parameter space.

i

ω(ξ, •) = dH(•) = h∇H, •i.

(3)

The function is then referred to as the Hamiltonian function
of the vector field. In our case, the antisymmetric matrix A
is a closed 2-form because A = dξ and d ◦ d = 0. It may
however be degenerate. It is therefore a presymplectic form
(Bottacin, 2005).
Setting ω = A, equation (3) can be rewritten in our notation
as
ω(ξ, •) = dH(•),
| {z } | {z }
A| ξ

∇H

justifying the terminology ‘Hamiltonian’ and ‘symplectic’.

E. Type consistency
The next two sections carefully work through the units in
classical mechanics and two-player games respectively. The
third section briefly describes a use-case for type consistency.
E.1. Units in classical mechanics
Consider the well-known Hamiltonian


1
1
H(p, q) =
κ · q 2 + · p2
2
µ

Differentiable Game Mechanics

where q is position, p = µ · q̇ is momentum, µ is mass, κ is
surface tension and H measures energy. The units (denoted
by τ ) are
τ (q) = m
τ (κ) =

τ (p) =

kg
s2

kg·m
s

τ (µ) = kg

where m is meters, kg is kilograms and s is seconds. Energy
is measured in joules, and indeed it is easy to check that
2
τ (H) = kg·m
s2 .
∂
1
Note that the units for differentation by x are τ ( ∂x
.
) = τ (x)
1
For example, differentiating by time has units s . Hamilton’s
∂H
1
equations state that q̇ = ∂H
∂p = µ · p and ṗ = − ∂q = −κ · q
where
τ (q̇) = m
τ (ṗ) = kg·m
s
s2

τ



∂
∂q



=

1
m

τ



∂
∂p



=

s
kg·m

The resulting flow describing the dynamics of the system is

kg · m2
.
s

Observe that contracting the flow with the Hamiltonian
obtains
ιξ ω = ω(ξ, •) = dH =
with units τ (dH) = τ (H) =

2

`2 (q, p) = −κ · qp
2

2

∂H ∂
∂H ∂
∂`1 ∂
∂`2 ∂
−
=
+
.
∂p ∂q
∂q ∂p
∂q ∂q
∂p ∂p

∂`2
· dp
∂p

(4)



= τ (`2 ) =

kg 2 · m2
s3

and we cannot add objects with different types.
To correct the type inconsistency, define the musical isomorphism as


∂
∂q

[

=

µ
· dq
2

and



2 ∂
·
µ ∂q

and

(dp) = 2κ ·

∂
∂p

[

=

1
· dp
2κ

with inverse
]

with units τ (`1 ) = ms and τ (`2 ) = kg s·m
. The Hamilto3
nian dynamics can then be recovered game-theoretically by
differentiating `1 and `2 with respect to q and p respectively.
It is easy to check that
ξ=



(dq) =

kg·m2
s2 .

1
· qp and
µ

∂`1
∂`2
· dq +
· dp
∂q
∂p

results in a type violation because


m2
∂`1
· dq = τ (`1 ) =
τ
∂q
s

∂H
∂H
· dq +
· dp
∂q
∂p

Losses in classical mechanics. Although there is no notion of “loss” in classical mechanics, it is useful (for the
next section) to keep pushing the formal analogy. Define
the “losses”
`1 (q, p) =

ξ[ =

τ

with units τ (ξ) = 1s . Hamilton’s equations can be reformulated more abstractly via symplectic geometry. Introduce
the symplectic form
with units τ (ω) =

It is instructive to compute ξ [ in the case of a classical
mechanical system and see what happens. Naively, we
 [
∂
would guess that the musical isomorphism is ∂q
= dq
 [
∂
and ∂p
= dp. However, applying the naive musical
isomorphism to ξ to get

whereas

∂
∂
1
∂
∂
ξ = q̇ ·
+ ṗ ·
= p·
− κq ·
∂q
∂p
µ ∂q
∂p

ω = dq ∧ dp

The duality between vector fields and differential forms.
Finally recall that the symplectic form in games was not
“pulled out of thin air” as ω = dq ∧ dp, but rather derived as
ω = dξ [ , where ξ [ is the differential form corresponding to
the vector field ξ under the musical isomorphism [ : T M →
T ∗M .

]

∂
.
∂p

The correction terms in the direction [ : T M → T ∗ M
invert the coupling terms κ and µ1 that were originally introduced into the Hamiltonian for physical reasons. Applying
the corrected musical isomorphism to ξ yields
ξ[ =

µ ∂f
1 ∂g
1
·
· dq +
·
· dp = (p · dq − q · dp) .
2 ∂q
2κ ∂p
2

The two terms of ξ [ then have coherent types


2
1
τ ∂`
·
µ
·
dq
=m
· kg · m = kg·m
∂q
s
s


2
kg·m s2
kg·m
1
2
τ ∂`
= kg·m
∂p · κ · dp = s2 · kg ·
s
s

as required. The associated two form is


∂2f
1 ∂2g
ω := dξ [ = − µ ·
− ·
dq∧dp = −dq∧dp
∂q∂p κ ∂q∂p

Differentiable Game Mechanics

which recovers the symplectic form (up to sign) with units
2
τ (ω) = kg·m
as required. Finally, observe that
s


1 p ∂
∂
[
hξ, ξ i =
·
− κq ·
, p · dq − q · dp
2 µ ∂q
∂p


1
1
=
κ · q 2 + · p2 = H(p, q)
2
µ
recovering the Hamiltonian.
E.2. Units in two-player games
Without loss of generality let w = (x; y) where we refer
to x as position and y as momentum so that τ (x) = m
and τ (y) = kg·m
s . The aim of this section is to check typeconsistency under these, rather arbitrarily assigned, units.
Since we are considering a game, we do not require that x
and y have the same dimension – even though this would
necessarily be the case for a physical system. The goal is to
verify that units can be consistently assigned to games.
Consider a quadratic two player game of the form

 
 
 b1
1 | |  A11 A12
x
x y
`1 (w) =
+ x| y |
A21 A22
y
b2
2
and

`2 (w) =



1 | |  C11
x y
C21
2

 
 
 d1
C12
x
|
|
+ x y
C22
y
d2

We restrict to quadratic games since our methods only involve first and second derivatives. We assume the matrices
A and C are symmetric without loss of generality so that,
for example, A12 = A|21 . Adding constant terms to `1 and
`2 makes no difference to the analysis so they are omitted.
2

2

By (4), the units for `1 and `2 should be ms and kg·m
s3
respectively. We can therefore derive the correct units for
each of the components of the quadratic losses as
1

m
1 
m
s
kg
s



 + m kg·m  
m kg·m 
s

1
kg

|

|

{z

kg·m
s

w| Aw

for `1 and
m

s
kg 2


kg·m
s

 kg2


s3

kg
s2

kg
s2

1
s

{z

w| Cw




m
kg·m
s

s

|

}



+ m
}

|

{z

m
kg

w| b


kg·m
s

}

{z

w| d

s3

kg·m
s2

ξ[ =

∂`1
∂`2
· dx +
· dy
∂x
∂y

1
which is type inconsistent because τ ( ∂`
∂x · dx) =
2
τ ( ∂`
∂y · dy) =

2

2

kg ·m
s3



}

m2
s

and

.

Type-consistency via SVD. It is necessary, as in section E.1, to correct the naive musical isomorphism by taking
into account the coupling constants for the mixed positionmomentum terms. In the classical setup the coupling constants were the scalars µ1 and κ, whereas in a game they are
the off-diagonal blocks A12 and C12 .
Apply singular value decomposition to factorize
A12 = U|A DA VA

and C12 = U|C DC VC

where the entries of the diagonal matrices have types
1
τ (DA ) = kg
and τ (DC ) = kg
s2 , and the types of the orthogonal matrices U and V are pure scalars. The diagonal
matrices DA and DC have the same types as µ1 and κ in
the classical system since they play the same coupling role.
Extending the procedure adopted in the section E.1, fix the
type-inconsistency by defining the musical isomorphisms as

and



∂
∂x

[

= U|A D−1
A UA · dx



∂
∂y

[

|
= VC
D−1
C VC · dy.

Alternatively, the isomorphisms
can be computed by noting
√
|
|
−1
that
U
D
U
=
(
A
A
)−1 and VC
D−1
A
12
21
A A
C VC =
√
−1
( C21 C12 ) .
The dual isomorphism ] : T ∗ M → T M is then
(dx) = U|A DA UA ·
]

∂
∂
]
|
and (dy) = VC
D C VC ·
∂x
∂y

If
ξ=

 kg2 ·m 


∂ [
∂ [
naively define ( ∂x
) = dx and ( ∂y
) = dy then



A12 y + b1
C21 x + d2



then it follows that
 | −1

UA DA UA b1 + U|A UA y
ξ[ =
|
|
VC
D−1
C VC d2 + VC VC x

for `2 . It follows from a straightforward computation that
∂`2 ∂
1
1 ∂
the vector field ξ = ∂`
∂x ∂x + ∂y ∂y has type τ (ξ) = s as
required.

with associated closed two form

The presymplectic form ω = dξ [ makes use of the musical
isomorphism [ : T M → T ∗ M . As in section E.1, if we

where the notation ωτ emphasizes that the two-form is typeconsistent.

ωτ = dξ [ = − (U|A VA − U|C VC ) dx ∧ dy.

Differentiable Game Mechanics

E.3. What does type-consistency buy?
Example 7. Consider the loss functions
f (x, y) = xy

and

g(x, y) = 2xy,

with ξ = (y, 2x). There is no function φ : R2 → R such
that ∇φ = ξ. However, there is a family of functions
φα (x, y) = α · xy which satisfies
ξ, ∇φα = α · (x2 + 2y 2 ) ≥ 0

for all α > 0.

Although ξ is not a potential field, there is a family of
functions on which ξ performs gradient descent – albeit
with coordinate-wise learning rates that may not be optimal.
The vector field ξ arguably does not require adjustment.
This kind of situation often arises when the learning rates
of different parameters are set adaptively during training of
neural nets, by rescaling them by positive numbers.
The vanilla and type-consistent 1-forms corresponding to ξ
are, respectively,
ξ [ = y · dx + 2x · dy

and

ξ [τ = y · dx + x · dy

with
ω = dξ [non = dx ∧ dy

and

ωτ = dξ [τ = 0.

It follows that the type-consistent symplectic gradient adjustment is zero. Type-consistency ‘detects’ that no gradient
adjustment is needed in example 7.

Differentiable Game Mechanics
SYMPLECTIC GRADIENT ADJUSTMENT
learning rate 1.0

OPTIMISTIC MIRROR DESCENT
learning rate 1.0

SYMPLECTIC GRADIENT ADJUSTMENT
learning rate 0.4

OPTIMISTIC MIRROR DESCENT
learning rate 0.4

SYMPLECTIC GRADIENT ADJUSTMENT
learning rate 1.16

OPTIMISTIC MIRROR DESCENT
learning rate 1.16

Figure 10. Individual runs on zero-sum bimatrix game in section 4.2.

