Wasserstein GAN
Martin Arjovsky Soumith Chintala Léon Bottou

Frustrated honeycomb-lattice bilayer quantum antiferromagnet in a magnetic field: 
       Unconventional phase transitions in a two-dimensional isotropic Heisenberg model
    Johannes Richter
    December 30, 2023
==========================================================================================================================================================================







§ INTRODUCTION

The problem this paper is concerned with is that of unsupervised learning.
Mainly, what does it mean to learn a probability
distribution? The classical answer to this is to learn a probability
density. This is often done by defining a parametric family of densities
(P_θ)_θ∈^d and finding the one that maximized the likelihood on our data:
if we have real data examples {x^(i)}_i=1^m, we would solve the problem

    max_θ∈^d1/m∑_i=1^m log P_θ(x^(i))

If the real data distribution _r admits a density and _θ is the
distribution of the parametrized density P_θ, then, asymptotically, this
amounts to minimizing the Kullback-Leibler divergence KL(_r _θ).

For this to make sense, we need the model density P_θ to exist.
This is not the case in the rather common situation where we are
dealing with distributions supported by low dimensional manifolds.
It is then unlikely that the model manifold and the true distribution's support
have a non-negligible intersection (see <cit.>),
and this means that the KL distance is not defined (or simply infinite).

The typical remedy is to add a noise term to the model
distribution. This is why virtually all generative models described in
the classical machine learning literature include a noise
component. In the simplest case, one assumes a Gaussian noise with
relatively high bandwidth in order to cover all the examples. It is
well known, for instance, that in the case of image generation models,
this noise degrades the quality of the samples and makes them
blurry.
For example, we can see in the recent paper <cit.>
that the optimal standard deviation of the noise added to the model
when maximizing likelihood is around 0.1 to each pixel in a generated image,
when the pixels were already normalized to be in the range [0, 1]. This is
a very high amount of noise, so much that when papers report the samples of
their models, they don't add the noise term on which they report likelihood numbers.
In other words, the added noise term
is clearly incorrect for the problem, but is needed to make the
maximum likelihood approach work.

Rather than estimating the density of _r which may not exist, we
can define a random variable Z with a fixed distribution p(z) and
pass it through a parametric function g_θ: 𝒵→𝒳 (typically a neural network of some kind) that directly
generates samples following a certain distribution _θ. By
varying θ, we can change this distribution and make it close to
the real data distribution _r.  This is useful in two ways. First
of all, unlike densities, this approach can represent distributions
confined to a low dimensional manifold.  Second, the ability to easily
generate samples is often more useful than knowing the numerical value
of the density (for example in image superresolution or semantic
segmentation when considering the conditional distribution of the
output image given the input image). In general, it is computationally
difficult to generate samples given an arbitrary high dimensional
density <cit.>.

Variational Auto-Encoders (VAEs) <cit.> and Generative
Adversarial Networks (GANs) <cit.> are well known examples
of this approach.  Because VAEs focus on the approximate likelihood of
the examples, they share the limitation of the standard models and
need to fiddle with additional noise terms. GANs offer much more
flexibility in the definition of the objective function, including
Jensen-Shannon <cit.>, and all f-divergences
<cit.> as well as some exotic combinations
<cit.>. On the other hand, training GANs is well known for
being delicate and unstable, for reasons theoretically investigated in
<cit.>.

In this paper, we direct our attention on the various ways to measure
how close the model distribution and the real distribution are, or
equivalently, on the various ways to define a distance or divergence
ρ(_θ,_r). The most fundamental difference between such
distances is their impact on the convergence of sequences of
probability distributions. A sequence of distributions
(_t)_t∈ converges if and only if there is a distribution
_∞ such that ρ(_t,_∞) tends to zero,
something that depends on how exactly the distance ρ is defined.
Informally, a distance ρ induces a weaker topology when it makes
it easier for a sequence of distribution to converge.[More
  exactly, the topology induced by ρ is weaker than that induced
  by ρ' when the set of convergent sequences under ρ is a
  superset of that under ρ'.]  Section <ref>
clarifies how popular probability distances differ in that respect.












In order to optimize the parameter θ, it is of course desirable
to define our model distribution _θ in a manner that makes
the mapping θ↦_θ continuous.
Continuity means that when a sequence of parameters θ_t converges
to θ, the distributions _θ_t also converge
to _θ. However, it is essential to remember that the
notion of the convergence of the distributions _θ_t depends
on the way we compute the distance between distributions.
The weaker this distance, the easier
it is to define a continuous mapping from θ-space to
_θ-space, since it's easier for the
distributions to converge.
The main reason we care about the mapping θ↦_θ
to be continuous is as follows. If ρ is our notion of
distance between two distributions, we would like to have a loss
function θ↦ρ(_θ, _r) that is continuous,
and this is equivalent to having the mapping θ↦_θ
be continuous when using the distance between distributions ρ.

The contributions of this paper are:


  * In Section <ref>, we provide a comprehensive
  theoretical analysis of how the Earth Mover (EM) distance behaves in comparison to
  popular probability distances and divergences used in the
  context of learning distributions.

  * In Section <ref>, we define a form of GAN called Wasserstein-GAN
  that minimizes a reasonable and efficient approximation of the EM distance,
  and we theoretically show that the corresponding optimization 
  problem is sound.

  * In Section <ref>, we empirically show that WGANs
  cure the main training problems of GANs. In particular, training
  WGANs does not require maintaining a careful balance in training of
  the discriminator and the generator, and does not require a careful
  design of the network architecture either. The mode dropping
  phenomenon that is typical in GANs is also drastically reduced.  One
  of the most compelling practical benefits of WGANs is the ability to
  continuously estimate the EM distance by training the
  discriminator to optimality. Plotting these learning curves is not
  only useful for debugging and hyperparameter searches, but also
  correlate remarkably well with the observed sample quality.














§ DIFFERENT DISTANCES




We now introduce our notation. Let  be a compact metric set
(such as the space of images [0,1]^d) and let Σ denote the
set of all the Borel subsets of . Let Prob()
denote the space of probability measures defined on .
We can now define elementary distances and divergences
between two distributions _r,_g∈Prob():


  * The Total Variation (TV) distance
  
    δ(_r,_g) = sup_A∈Σ |_r(A)-_g(A)| .


  * The Kullback-Leibler (KL) divergence
  
    KL(_r_g) = ∫log(P_r(x)/P_g(x)) P_r(x) dμ(x) ,

  where both _r and _g are assumed to be absolutely continuous,
  and therefore admit densities, with respect to a same measure μ defined on .[
    Recall that a probability distribution
    _r∈Prob() admits a density p_r(x) with
    respect to μ, that is, ∀ A∈Σ, _r(A) =
    ∫_A P_r(x) dμ(x), if and only it is absolutely continuous
    with respect to μ, that is, ∀ A∈Σ,
    μ(A)=0⇒_r(A)=0 .]
  The KL divergence is famously assymetric and possibly infinite when there
  are points such that P_g(x)=0 and P_r(x)>0.

  * The Jensen-Shannon (JS) divergence
  
    JS(_r,_g) = KL(_r_m)+KL(_g_m) ,

  where _m is the mixture (_r+_g)/2. This divergence
  is symmetrical and always defined because we can choose μ=_m.

  * The Earth-Mover (EM) distance or Wasserstein-1
  
    W(_r, _g) = inf_γ∈Π(_r ,_g)_(x, y) ∼γ[ x - y ] ,

  where Π(_r,_g) denotes the set of all joint distributions
  γ(x,y) whose marginals are respectively _r and
  _g. Intuitively, γ(x,y) indicates how much “mass” must
  be transported from x to y in order to transform the distributions
  _r into the distribution _g. The EM distance then
  is the “cost” of the optimal transport plan.


The following example illustrates how apparently simple sequences of
probability distributions converge under the EM distance but do not
converge under the other distances and divergences defined above.

[Learning parallel lines] 
Let Z ∼ U[0,1] the uniform distribution
on the unit interval. Let _0 be the
distribution of (0, Z) ∈^2 (a 0 on the x-axis
and the random variable Z on the y-axis), uniform on a straight
vertical line passing through the origin. Now
let g_θ(z) = (θ, z) with θ
a single real parameter. It is easy to see
that in this case,


  * W(_0, _θ) = |θ|,

  * JS(_0,_θ) =
  log 2      if θ≠ 0 , 

    0      if θ = 0 ,

  * KL(_θ_0) = KL(_0 _θ) =
  
    +∞     if θ≠ 0 , 

    0      if θ = 0 ,

  * and δ(_0,_θ) = 
  
    1      if θ≠ 0 , 

    0      if θ = 0 .

When θ_t→0,
the sequence (_θ_t)_t∈ converges
to _0 under the EM distance, but does not converge
at all under either the JS, KL, reverse KL, or TV divergences.
<ref> illustrates this for the case of the EM and JS distances.




Example <ref> gives us a case where we can learn a
probability distribution over a low dimensional manifold by doing
gradient descent on the EM distance. This cannot be done with the
other distances and divergences because the resulting loss function is
not even continuous. Although this simple example features
distributions with disjoint supports, the same conclusion holds when
the supports have a non empty intersection contained in a set of
measure zero. This happens to be the case when two low dimensional
manifolds intersect in general position <cit.>.




0
We now introduce our notation. Let 𝒳⊆^d
be a compact set (such as [0,1]^d the space of images). We define
Prob() to be the space of probability measures over
. We note

    C_b() = {f: →, f is continuous and bounded}

Note that if f ∈ C_b(),
we can define f_∞ = max_x ∈ |f(x)|, since
f is bounded. With this norm, the space (C_b(), ·_∞)
is a normed vector space. As for any normed vector space, we can define
its dual

    C_b()^* = {ϕ: C_b() →, ϕ is linear
    and continuous}

and give it the dual norm ϕ = sup_f ∈ C_b(), f_∞≤ 1 |ϕ(f)|.

With this definitions, (C_b()^*, ·) is another normed space.
Now let μ be a signed measure over , and let us define
the total variation distance

    μ_TV = sup_A ⊆ |μ(A)|

where the supremum is taken all Borel sets in .
Since the total variation is a norm, then if we have _r
and _θ two probability distributions over ,

    δ(_r, _θ) := _r - _θ

is a distance in Prob() (called the total variation
distance).

We can consider

    Φ: (Prob(), δ) → (C_b()^*, ·)

where Φ()(f) := _x ∼[f(x)] is a linear function
over C_b(). The Riesz Representation theorem (<cit.>,
Theorem 10) tells us that Φ is an isometric immersion. This
tells us that we can effectively consider Prob() with
the total variation distance as a subset of C_b()^* with
the norm distance. Thus, just to accentuate it one more time,
the total variation over Prob() is exactly
the norm distance over C_b()^*.

Let us stop for a second and analyze what all this technicality meant.
The main thing to carry is that we introduced a distance δ
over probability distributions. When looked as a distance over
a subset of C_b()^*, this distance gives the norm topology.
The norm topology is very strong. Therefore, we can expect that
not many functions θ↦_θ will be continuous
when measuring distances between distributions with δ. As
we will show later in Theorem <ref>, δ gives the same topology
as the Jensen-Shannon divergence, pointing to the fact that the
JS is a very strong distance, and is thus more propense to
give a discontinuous loss function.

Now, all dual spaces (such as C_b()^* and thus
Prob()) have a strong topology (induced by the norm),
and a weak* topology. As the name suggests, the weak* topology
is much weaker than the strong topology. In the case of
Prob(), the strong topology is given by the
total variation distance, and the weak* topology is given
by the Wasserstein distance (among others) <cit.>.

Since the Wasserstein distance is much weaker than the strong
topology, we can now ask whether W(_r, _θ) is
a continuous loss function on θ 
under mild assumptions. This, and more, is
true, as we now state and prove.

  Let _r be a fixed distribution over 𝒳.
Let Z be a random variable (e.g Gaussian) over another
space 𝒵. Let g: 𝒵×^d →𝒳
be a function, that will be denoted g_θ(z) with z the first coordinate
and θ the second. Let _θ denote the distribution of g_θ(Z).
Then,


  * If g is continuous in θ, so is W(_r, _θ).

  * If g is locally Lipschitz and satisfies regularity
assumption <ref>,
then W(_r, _θ) is continuous
everywhere, and differentiable almost everywhere.

  * Statements 1-2 are false for the Jensen-Shannon divergence JS(_r, _θ)
and all the KLs.


 
See Appendix <ref>


[Learning parallel lines] 
Let Z ∼ U[0,1] the uniform distribution
on the unit interval. Let _0 be the
distribution of (0, Z) ∈^2 (a 0 on the x-axis
and the random variable Z on the y-axis), uniform on a straight
vertical line passing through the origin. Now
let g_θ(z) = (θ, z) with θ
a single real parameter. It is easy to see
in this case W(_0, _θ) = |θ|
and

    JS(_0,_θ) =
    log 2      if θ≠ 0 
    
      0      if θ = 0

which is clearly not continuous. Similarly, in the case of
the KLs, we would see that

    KL(_θ_0) = KL(_0 _θ) =
    
     +∞     if θ≠ 0 
    
      0      if θ = 0




Example <ref> gives us a case where we can learn
a probability distribution over a low dimensional manifold
by doing gradient descent on the Wasserstein distance. Something
we know can't be done with the JS because of the theorems
of <cit.>. Note that the discontinuity problem
would arize even if the manifolds intersect, as proved in
<cit.>. This is because unless they are exactly
the same, the intersection would be negligible (lying on a set
of measure 0 for both manifolds, in this case a point).




Since the Wasserstein distance is much weaker than the JS distance[
The argument for why this happens, and indeed
how we arrived to the idea that Wasserstein is what
we should really be optimizing is displayed in Appendix
<ref>. We strongly encourage the interested
reader who is not afraid of the mathematics to go through it.],
we can now ask whether W(_r, _θ) is
a continuous loss function on θ 
under mild assumptions. This, and more, is
true, as we now state and prove.

  Let _r be a fixed distribution over 𝒳.
Let Z be a random variable (e.g Gaussian) over another
space 𝒵. Let g: 𝒵×^d →𝒳
be a function, that will be denoted g_θ(z) with z the first coordinate
and θ the second. Let _θ denote the distribution of g_θ(Z).
Then,


  * If g is continuous in θ, so is W(_r, _θ).

  * If g is locally Lipschitz and satisfies regularity
assumption <ref>,
then W(_r, _θ) is continuous
everywhere, and differentiable almost everywhere.

  * Statements 1-2 are false for the Jensen-Shannon divergence JS(_r, _θ)
and all the KLs.


 
See Appendix <ref>


The following corollary tells us that learning by minimizing
the EM distance makes sense (at least in theory) with neural networks.
 
Let g_θ be any feedforward neural network[By a
feedforward neural network we mean a function composed by
affine transformations and pointwise nonlinearities which are
smooth Lipschitz
functions (such as the sigmoid, tanh, elu, softplus, etc).
Note: the statement is also true for rectifier
nonlinearities but the proof is more
technical (even though very similar) so we omit it.] parameterized by θ, 
and p(z) a prior over z such that _z ∼ p(z)[z] < ∞ (e.g.
Gaussian, uniform, etc.). Then assumption
<ref> is satisfied and therefore W(_r, _θ)
is continuous everywhere and differentiable almost everywhere.


See Appendix <ref>


All this shows that EM is a much more sensible
cost function for our problem than at least the Jensen-Shannon
divergence. The following theorem describes the relative strength of
the topologies induced by these distances and divergences, with KL the strongest,
followed by JS and TV, and EM the weakest.

 
Let  be a distribution on a compact space  and 
(_n)_n ∈ be a sequence
of distributions on . Then, considering
all limits as n →∞,


  * The following statements are equivalent

  
  * δ(_n, ) → 0
with δ the total variation distance.
  
  * JS(_n,) → 0 with
JS the Jensen-Shannon divergence.


  * The following statements are equivalent

  
  * W(_n, ) → 0.
  
  * _n where  represents
convergence in distribution for random variables.


  * KL(_n ) → 0 or KL(_n) → 0 imply
the statements in (1). 

  * The statements in (1) imply the statements in (2).



See Appendix <ref>


This highlights the fact that 
the KL, JS, and TV distances are not sensible
cost functions when learning distributions
supported by low dimensional manifolds.
However the EM distance is sensible
in that setup. This obviously leads us to the next section
where we introduce a practical approximation
of optimizing the EM distance.



§ WASSERSTEIN GAN



Again, Theorem <ref> points to the fact that
W(_r, _θ) might have nicer properties
when optimized than JS(_r,_θ).
However, the infimum in
(<ref>) is highly intractable. On the other hand,
the Kantorovich-Rubinstein duality <cit.>
tells us that

    W(_r, _θ) = sup_f_L ≤ 1_x ∼_r
    [f(x)] - _x ∼_θ[f(x)]

where the supremum is over all the 1-Lipschitz functions
f: →. Note that if we replace f_L ≤ 1
for f_L ≤ K (consider K-Lipschitz for some constant K), then
we end up with K · W(_r, _g). Therefore, if we have a
parameterized family of functions {f_w}_w ∈𝒲
that are all K-Lipschitz for some K, we could consider
solving the problem

    max_w ∈𝒲_x ∼_r[f_w(x)] -
    _z ∼ p(z) [f_w(g_θ(z)]

and if the supremum in (<ref>) is attained
for some w ∈𝒲 (a pretty strong assumption
akin to what's assumed when proving consistency of an
estimator), this process would
yield a calculation of W(_r, _θ) up to
a multiplicative constant. Furthermore, we could consider
differentiating W(_r, _θ) (again, up to a constant)
by back-proping through equation (<ref>) via
estimating _z ∼ p(z)[∇_θ f_w(g_θ(z))].
While this is all intuition, we now prove that this process
is principled under the optimality assumption.
 
Let _r be any distribution. Let _θ be
the distribution of g_θ(Z) with Z a random
variable with density p and g_θ
a function satisfying assumption <ref>. 
Then, there is a solution f: →
to the problem

    max_f_L ≤ 1_x ∼_r[f(x)] - 
      _x ∼_θ [f(x)]

and we have

    ∇_θ W(_r, _θ)
    = -_z ∼ p(z)[∇_θ f(g_θ(z))]

when both terms are well-defined.

 See Appendix <ref>


Now comes the question of finding the function f that
solves the maximization problem in equation (<ref>).
To roughly approximate
this, something that we can do is train a neural network
parameterized with weights w lying in a compact space
𝒲 and then backprop through
_z ∼ p(z)[∇_θ f_w(g_θ(z))], as we
would do with a typical GAN. Note that the fact that 
𝒲 is compact implies that all the functions
f_w will be K-Lipschitz for some K that only depends
on 𝒲 and not the individual weights, therefore
approximating (<ref>) up to an irrelevant scaling factor
and the capacity of the `critic' f_w. In order to have parameters w
lie in a compact space, something simple we can do is clamp
the weights to a fixed box (say 𝒲 = [-0.01,0.01]^l) after each
gradient update. The Wasserstein Generative Adversarial
Network (WGAN) procedure is described in Algorithm <ref>.

Weight clipping is a clearly terrible way to enforce a Lipschitz constraint.
If the clipping parameter is large, then it can take a long time
for any weights to reach their limit, thereby making it harder
to train the critic till optimality. If the clipping is small, this
can easily lead to vanishing gradients when the number of layers is big,
or batch normalization is not used (such as in RNNs). We experimented
with simple variants (such as projecting the weights to a sphere) with
little difference, and we stuck with weight clipping due to its simplicity
and already good performance. However, we do leave the topic of
enforcing Lipschitz constraints in a neural network setting for further
investigation, and we actively encourage interested researchers to improve
on this method.





The fact that the EM distance is continuous and differentiable a.e.
means that we can (and should) train
the critic till optimality. The argument is simple,
the more we train the critic, the more reliable gradient of
the Wasserstein we get, which is actually useful by the
fact that Wasserstein is differentiable almost everywhere.
For the JS, as the discriminator gets better the gradients get
more reliable but the true gradient is 0 since the JS is locally
saturated and we get vanishing gradients,
as can be seen in <ref> of this paper
and Theorem 2.4 of <cit.>. In <ref>
we show a proof of concept of this, where we train
a GAN discriminator and a WGAN critic till optimality.
The discriminator learns very quickly to distinguish between
fake and real, and as expected provides no reliable gradient
information. The critic, however, can't saturate, and converges
to a linear function that gives remarkably clean gradients everywhere.
The fact that we constrain the weights limits the possible
growth of the function to be at most linear in different parts
of the space, forcing the optimal critic to have this behaviour.



Perhaps more importantly, the fact that we can train the critic
till optimality makes it impossible to collapse modes when we do.
This is due to the fact that mode collapse comes from the fact
that the optimal generator for a fixed discriminator
is a sum of deltas on the points the discriminator assigns
the highest values, as observed by <cit.> and
highlighted in <cit.>.

In the following section we display the practical benefits
of our new algorithm, and we provide an in-depth comparison
of its behaviour and that of traditional GANs.



§ EMPIRICAL RESULTS



  We run experiments on image generation using our Wasserstein-GAN algorithm
and show that there are significant practical benefits to using it over the 
formulation used in standard GANs.

We claim two main benefits:


  * a meaningful loss metric that correlates with the generator's 
convergence and sample quality

  * improved stability of the optimization process




 §.§ Experimental Procedure


We run experiments on image generation. The target distribution to learn is the 
LSUN-Bedrooms dataset <cit.> – a collection of natural images of
indoor bedrooms. Our baseline comparison is DCGAN <cit.>,
a GAN with a convolutional architecture trained with the standard GAN procedure
using the -log D trick <cit.>.
The generated samples are 3-channel images of 64x64 pixels in size.
We use the hyper-parameters specified in Algorithm <ref> for 
all of our experiments.



 §.§ Meaningful loss metric


Because the WGAN algorithm attempts to train the critic f (lines 2–8 in Algorithm <ref>)
relatively well before each generator update (line 10 in Algorithm <ref>),
the loss function at this point is an estimate of the EM distance, up to constant
factors related to the way we constrain the Lipschitz constant of f.

Our first experiment illustrates how this estimate correlates well
with the quality of the generated samples. Besides the convolutional
DCGAN architecture, we also ran experiments where we replace the
generator or both the generator and the critic by 4-layer
ReLU-MLP with 512 hidden units.






<ref> plots the evolution of the WGAN estimate
(<ref>) of the EM distance during WGAN training for all three
architectures.  The plots clearly show that these curves correlate
well with the visual quality of the generated samples.

To our knowledge, this is the first time in GAN literature that such a
property is shown, where the loss of the GAN shows properties of
convergence. This property is extremely useful when doing research in
adversarial networks as one does not need to stare at the generated
samples to figure out failure modes and to gain information on which
models are doing better over others.

However, we do not claim that this is a new method to quantitatively
evaluate generative models yet. The constant scaling
factor that depends on the critic's architecture means it's hard
to compare models with different critics. Even more,
in practice the fact that the critic doesn't have
infinite capacity makes it hard to know just how
close to the EM distance our estimate really is.
This being said, we have succesfully used the loss metric to
validate our experiments repeatedly and without failure, and 
we see this as a huge improvement in training GANs which 
previously had no such facility.

In contrast, <ref> plots the evolution of the GAN estimate
of the JS distance during GAN training. More precisely, during GAN training,
the discriminator is trained to maximize
 
    L(D, g_θ) = _x ∼_r[log D(x)] + _x ∼_θ[log(1 - D(x))]

which is is a lower bound of 2 JS(_r,_θ) - 2 log 2.
In the figure, we plot the quantity 1/2 L(D, g_θ) + log 2,
which is a lower bound of the JS distance.

This quantity clearly correlates poorly the sample quality. Note
also that the JS estimate usually stays constant or goes up instead
of going down. In fact
it often remains very close to log2≈0.69 which is the highest
value taken by the JS distance. In other words, the JS distance
saturates, the discriminator has zero loss, and the generated samples
are in some cases meaningful (DCGAN generator, top right plot) and in
other cases collapse to a single nonsensical image <cit.>.
This last phenomenon has been theoretically explained in
<cit.> and highlighted in <cit.>.

When using the -log D trick <cit.>,
the discriminator loss and the generator loss are different.
Figure <ref> in Appendix <ref>
reports the same plots for GAN training, but using
the generator loss instead of the discriminator loss.
This does not change the conclusions.

Finally, as a negative result, we report that WGAN training becomes unstable
at times when one uses a momentum based optimizer such as Adam <cit.> (with β_1 > 0)
on the critic, or
when one uses high learning rates. Since the loss for the critic is nonstationary, momentum
based methods seemed to perform worse. We identified
momentum as a potential cause because, as the loss blew up and samples got worse,
the cosine between the Adam step and the gradient usually turned negative. The
only places where this cosine was negative was in these situations of
instability. We therefore switched to RMSProp <cit.> which is known
to perform well even on very nonstationary problems <cit.>.



 §.§ Improved stability


One of the benefits of WGAN is that it allows us to train the critic
till optimality. When the critic is trained to completion, it simply
provides a loss to the generator that we can train as any other neural
network. This tells us that we no longer need to balance generator and
discriminator's capacity properly. The better the critic, the higher
quality the gradients we use to train the generator.

We observe that WGANs are much more robust than GANs when one varies
the architectural choices for the generator. We illustrate this
by running experiments on three generator architectures:
(1) a convolutional DCGAN generator, (2) a convolutional DCGAN generator
without batch normalization and with a constant number of filters,
and (3) a 4-layer ReLU-MLP with 512 hidden units.
The last two are known to perform very poorly with GANs.
We keep the convolutional DCGAN architecture for
the WGAN critic or the GAN discriminator.



Figures <ref>, <ref>, and <ref> show
samples generated for these three architectures using both the WGAN
and GAN algorithms. We refer the reader to Appendix
<ref> for full sheets of generated samples.
Samples were not cherry-picked.

In no experiment did we see evidence of mode collapse for the WGAN algorithm.






§ RELATED WORK


There's been a number of works on the so called
Integral Probability Metrics (IPMs) <cit.>.
Given  a set of functions from 𝒳
to , we can define

    d_(_r, _θ) = sup_f ∈_x ∼_r[f(x)] - _x ∼_θ[f(x)]

as an integral probability metric associated with the
function class . It is easily verified that
if for every f ∈ we have -f ∈
(such as all examples we'll consider), then
d_ is nonnegative, satisfies the triangular
inequality, and is symmetric. Thus, d_ is
a pseudometric over Prob().

While IPMs might seem to share a similar formula,
as we will see different classes of functions
can yeald to radically different metrics.



  * By the Kantorovich-Rubinstein duality <cit.>,
we know that W(_r, _θ) = d_(_r,
_θ) when  is the set of 1-Lipschitz
functions. Furthermore, if  is the set
of K-Lipschitz functions, we get K · W(_r,
_θ) = d_(_r, _θ).


  * When  is the set of all measurable
functions bounded between -1 and 1 (or all
continuous functions between -1 and 1), we
retrieve d_(_r, _θ) = δ(_r,
_θ) the total variation distance <cit.>.
This already tells us that going from 1-Lipschitz
to 1-Bounded functions drastically changes the
topology of the space, and the regularity
of d_(_r, _θ) as a loss
function (as by Theorems <ref>
and <ref>).


  * Energy-based GANs (EBGANs) <cit.>
can be thought of
as the generative approach to the total variation
distance. This connection is stated and
proven in depth in Appendix <ref>.
At the core of the connection is that the discriminator
will play the role of f maximizing equation
(<ref>) while its only restriction is
being between 0 and m for
some constant m. This will yeald the same
behaviour as being restricted to be between -1
and 1 up to a constant scaling factor irrelevant
to optimization. Thus, when the discriminator
approaches optimality the cost for the generator
will aproximate the total variation distance
δ(_r, _θ).

Since the total variation distance displays the
same regularity as the JS, it can be seen that
EBGANs will suffer from the same problems
of classical GANs regarding not being able
to train the discriminator till optimality
and thus limiting itself to very imperfect
gradients.


  * Maximum Mean Discrepancy (MMD) <cit.> is
a specific case of integral probability metrics when
= {f ∈ℋ: f_∞≤ 1} for
ℋ some Reproducing Kernel Hilbert Space (RKHS)
associated with a given kernel k: ×→.
As proved on <cit.> we know that MMD is a proper
metric and not only a pseudometric when the kernel is universal.
In the specific case where
ℋ = L^2(, m) for m the normalized Lebesgue
measure on , we know that {f ∈ C_b(), f_∞≤ 1}
will be contained in , and therefore d_(_r, _θ)
≤δ(_r, _θ) so the regularity of the MMD distance
as a loss function will be at least as bad as the one of the total
variation. Nevertheless this is a very extreme case, since we would
need a very powerful kernel to approximate the whole L^2. However,
even Gaussian kernels are able to detect tiny noise patterns
as recently evidenced by <cit.>. This points to the
fact that especially with low bandwidth kernels, the distance
might be close to a saturating regime similar as with total
variation or the JS. This obviously doesn't need to
be the case for every kernel, and figuring out how and which different MMDs
are closer to Wasserstein or total variation distances is an interesting
topic of research.

The great aspect of MMD is that via the kernel trick there is no need
to train a separate network to maximize equation (<ref>) for the ball
of a RKHS. However, this has the disadvantage that evaluating the MMD distance
has computational cost that grows quadratically with the amount of samples
used to estimate the expectations in (<ref>). This last point
makes MMD have limited scalability, and is sometimes inapplicable to
many real life applications because of it. There are estimates with
linear computational cost for the MMD <cit.> which
in a lot of cases makes MMD very useful, but they also have worse sample complexity.


  * Generative Moment Matching Networks (GMMNs) <cit.>
are the generative counterpart of MMD. By backproping through
the kernelized formula for equation (<ref>), they directly
optimize d_MMD(_r, _θ) (the IPM when  is
as in the previous item). As mentioned, this has the advantage
of not requiring a separate network to approximately maximize
equation (<ref>). However, GMMNs have enjoyed limited applicability.
Partial explanations for their unsuccess are the quadratic cost as a function
of the number of samples and vanishing gradients for low-bandwidth kernels.
Furthermore, it may be possible that some
kernels used in practice are unsuitable for capturing very complex
distances in high dimensional sample spaces such as natural images.
This is properly justified by the fact that <cit.>
shows that for the typical Gaussian MMD test to be reliable (as in it's power
as a statistical test approaching 1), we need the number of
samples to grow linearly with the number of dimensions. Since
the MMD computational cost grows quadratically with the number
of samples in the batch used to estimate equation (<ref>),
this makes the cost of having a reliable estimator
grow quadratically with the number of dimensions, which makes it
very inapplicable for high dimensional problems. Indeed, for
something as standard as 64x64 images, we would need minibatches
of size at least 4096 (without taking into account the constants
in the bounds of <cit.> which would make this number
substantially larger) and a total cost per iteration of
4096^2, over 5 orders of magnitude more than a
GAN iteration when using the standard batch size of 64.

That being said, these numbers can be a bit unfair to the MMD,
in the sense that we are comparing empirical sample complexity of
GANs with the theoretical sample complexity of MMDs, which tends
to be worse. However, in the original GMMN paper <cit.> they
indeed used a minibatch of size 1000, much larger than the standard
32 or 64 (even when this incurred in quadratic computational cost).
While estimates that have linear computational cost as a function of the number
of samples exist <cit.>,
they have worse sample complexity, and to the best of
our knowledge they haven't been yet applied in a generative context
such as in GMMNs.



On another great line of research, the recent work of <cit.>
has explored the use of Wasserstein distances in the context of learning
for Restricted Boltzmann Machines for discrete spaces. The motivations
at a first glance might seem quite different, since the manifold setting is restricted
to continuous spaces and in finite discrete spaces the weak and strong
topologies (the ones of W and JS respectively) coincide. However, in the
end there is more in commmon than not about our motivations. We both
want to compare distributions in a way that leverages the geometry of the
underlying space, and Wasserstein allows us to do exactly that.

Finally, the work of <cit.> shows new algorithms for calculating
Wasserstein distances between different distributions. We believe this direction
is quite important, and perhaps could lead to new ways of evaluating generative models.



§ CONCLUSION

  We introduced an algorithm that we deemed WGAN, an alternative
to traditional GAN training. In this new model, we showed that we can
improve the stability of learning, get rid of problems like mode collapse,
and provide meaningful learning curves useful for debugging and hyperparameter
searches. Furthermore, we showed that the corresponding optimization problem
is sound, and provided extensive theoretical work highlighting the
deep connections to other distances between distributions.



§ ACKNOWLEDGMENTS

We would like to thank
Mohamed Ishmael Belghazi,
Emily Denton,
Ian Goodfellow,
Ishaan Gulrajani,
Alex Lamb,
David Lopez-Paz,
Eric Martin,
Maxime Oquab,
Aditya Ramesh,
Ronan Riochet,
Uri Shalit,
Pablo Sprechmann,
Arthur Szlam,
Ruohan Wang,
for helpful comments and advice.


plain






§ WHY WASSERSTEIN IS INDEED WEAK
 
We now introduce our notation. Let 𝒳⊆^d
be a compact set (such as [0,1]^d the space of images). We define
Prob() to be the space of probability measures over
. We note

    C_b() = {f: →, f is continuous and bounded}

Note that if f ∈ C_b(),
we can define f_∞ = max_x ∈ |f(x)|, since
f is bounded. With this norm, the space (C_b(), ·_∞)
is a normed vector space. As for any normed vector space, we can define
its dual

    C_b()^* = {ϕ: C_b() →, ϕ is linear
    and continuous}

and give it the dual norm ϕ = sup_f ∈ C_b(), f_∞≤ 1 |ϕ(f)|.

With this definitions, (C_b()^*, ·) is another normed space.
Now let μ be a signed measure over , and let us define
the total variation distance

    μ_TV = sup_A ⊆ |μ(A)|

where the supremum is taken all Borel sets in .
Since the total variation is a norm, then if we have _r
and _θ two probability distributions over ,

    δ(_r, _θ) := _r - _θ_TV

is a distance in Prob() (called the total variation
distance).

We can consider

    Φ: (Prob(), δ) → (C_b()^*, ·)

where Φ()(f) := _x ∼[f(x)] is a linear function
over C_b(). The Riesz Representation theorem (<cit.>,
Theorem 10) tells us that Φ is an isometric immersion. This
tells us that we can effectively consider Prob() with
the total variation distance as a subset of C_b()^* with
the norm distance. Thus, just to accentuate it one more time,
the total variation over Prob() is exactly
the norm distance over C_b()^*.

Let us stop for a second and analyze what all this technicality meant.
The main thing to carry is that we introduced a distance δ
over probability distributions. When looked as a distance over
a subset of C_b()^*, this distance gives the norm topology.
The norm topology is very strong. Therefore, we can expect that
not many functions θ↦_θ will be continuous
when measuring distances between distributions with δ. As
we will show later in Theorem <ref>, δ gives the same topology
as the Jensen-Shannon divergence, pointing to the fact that the
JS is a very strong distance, and is thus more propense to
give a discontinuous loss function.

Now, all dual spaces (such as C_b()^* and thus
Prob()) have a strong topology (induced by the norm),
and a weak* topology. As the name suggests, the weak* topology
is much weaker than the strong topology. In the case of
Prob(), the strong topology is given by the
total variation distance, and the weak* topology is given
by the Wasserstein distance (among others) <cit.>.


§ ASSUMPTION DEFINITIONS


 
Let g: 𝒵×^d →𝒳 be
locally Lipschitz between finite dimensional vector spaces.
We will denote g_θ(z) it's evaluation on coordinates
(z, θ). We say that g satisfies assumption <ref>
for a certain probability distribution p over 𝒵
if there are local Lipschitz constants L(θ, z) such
that

    _z ∼ p[L(θ, z)] < +∞
 




§ PROOFS OF THINGS
 


Let θ and θ' be two parameter vectors in ^d. Then, we 
will first attempt to bound W(_θ, _θ'), from where the
theorem will come easily. The main element of the proof is the use of the
coupling γ, the distribution of the joint (g_θ(Z), g_θ'(Z)),
which clearly has γ∈Π(_θ, _θ').

By the definition of the Wasserstein distance, we have

    W(_θ, _θ')    ≤∫_×x - yγ
       = _(x, y) ∼γ [x - y ] 
       = _z[g_θ(z) - g_θ'(z)]

If g is continuous in θ, then g_θ(z) →_θ→θ' g_θ'(z),
so g_θ - g_θ'→ 0 pointwise as functions of z. Since
 is compact, the distance of any two elements in it has to be uniformly bounded by
some constant M, and therefore g_θ(z) - g_θ'(z)≤ M for
all θ and z uniformly. By the bounded convergence theorem, we therefore
have

    W(_θ, _θ') ≤_z[g_θ(z) - g_θ'(z)] →_θ→θ' 0

Finally, we have that

    |W(_r, _θ) - W(_r, _θ') | ≤ W(_θ, _θ') →_θ→θ' 0

proving the continuity of W(_r, _θ).

Now let g be locally Lipschitz. Then, for a given
pair (θ, z) there is a constant L(θ, z)
and an open set U such that (θ, z) ∈ U,
such that for every (θ', z') ∈ U we have

    g_θ(z) - g_θ'(z')≤ L(θ, z) (θ - θ' + z - z')


By taking expectations and z'=z we

    _z[g_θ(z) - g_θ'(z)] 
      ≤θ - θ'_z[L(θ, z)]

whenever (θ', z) ∈ U. Therefore, we can define
U_θ = {θ' | (θ', z) ∈ U}. It's easy
to see that since U was open, U_θ is as well.
Furthermore, by assumption <ref>, we can
define L(θ) = _z[L(θ, z)] and achieve

    |W(_r, _θ) - W(_r, _θ')| 
    ≤ W(_θ, _θ') 
    ≤ L(θ) θ - θ'

for all θ' ∈ U_θ, meaning that W(_r, _θ)
is locally Lipschitz. This obviously implies that
W(_r, _θ) is everywhere continuous, and
by Radamacher's theorem we know it has to be differentiable
almost everywhere.

The counterexample for item 3 of the Theorem is indeed
Example <ref>.



We begin with the case of smooth nonlinearities. Since g is
C^1 as a function of (θ, z) then for any
fixed (θ, z) we have L(θ, Z) ≤∇_θ, xg_θ(z) + ϵ
is an acceptable local Lipschitz constant for all ϵ > 0.
Therefore, it suffices to prove

    _z ∼ p(z)[∇_θ, z g_θ(z)]< +∞

If H is the number of layers we know
that ∇_z g_θ(z) = ∏_k=1^H W_k D_k where
W_k are the weight matrices and D_k is are the diagonal Jacobians
of the nonlinearities. Let f_i:j be the application
of layers i to j inclusively (e.g. g_θ = f_1:H).
Then, ∇_W_k g_θ(z) = ((∏_i=k+1^H W_i D_i
) D_k ) f_1:k-1(z).
We recall that if L is the Lipschitz constant
of the nonlinearity, then D_i≤ L and
f_1:k-1(z)≤z L^k-1∏_i=1^k-1W_i. Putting this together,

    ∇_z, θ g_θ(z)   ≤∏_i=1^H W_i D_i
    + ∑_k=1^H ((∏_i=k+1^H W_i D_i ) D_k ) 
    f_1:k-1(z)
       ≤ L^H ∏_i=H^K W_i + ∑_k=1^H z L^H 
    (∏_i=1^k-1W_i)
    (∏_i=k+1^HW_i)

If C_1(θ) = L^H(∏_i=1^HW_i) and
C_2(θ) = ∑_k=1^H L^H 
(∏_i=1^k-1W_i)
(∏_i=k+1^HW_i) then

    _z ∼ p(z)[∇_θ, z g_θ(z)] 
    ≤ C_1(θ) + C_2(θ) _z ∼ p(z)[z] < +∞

finishing the proof



 
    
  * 
      
  * (δ(_n, ) → 0 ⇒ JS(_n,) → 0)   —   Let _m be the mixture distribution _m = 1/2_n
        + 1/2 (note that _m depends on n).
        It is easily verified that δ(_m, _n)
        ≤δ(_n, ), and in particular this tends to 0 (as
        does δ(_m, )). We now show this for completeness.
        Let μ be a signed measure,
        we define μ_TV = sup_A ⊆𝒳 |μ(A)|.
        for all Borel sets A.
        In this case,
        
    δ(_m, _n)    = _m - _n _TV
       = 1/2 + 1/2_n - _n _TV
       = 1/2 - _n _TV
       = 1/2δ(_n, ) ≤δ(_n, )

        
        Let f_n = d _n/d _m be the Radon-Nykodim
        derivative between _n and the mixture. Note that by
        construction for every Borel set A we have _n(A) ≤ 
        2 _m(A). If A = {f_n > 3} then we get
        
    _n(A) = ∫_A f_n _m ≥ 3 _m(A)

        which implies _m(A) = 0. This means that f_n
        is bounded by 3 _m(and therefore _n and
        )-almost everywhere. We could have done this
        for any constant larger than 2 but for our
        purposes 3 will sufice.
        
        Let ϵ > 0 fixed,
        and A_n = {f_n > 1 + ϵ}. Then, 
        
    _n(A_n) = ∫_A_n f_n _m ≥ (1 + ϵ) _m(A_n)

        Therefore,
        
    ϵ_m(A_n)    ≤_n(A_n) - _m(A_n) 
       ≤ |_n(A_n) - _m(A_n)| 
       ≤δ(_n, _m) 
       ≤δ(_n, ).

        Which implies _m(A_m) ≤1/ϵδ(_n, ). Furthermore,
        
    _n(A_n)    ≤_m(A_n) + |_n(A_n) - _m(A_n)| 
       ≤1/ϵδ(_n, ) + δ(_n, _m) 
       ≤1/ϵδ(_n, ) + δ(_n, ) 
       ≤(1/ϵ + 1) δ(_n, )

        We now can see that
        
    KL(_n _m)    = ∫log(f_n) _n 
       ≤log(1 + ϵ) + ∫_A_nlog(f_n) _n 
       ≤log(1 + ϵ) + log(3) _n(A_n) 
       ≤log(1 + ϵ) + log(3) (1/ϵ + 1) δ(_n, )

        Taking limsup we get 0 ≤lim sup KL(_n _m) ≤log(1 + ϵ)
        for all ϵ > 0, which means KL(_n _m) → 0.
        
        In the same way, we can define g_n = d /d _m, and
        
    2 _m({g_n > 3}) ≥({g_n > 3}) ≥ 3 _m({g_n > 3})

        meaning that _m({g_n > 3}) = 0 and therefore
        g_n is bounded by 3 almost everywhere for _n, _m
        and . With the same calculation, B_n = {g_n > 1 + ϵ} and
        
    (B_n) = ∫_B_n g_n _m ≥ (1 + ϵ) _m(B_n)

        so _m(B_n) ≤1/ϵδ(, _m) → 0, and therefore
        (B_n) → 0. We can now show
        
    KL(_m)    = ∫log(g_n) 
       ≤log(1 + ϵ) + ∫_B_nlog (g_n) 
       ≤log(1 + ϵ) + log(3) (B_n)

        so we achieve 0 ≤lim sup KL(_m) ≤log(1 + ϵ)
        and then KL(_m) → 0. Finally, we conclude
        
    JS(_n,) = 1/2 KL(_n _m) + 1/2 KL(_m) → 0

        
      
  * (JS(_n,) → 0 ⇒δ(_n, ) → 0)   —   by a simple
        application of the triangular and Pinsker's inequalities we get
        
    δ(_n, )    ≤δ(_n, _m) + δ(, _m) 
       ≤√(1/2 KL(_n _m)) + √(1/2 KL(_m))
       ≤ 2 √(JS(_n,))→ 0

      
    
  * This is a long known fact that W metrizes
      the weak* topology of (C(), ·_∞)
      on Prob(), and by definition this
      is the topology of convergence in distribution.
      A proof of this can be found (for example) in <cit.>.
    
  * This is a straightforward application of Pinsker's inequality
      
    δ(_n, ) ≤√(1/2 KL(_n ))→ 0 
    δ(, _n) ≤√(1/2 KL(_n))→ 0

      

    
  * This is trivial by recalling the fact that δ and W give the
      strong and weak* topologies on the dual of (C(), ·_∞)
      when restricted to Prob().
  



  Let us define
  
    V(f̃, θ)    = _x ∼_r[f̃(x)] - _x ∼_θ [f̃(x)] 
       = _x ∼_r[f̃(x)] - _z ∼ p(z) [f̃(g_θ(z))]

  where f̃ lies in = {f̃: → , f̃∈ C_b(), f̃_L ≤ 1} and
  θ∈^d.
  
  Since  is compact, we know 
  by the Kantorovich-Rubenstein duality <cit.> that
  there is an f ∈ that attains the value
  
    W(_r, _θ) = sup_f̃∈ V(f̃, θ) = V(f, θ)

  Let us define X^*(θ) = {f ∈: V(f, θ) = W(_r, _θ)}. By
  the above point we know then that X^*(θ) is non-empty. We know
  that by a simple envelope theorem (<cit.>, Theorem 1) that
  
    ∇_θ W(_r, _θ) = ∇_θ V(f, θ)

  for any f ∈ X^*(θ) when both terms are well-defined.
  
  Let f ∈ X^*(θ), which we knows exists
  since X^*(θ) is non-empty for all θ. Then, we get 
  
    ∇_θ W(_r, _θ)    = ∇_θ V(f, θ) 
       = ∇_θ[ _x ∼_r[f(x)] - _z ∼ p(z)[f(g_θ(z))] 
       = -∇_θ_z ∼ p(z)[f(g_θ(z))]

  under the condition that the first and last terms are well-defined.
  The rest of the proof will be dedicated to show that
  
    -∇_θ_z ∼ p(z)[f(g_θ(z))] = -_z ∼ p(z)[∇_θ f(g_θ(z))]

  when the right hand side is defined. For the reader who is
  not interested in such technicalities, he or she can skip the
  rest of the proof.
  
  Since f ∈, we know that it is 1-Lipschitz.
  Furthermore, g_θ(z) is locally
  Lipschitz as a function of (θ, z). Therefore, 
  f(g_θ(z)) is locally Lipschitz on (θ, z)
  with constants L(θ, z) (the same ones as g).
  By Radamacher's Theorem, f(g_θ(z)) has to be
  differentiable almost everywhere for (θ, z)
  jointly. Rewriting this, the set A = {(θ, z):
  f ∘ g is not differentiable} has
  measure 0. By Fubini's Theorem, this implies that
  for almost every θ the section A_θ
  = {z: (θ, z) ∈ A} has measure 0.
  Let's now fix a θ_0 such that 
  the measure of A_θ_0 is null (such
    as when the right hand side of equation (<ref>)
    is well defined). For this
  θ_0 we have ∇_θ f(g_θ(z))|_θ_0
  is well-defined for almost any z, and since p(z)
  has a density, it is defined p(z)-a.e. By assumption
  <ref> we know that
  
    _z ∼ p(z) [∇_θ f(g_θ(z))|_θ_0]
      ≤_z ∼ p(z) [L(θ_0, z)] < + ∞

  so _z ∼ p(z) [∇_θ f(g_θ(z))|_θ_0]
  is well-defined for almost every θ_0. Now, we can see
  
    _z∼ p(z)[f(g_θ(z))] - _z ∼ p(z)[f(g_θ_0(z))]
          - ⟨ (θ - θ_0), _z ∼ p(z) [∇_θ f(g_θ(z))|_θ_0]
          ⟩/θ - θ_0

  
    = _z ∼ p(z)[ f(g_θ(z)) - f(g_θ_0(z))
            - ⟨(θ - θ_0), ∇_θ f(g_θ(z))|_θ_0⟩/θ - θ_0]

  By differentiability, the term inside the integral converges p(z)-a.e. to 0
  as θ→θ_0. Furthermore,
  
    f(g_θ(z)) - f(g_θ_0(z))
          - ⟨(θ - θ_0), ∇_θ f(g_θ(z))|_θ_0⟩/θ - θ_0
    ≤θ - θ_0 L(θ_0, z)
          + θ - θ_0∇_θ f(g_θ(z))|_θ_0/θ - θ_0
    ≤ 2 L(θ_0, z)

  and since _z ∼ p(z)[2 L(θ_0, z)] < +∞ by assumption 1,
  we get by dominated convergence that Equation <ref> converges
  to 0 as θ→θ_0 so
  
    ∇_θ_z ∼ p(z) [f(g_θ(z))] = _z ∼ p(z) [∇_θ f(g_θ(z))]

  for almost every θ, and in particular when the right
  hand side is well defined.
  Note that the mere existance of the left
  hand side (meaning the differentiability a.e. of _z ∼ p(z)
  [f(g_θ(z))]) had to be proven, which we just did.




§ ENERGY-BASED GANS OPTIMIZE TOTAL VARIATION
 
In this appendix we show that under an optimal discriminator,
energy-based GANs (EBGANs) <cit.> optimize the total variation
distance between the real and generated distributions.

Energy-based GANs are trained in a similar fashion to GANs, only under
a different loss function. They have a discriminator D who tries to
minimize 

    L_D(D, g_θ) = _x ∼_r[D(x)] + _z ∼ p(z)[[m - D(g_θ(z))]^+]

for some m > 0 and [x]^+ = max(0, x) and a generator network g_θ that's trained to minimize

    L_G(D, g_θ) = _z ∼ p(z) [D(g_θ(z))] - _x ∼_r[D(x)]

Very importantly, D is constrained to be non-negative,
since otherwise the trivial solution for D would be to set everything to
arbitrarily low values. The original EBGAN paper used only _z ∼ p(z)[D(g_θ(z))] for
the loss of the generator, but this is obviously equivalent to our
definition since the term _x ∼_r[D(x)] does not dependent
on θ for a fixed discriminator (such as when backproping to the
generator in EBGAN
training) and thus minimizing one or the other is equivalent.

We say that a measurable function D^*: 𝒳→ [0, +∞)
is optimal for g_θ (or _θ) if L_D(D^*, g_θ) ≤ L_D(D, g_θ) for all
other measurable functions D. We show that such a discriminator
always exists for any two distributions _r and _θ,
and that under such a discriminator, L_G(D^*, g_θ) is
proportional to δ(_r, _θ). As a simple corollary,
we get the fact that L_G(D^*, g_θ) attains its minimum
value if and only if δ(_r, _θ) is at its
minimum value, which is 0, and _r = _θ
(Theorems 1-2 of <cit.>).

 Let _r be a the real data distribution over
a compact space 𝒳.
Let g_θ: 𝒵→𝒳 be a
measurable function (such as any neural network). Then, 
an optimal discriminator D^* exists for _r and
_θ, and 

    L_G(D^*, g_θ) = m/2δ(_r, _θ)


 First, we prove that there exists an optimal
discriminator. Let D: 𝒳→ [0, +∞)
be a measurable function, then D'(x) := min(D(x), m) is
also a measurable function, and L_D(D', g_θ) ≤ L_D(D, g_θ).
Therefore, a function D^*: 𝒳→ [0, +∞) is
optimal if and only if D^*' is. Furthermore, it is optimal if and
only if L_D(D^*, g_θ) ≤ L_D(D, g_θ) for all D: 𝒳→ [0, m]. We are then interested to see if there's an
optimal discriminator for the problem
min_0 ≤ D(x) ≤ m L_D(D, g_θ).

Note now that if 0 ≤ D(x) ≤ m we have

    L_D(D, g_θ)    = _x ∼_r[D(x)] + _z ∼ p(z)[[m - D(g_θ(z))]^+] 
       = _x ∼_r[D(x)] + _z ∼ p(z)[m - D(g_θ(z))] 
       = m + _x ∼_r[D(x)] - _z ∼ p(z)[D(g_θ(z))] 
       = m + _x ∼_r[D(x)] - _x ∼_θ[D(x)]

Therefore, we know that

    inf_0 ≤ D(x) ≤ m L_D(D, g_θ)    = m + inf_0 ≤ D(x) ≤ m_x ∼_r[D(x)] - _x ∼_θ[D(x)] 
       = m + inf_-m/2≤ D(x) ≤m/2_x ∼_r[D(x)] - _x ∼_θ[D(x)] 
       = m + m/2inf_-1 ≤ f(x) ≤ 1_x ∼_r[f(x)] - _x ∼_θ[f(x)]

The interesting part is that

    inf_-1 ≤ f(x) ≤ 1_x ∼_r[f(x)] - _x ∼_θ[f(x)] = - δ(_r, _θ)

and there is an f^*: → [-1,1] such that _x ∼_r[f^*(x)] -
_x ∼_θ[f^*(x)] = - δ(_r, _θ). This is a long known fact,
found for example in <cit.>, but we prove it later for completeness. In that case,
we define D^*(x) = m/2f^*(x) + m/2. We then have 0 ≤ D(x) ≤ m and

    L_D(D^*, g_θ) 
           = m + _x ∼_r[D^*(x)] - _x ∼_θ[D^*(x)] 
       = m + m/2_x ∼_r[D^*(x)] - _x ∼_θ[f^*(x)] 
       = m - m/2δ(_r, _θ) 
       = inf_0 ≤ D(x) ≤ m L_D(D, g_θ)

This shows that D^* is optimal and L_D(D^*, g_θ) = m - m/2δ(_r, _θ). Furthermore,

    L_G(D^*, g_θ)    = _z ∼ p(z)[D^*(g_θ(z))] - _x ∼_r[D^*(x)] 
       = -L_D(D^*, g_θ) + m 
       = m/2δ(_r, _g)

concluding the proof.

For completeness, we now show a proof for equation (<ref>)
and the existence of said f^* that attains the value of the infimum.
Take μ = _r - _θ,
which is a signed measure, and (P, Q) its Hahn decomposition.
Then, we can define f^* := 1_Q - 1_P.
By construction, then

    EE_x ∼_r[f^*(x)] - _x ∼_θ[f^*(x)]
       = ∫ f^* μ̣= μ(Q) - μ(P) 
       = -(μ(P) - μ(Q)) = -μ_TV
       = -_r - _θ_TV
       = -δ(_r, _θ)

Furthermore, if f is bounded between -1 and 1, we get

    |_x ∼_r[f(x)] - _x ∼_θ[f(x)] |
         = |∫ f _r - ∫ f _θ| 
       = |∫ f μ̣| 
       ≤∫ |f| |̣μ| ≤∫ 1 |̣μ|
       =|μ|(𝒳) = μ_TV = δ(_r, _θ)

Since δ is positive, we can conclude _x ∼_r[f(x)] - _x ∼_θ[f(x)] ≥ -δ(_r, _θ).





§ GENERATOR'S COST DURING NORMAL GAN TRAINING
 




§ SHEETS OF SAMPLES
 
top=0cm, bottom=0cm














