McGan: Mean and Covariance Feature Matching GAN
Youssef Mroueh Tom Sercu Vaibhava Goel
 

[
McGan: Mean and Covariance Feature Matching GAN














equal*


Youssef  Mrouehequal,aif,to
Tom Sercuequal,aif,to
Vaibhava Goelto


aifAI Foundations.
IBM T.J. Watson Research Center, NY, USA
toWatson Multimodal Algorithms and Engines Group.
IBM T.J. Watson Research Center, NY, USA


Youssef Mrouehmroueh@us.ibm.com




Generative Adversarial Networks, Integral probability metrics, McGan, Covariance Feature Matching

0.3in
]










 


 
We introduce new families of  Integral Probability Metrics (IPM) for training Generative Adversarial Networks (GAN).
Our IPMs are based on matching  statistics of distributions  embedded in a finite dimensional feature space. Mean and covariance feature matching IPMs allow for stable training of GANs, which we will call McGan. McGan minimizes a meaningful loss between distributions. 

 




§ INTRODUCTION

Unsupervised learning of distributions is an important problem, in which we aim to learn underlying features that unveil the hidden the structure in the data.
The classic approach to learning distributions is by explicitly parametrizing the data likelihood and fitting this model by maximizing the likelihood of the real data.
An alternative recent approach is to learn a generative model of the data without explicit parametrization of the likelihood. 
Variational Auto-Encoders (VAE) <cit.> and Generative Adversarial Networks (GAN) <cit.> fall under this category. 

We focus on the GAN approach.
In a nutshell GANs learn a generator of the data via a min-max game between the generator and a discriminator, which learns to distinguish between “real” and “fake” samples.
In this work we focus on the objective function that is being minimized between the learned generator distribution ℙ_θ and the real data distribution ℙ_r.

The original work of <cit.> showed that in GAN  this objective is the Jensen-Shannon divergence. <cit.> showed that other φ-divergences can be successfully used. The Maximum Mean Discrepancy objective (MMD) for GAN training was proposed in <cit.>. As shown empirically in <cit.>, one can train the GAN discriminator using the objective of <cit.> while training the generator using mean feature matching. An energy based objective for GANs was also developed recently <cit.>. Finally, closely related  to our paper, the recent work Wasserstein GAN (WGAN) of <cit.> proposed to use the Earth Moving distance (EM) as an objective for training GANs. Furthermore <cit.> show that the EM objective has many advantages as the loss function correlates with the quality of the generated samples and the mode dropping problem is reduced in WGAN.    

In this paper, inspired by the MMD distance and the kernel mean embedding of distributions <cit.> we propose to embed distributions in a finite dimensional feature space and to match them based on their mean and covariance feature statistics. Incorporating first and second order statistics has a better chance to  capture the various modes of the distribution. While mean feature matching was empirically used in <cit.>, we show in this work that it is theoretically grounded: similarly to the EM distance in <cit.>, mean and covariance feature matching of two distributions can be written as a distance in the framework of Integral Probability Metrics (IPM) <cit.>. 
To match the means, we can use any ℓ_q norm, hence we refer to mean matching IPM, as IPM_μ,q. 
For matching covariances, in this paper we consider the Ky-Fan norm, which can be computed cheaply without explicitly constructing the full covariance matrices, and refer to the corresponding IPM as IPM_Σ.  

Our technical contributions can be summarized as follows:

a) We show in Section <ref> that the ℓ_q mean feature matching  IPM_μ,q has two equivalent primal and dual formulations and can be used as an objective for GAN training in both formulations.

b) We show in Section <ref> that the parametrization used in Wasserstein GAN corresponds to ℓ_1 mean feature matching GAN (IPM_μ,1 GAN in our framework). 

c)  We show in Section <ref> that the covariance feature matching IPM_Σ admits also two dual formulations, and can be used as an objective for GAN training.

d)  Similar to Wasserstein GAN, we show that mean feature matching  and covariance matching GANs (McGan) are stable to train, have a reduced mode dropping and the IPM loss correlates with the quality of the generated samples.



§ INTEGRAL PROBABILITY METRICS

We define in this Section IPMs as a distance between distribution. Intuitively each IPM finds  a “critic" f <cit.> which maximally discriminates between the distributions. 


 §.§ IPM Definition

Consider a compact space X in ℝ^d. Let ℱ be a set of measurable and bounded real valued  functions on X. Let 𝒫(X) be the set of measurable probability distributions on X. Given two probability distributions ℙ,ℚ∈𝒫(X), the Integral probability metric (IPM) indexed by the function space ℱ is defined as follows <cit.>:

    d_ℱ(ℙ,ℚ)= sup_f ∈ℱ | x∼ℙ𝔼 f(x) -x∼ℚ𝔼f(x)|.

In this paper we are interested in symmetric function spaces ℱ, i.e ∀ f ∈ℱ, -f ∈ℱ, hence we can write the IPM in that case without the absolute value:

    d_ℱ(ℙ,ℚ)= sup_f ∈ℱ{x∼ℙ𝔼 f(x) -x∼ℚ𝔼f(x)}.
 
It is easy to see that d_ℱ defines a pseudo-metric over 𝒫(X). (d_ℱ non-negative, symmetric and satisfies the triangle inequality. A pseudo metric means that d_ℱ(ℙ,ℙ)=0 but d_ℱ(ℙ,ℚ)=0 does not necessarily imply ℙ=ℚ).




By choosing ℱ appropriately <cit.>, various distances between probability measures can be defined. In the next subsection following <cit.> we show how to use IPM to learn generative models of distributions, we then specify a special set of functions ℱ that makes the learning tractable.


 §.§ Learning Generative Models with IPM 

In order to learn a generative model of  a distribution ℙ_r∈𝒫(X), we learn a function 
    g_θ: Z⊂ℝ^n_z→X,

such that for z∼ p_z, the distribution of g_θ(z) is close to the real data distribution ℙ_r, where p_z is a fixed distribution on Z (for instance z∼𝒩(0,I_n_z)). 
Let ℙ_θ be the distribution of g_θ(z),z∼ p_z. Using an IPM indexed by a function class ℱ we shall solve therefore the following problem:

    min_g_θ d_ℱ(ℙ_r,ℙ_θ)

Hence this amounts to solving the following min-max problem:

    min_g_θsup_f∈ℱx∼ℙ_r𝔼f(x) -z ∼ p_z𝔼f(g_θ(z))

Given samples {x_i ,1… N} from ℙ_r and samples {z_i ,1… M} from p_z we shall solve the following empirical problem:

    min_g_θsup_f∈ℱ1/N∑_i=1^N f(x_i) - 1/M∑_j=1^M f(g_θ(z_j)),

in the following we consider for simplicity M=N.



§ MEAN FEATURE MATCHING GAN

In this Section we introduce a class of functions ℱ having the form vΦ_ω(x), where  vector v ∈ℝ^m and  Φ_ω: X→ℝ^m a non linear feature map (typically parametrized by a neural network). We show in this Section that the IPM defined by this function class corresponds to the distance between the mean of the distribution in  the Φ_ω space.  


 §.§ IPM_μ,q: Mean Matching IPM


More formally consider the following function space: 
-2em

    ℱ_v,ω,p={ f(x)= ⟨ v,Φ_ω(x)⟩ | v ∈ℝ^m, v_p≤ 1,
       Φ_ω: X→ℝ^m, ω∈Ω},

where ._p is the ℓ_p norm.
ℱ_v,ω,p is the space of bounded linear functions defined in the non linear feature space induced by the parametric feature map  Φ_ω. Φ_ω is typically a multi-layer neural network. The parameter space Ω is chosen so that  the function space ℱ is bounded. Note that for a given ω, ℱ_v,ω,p  is a finite dimensional Hilbert space.

We recall here  simple definitions on dual norms that will be necessary for the analysis in this Section.
Let p,q ∈ [1,∞], such that 1/p+1/q=1. 

By duality of norms we have:
x_q=max_v, v_p≤ 1vx and the  Holder inequality:
|xy| ≤x_py_q.

From Holder inequality  we obtain the following bound:


    |f(x)|=|vΦ_ωx|≤v_pΦ_ω(x)_q≤Φ_ω(x)_q.
 

To ensure that f is bounded, it is enough to consider Ω such that Φ_ω(x)_q≤ B, ∀  x ∈X. Given that the space X is bounded it is sufficient to control the norm of the weights and biases of the neural network Φ_ω by regularizing the ℓ_∞ (clamping) or ℓ_2 norms (weight decay) to ensure the boundedness of  ℱ_v,ω,p.

Now that we ensured the boundedness of ℱ_v,ω,p , we  look at its corresponding IPM:
 
    d_ℱ_v,ω,p(ℙ,ℚ)=sup_f ∈ℱ_v,ω,px∼ℙ𝔼 f(x) -x∼ℚ𝔼f(x) 
       =   max_ω∈Ω, v,||v||_p≤1 vx∼ℙ𝔼Φ_ω(x)- x∼ℚ𝔼Φ_ω(x) 
       =   max_ω∈Ω[ max_ v,||v||_p≤1 vx∼ℙ𝔼Φ_ω(x)- x∼ℚ𝔼Φ_ω(x) ]
       =   max_ω∈Ωμ_ω(ℙ)-μ_ω(ℚ)_q,

where we used the linearity of the function class and expectation in the first equality and the definition of the dual norm ._q in the last equality and our
definition of  the mean feature embedding of  a distribution ℙ∈𝒫(X):

    μ_ω(ℙ)= x∼ℙ𝔼[ Φ_ω(x)] ∈ℝ^m.



We see that the IPM indexed by ℱ_v,ω,p, corresponds to the Maximum mean feature Discrepancy between the two distributions. Where the maximum is taken over the parameter set Ω, and the discrepancy is measured in the ℓ_q sense between the mean feature embedding of ℙ and ℚ. In other words this IPM is equal to the worst case ℓ_q distance between mean feature embeddings of distributions. We refer in what follows to d_ℱ_v,ω,p as IPM_μ,q.



 §.§ Mean Feature Matching GAN

We turn now to the problem of learning generative models with IPM_μ,q.
Setting ℱ to ℱ_v,ω,p in Equation (<ref>) yields to the following min-max problem for learning generative models:

    min_g_θmax_ω∈Ωmax_v,||v||_p≤ 1ℒ_μ(v,ω,θ),

where
 -0.3in

    ℒ_μ(v,ω,θ)= vx∼ℙ_r𝔼Φ_ω(x)- z ∼ p_z𝔼Φ_ω(g_θ(z)) ,

or equivalently using the dual norm:

    min_g_θmax_ω∈Ωμ_ω(ℙ_r)-μ_ω(ℙ_θ)_q ,

where μ_ω(ℙ_θ)=z ∼ p_z𝔼Φ_ω(g_θ(z)).

We refer to formulations (<ref>) and (<ref>) as primal and dual formulation respectively.

The dual formulation in Equation (<ref>) has a simple interpretation as an adversarial learning game: while the feature space Φ_ω tries to  map the mean feature embeddings of the real distribution ℙ_r and the fake  distribution ℙ_θ to be far apart (maximize the ℓ_q distance between the mean embeddings), the generator g_θ tries to put them close one to another. Hence we refer to this IPM as mean matching IPM.

We devise empirical estimates of both formulations in Equations (<ref>) and (<ref>), given samples {x_i,i=1… N} from ℙ_r, and {z_i,i=1… N} from p_z. The primal formulation (<ref>) is more amenable to stochastic gradient descent since the expectation operation appears in a linear way in the cost function of Equation (<ref>), while it is non linear in the cost function of the dual formulation (<ref>) (inside the norm). We give here the empirical estimate of the primal  formulation by giving empirical estimates  ℒ̂_μ(v,ω,θ) of the primal cost function:

    (P_μ):min_g_θ   ω∈Ω
    v,||v||_p≤1max    v1/N∑_i=1^NΦ_ω(x_i)- 1/N∑_i=1^N Φ_ω(g_θ(z_i))



An empirical estimate of the dual formulation can be also given as follows:

    (D_μ):min_g_θ   ω∈Ωmax   1/N∑_i=1^NΦ_ω(x_i)-1/N∑_i=1^N Φ_ω(g_θ(z_i)) _q


In what follows we refer to the problem given in (P_μ) and (D_μ) as ℓ_q Mean Feature Matching GAN.
Note that while (P_μ) does not need real samples for optimizing the generator, (D_μ) does need samples from real and fake. 
Furthermore we will need a large minibatch of real data in order to get a good estimate of the expectation. This makes the primal formulation more appealing computationally.





 §.§ Related Work
 
We show in this Section that several previous works on GAN, can be written within the ℓ_q mean feature matching IPM (IPM_μ,q) minimization framework:

a) Wasserstein GAN (WGAN): <cit.> recently introduced Wasserstein GAN. While the main motivation of this paper is to consider the IPM indexed by Lipchitz functions on X, we show that the particular parametrization considered in <cit.>  corresponds to a  mean feature matching IPM.

Indeed <cit.> consider the function set parametrized by a convolutional neural network with a linear output layer and weight clipping.
Written in our notation, the last linear layer corresponds to v, and the convolutional neural network below corresponds to Φ_ω.
Since v and ω are simultaneously clamped, this corresponds to restricting v to be in the ℓ_∞ unit ball, and to define in Ω constraints on the ℓ_∞ norms of ω. In other words <cit.> consider functions in ℱ_v,ω,p, where p=∞ . 
Setting p=∞ in Equation (<ref>), and q=1 in Equation (<ref>), we see that in WGAN we are minimizing d_ℱ_v,ω,∞, that corresponds to ℓ_1 mean feature matching GAN.
 
b) MMD GAN: Let ℋ be a Reproducing Kernel Hilbert Space (RKHS) with k its reproducing kernel. For any valid PSD kernel k there exists an infinite dimensional feature map Φ: X→ℋ such that: k(x,y)=⟨Φ(x),Φ(y)⟩_ℋ. For an RKHS Φ is noted usually k(x,.) and satisfies the reproducing proprety:

    f(x)=⟨ f,Φ(x)⟩_ℋ,  for all   f ∈ℋ.

 Setting ℱ={ f | || f ||_ℋ≤ 1} in Equation (<ref>) the IPM  d_ℱ has a simple expression:

    d_ℱ(ℙ,ℚ)   =   sup_f, ||f||_ℋ≤ 1{⟨ f,  x∼ℙ𝔼Φ (x) -x∼ℚ𝔼Φ(x)⟩}
       =    |  |μ(ℙ) -μ(ℚ)| |_ℋ,

where μ(ℙ)=x∼ℙ𝔼Φ (x) ∈ℋ is the so called kernel mean embedding <cit.>. d_ℱ in this case is the so called Maximum kernel Mean Discrepancy (MMD) <cit.> . Using the reproducing property MMD has a closed form in term of the kernel k. Note that IPM_μ,2 is a special case of MMD when the feature map is finite dimensional, with the main difference that the feature map is fixed in case of MMD and learned in the case of IPM_μ,2.  
<cit.> showed that GANs can be learned using MMD with a fixed gaussian kernel.

c) Improved GAN: Building on the pioneering work of <cit.>, <cit.> suggested to learn the discriminator with the binary cross entropy criterium of GAN while  learning the generator with ℓ_2 mean feature matching. The main difference of our IPM_μ,2 GAN is that both “discriminator" and “generator" are learned using the mean feature matching criterium, with additional constraints on Φ_ω.
 

 
 

§ COVARIANCE FEATURE MATCHING GAN

 

 §.§ IPM_Σ: Covariance Matching IPM

 As follows from our discussion of mean matching IPM comparing two distributions amounts to comparing  a first order statistics, the mean of their feature embeddings.
Here we ask the question how to incorporate second order statistics, i.e covariance information of feature embeddings.
  
In this Section we will provide a function space ℱ  such that the IPM in Equation (<ref>) captures second order information. 
Intuitively a distribution of points represented in a feature space can be approximately captured by its mean and its covariance.
Commonly in unsupervised learning, this covariance is approximated by its first k principal components (PCA directions), which capture the directions of maximal variance in the data.
Similarly, the metric we define in this Section will find k directions that maximize the discrimination between the two covariances.
Adding second order information would enrich the discrimination power of the feature space (See Figure <ref>).



 
 

This intuition motivates the following function space of bilinear functions  in Φ_ω :



    ℱ_U,V,ω={f(x)=∑_j=1^k u_jΦ_ω(x)v_jΦ_ω(x)
       {u_j},{v_j}∈ℝ^m orthonormal  j=1… k,  ω∈Ω}.

Note that the set ℱ_U,V,ω is symmetric and hence the IPM indexed by this set  (Equation (<ref>)) is well defined. It is easy to see that ℱ_U,V,ω can be written as:

    ℱ_U,V,ω={ f(x)= U^⊤Φ_ω(x)V^⊤Φ_ω(x) ) |
        U,V ∈ℝ^m× k, U^⊤U=I_k,V^⊤V=I_k,  ω∈Ω}

the parameter set Ω is such that the function space remains bounded. 
Let 
    Σ_ω(ℙ)= x∼ℙ𝔼Φ_ω(x)Φ_ω(x)^⊤,

be the uncentered feature covariance embedding of ℙ. It is easy to see that x∼ℙ𝔼f(x) can be written in terms of U,V, and Σ_ω(ℙ):

    x∼ℙ𝔼f(x)=x∼ℙ𝔼U^⊤Φ(x)V^⊤Φ(x)=Trace(U^⊤Σ_ω(ℙ)V).

-0.14 in
 For a matrix A ∈ℝ^m× m, we note by σ_j(A) the singular value of A, j=1… m in descending order.
The 1-schatten norm or the nuclear norm is defined as the sum of singular values, A_*=∑_j=1^m σ_j. We note by [A]_k the k-th rank approximation of A.
We note O_m,k={M∈ℝ^m× k | M^⊤M=I_k}.
Consider the IPM induced by this function set. Let ℙ,ℚ∈𝒫(X) we have:

    d_ℱ_U,V,ω(ℙ,ℚ)=sup_f ∈ℱ_U,V,ωx∼ℙ𝔼f(x) - x∼ℚ𝔼f(x)  
       =ω∈Ω
     U,V ∈O_m,kmax x∼ℙ𝔼f(x)-x∼ℚ𝔼f(x) 
       = ω∈Ωmax    U,V ∈O_m,kmax  Trace[U^⊤( Σ_ω(ℙ)-Σ_ω(ℚ) )V]
       = max_ω∈Ω∑_j=1^k σ_j(Σ_ω(ℙ)-Σ_ω(ℚ))
       = max_ω∈Ω[Σ_ω(ℙ)- Σ_ω(ℚ)]_k_*,

where we used the variational definition of singular values and the definition of the nuclear norm.
Note that U,V are the left and right singular vectors of Σ_ω(ℙ)- Σ_ω(ℚ).
Hence d_ℱ_U,V,ω measures the worst case distance between the covariance feature embeddings of the two distributions, this distance is measured with the  Ky Fan k-norm (nuclear norm of truncated covariance difference). Hence we call this IPM covariance matching IPM, IPM_Σ.




 §.§ Covariance Matching GAN

Turning now to the problem of learning a generative model g_θ of ℙ_r∈𝒫(X) using  IPM_Σ we shall solve:

    min_g_θd_ℱ_U,V,ω(ℙ_r,ℙ_θ),

this has the following primal formulation:

    min_g_θ         ω∈Ω, U,V ∈O_m,kmax      ℒ_σ(U,V,ω,θ)  ,
 
-0.2in 

    where ℒ_σ(U,V,ω,θ)=x∼ℙ_r𝔼UΦ_ω(x)VΦ_ω(x))
       -z∼ p_z𝔼UΦ_ω(g_θ(z))VΦ_ω(g_θ(z)),

or equivalently the following dual formulation: 
 
    min_g_θmax_ω∈Ω[Σ_ω(ℙ_r)- Σ_ω(ℙ_θ)]_k_*,

 where Σ_ω(ℙ_θ)=𝔼_z∼ p_zΦ_ω(g_θ(z))Φ_ω(g_θ(z))^⊤.
 
The dual formulation in Equation (<ref>) shows that learning generative models with IPM_Σ, consists in an adversarial game between the feature map and the generator, when the feature maps tries to maximize the distance between the  feature covariance  embeddings of the distributions, the generator tries to minimize this distance. Hence we call learning with IPM_Σ, covariance matching GAN.

 
We give here an empirical estimate of the primal formulation in  Equation (<ref>) which is amenable to stochastic gradient. The dual requires nuclear norm minimization and is more involved.
 Given {x_i, x_i∼ℙ_r}, and {z_j, z_j ∼ p_z}, the covariance matching GAN can be written as follows:

    min_g_θ         ω∈Ω,U,V ∈O_m,kmax      ℒ̂_σ(U,V,ω,θ)  ,
 

   where ℒ̂_σ(U,V,ω,θ) =1/N ∑_i=1^N UΦ_ω(x_i)VΦ_ω(x_i)
 
    -1/N ∑_j=1^N U Φ_ω(g_θ(z_j))VΦ_ω(g_θ(z_j)).





 §.§ Mean and Covariance Matching GAN

In order to match first and second order statistics we propose the following simple extension:

    min_g_θω∈Ω,v,||v||_p≤ 1
    U,V ∈O_m,kmaxℒ_μ(v,ω,θ)+ℒ_σ(U,V,ω,θ),

that has a simple dual adversarial game interpretation

    min_g_θω∈Ωmaxμ_ω(ℙ)-μ_ω(ℙ_θ)_q+ [Σ_ω(ℙ_r)- Σ_ω(ℙ_θ)]_k_*,

where the discriminator finds a feature space that discriminates between means and variances of real and fake, and the generator tries to match the real statistics.
We can also give empirical estimates of the primal formulation similar to expressions given in the paper.



§ ALGORITHMS

We present in this Section our algorithms for mean and covariance feature matching GAN (McGan) with IPM_μ,q and IPM_Σ.

Mean Matching GAN. Primal P_μ: We give in Algorithm <ref> an algorithm for solving the primal IPM_μ,q GAN (P_μ). Algorithm <ref> is adapted from <cit.> and corresponds to their algorithm for p=∞. The main difference is that we allow projection of v on different ℓ_p balls, and we maintain the clipping of ω to ensure boundedness of Φ_ω. For example for  p=2, proj_B_ℓ_2(v)=min(1,1/v_2)v. For p=∞ we obtain the same clipping in <cit.> proj_B_ℓ_∞(v)=clip(v,-c,c) for c=1.

Dual D_μ: We give in Algorithm <ref> an algorithm for solving the dual formulation IPM_μ,q GAN (D_μ). As mentioned earlier we need samples from “real" and “fake" for training both generator and  the “critic" feature space.

Covariance Matching GAN. Primal P_Σ: We give in Algorithm <ref> an algorithm for solving the primal of IPM_Σ GAN (Equation (<ref>)). The algorithm performs a stochastic gradient  ascent on (ω,U,V) and a descent on θ.  We maintain clipping on ω to ensure boundedness of Φ_ω, and perform a QR retraction on the Stiefel manifold O_m,k <cit.>, maintaining orthonormality of U and V. 









§ EXPERIMENTS


We train McGan for image generation with both Mean Matching and Covariance Matching objectives. 
We show generated images on the labeled faces in the wild (lfw) <cit.>, LSUN bedrooms <cit.>, and cifar-10 <cit.> datasets.

It is well-established that evaluating generative models is hard <cit.>.
Many GAN papers rely on a combination of samples for quality evaluation, supplemented by a number of heuristic quantitative measures.
We will mostly focus on training stability by showing plots of the loss function,
and will provide generated samples to claim comparable sample quality between methods, but we will avoid claiming better sample quality.
These samples are all generated at random and are not cherry-picked.

The design of g_θ and Φ_ω are following DCGAN principles <cit.>, with both g_θ and Φ_ω being a convolutional network with batch normalization <cit.> and ReLU activations.
Φ_ω has output size bs × F × 4 × 4. 
The inner product can then equivalently be implemented as  or .
We generate 64×64 images for lfw and LSUN and 32×32 images on cifar, and train with minibatches of size 64.
We follow the experimental framework and implementation of <cit.>, where we ensure the boundedness of Φ_ω by clipping the weights pointwise to the range [-0.01, 0.01].


Primal versus dual form of mean matching.
To illustrate the validity of both the primal and dual formulation,
we trained mean matching GANs both in the primal and dual form, see respectively Algorithm <ref> and <ref>.
Samples are shown in Figure <ref>.
Note that optimizing the dual form is less efficient and only feasible for mean matching, not for covariance matching.
The primal formulation of IPM_μ,1 GAN corresponds to clipping v, i.e. the original WGAN, while for IPM_μ,2 we divide v by its ℓ_2 norm if it becomes larger than 1.
In the dual, 

for q=2 we noticed little difference between maximizing the ℓ_2 norm or its square. 





We observed that the default learning rates from WGAN (5e-5) are optimal for both primal and dual formulation.
Figure <ref> shows the loss (i.e. IPM estimate) dropping steadily for both the primal and dual formulation independently of the choice of the ℓ_q norm.

We also observed that during the whole training process, samples generated from the same noise vector across iterations, remain similar in nature (face identity, bedroom style), while details and background will evolve.
This qualitative observation indicates valuable stability of the training process.

For the dual formulation (Algorithm <ref>), we confirmed the hypothesis that we need a good estimate of μ_ω(ℙ_r) in order to compute the gradient of the generator ∇_θ: 
we needed to increase the minibatch size of real threefold to 3×64.





Covariance GAN.
We now experimentally investigate the IPM defined by covariance matching.
For this section and the following, we use only the primal formulation, i.e. with explicit u_j and v_j orthonormal (Algorithm <ref>).
Figure <ref> and <ref> show samples and loss from lfw and LSUN training respectively.
We use Algorithm <ref> with k=16 components. 
We obtain samples of comparable quality to the mean matching formulations (Figure <ref>),
and we found training to be stable independent of hyperparameters like number of components k varying between 4 and 64.






Covariance GAN with labels and conditioning.


Finally, we conduct experiments on the cifar-10 dataset, where we will leverage the additional label information
by training a GAN with conditional generator g_θ(z,y) with label y ∈ [1, K] supppplied as one-hot vector concatenated with noise z.
conditionalGAN,infogan,acgan
Similar to Infogan <cit.> and AC-GAN <cit.>, we add a new output layer, S ∈ℝ^K× m and will write the logits SΦ_ω(x)∈ℝ^K.
We now optimize a combination of the IPM loss and the cross-entropy loss CE(x,y; S, Φ_ω) = -log[Softmax(SΦ_ω(x))_y ].

The critic loss becomes
ℒ_D = ℒ̂_σ - λ_D 1/N∑_(x_i,y_i) ∈lab CE(x_i,y_i;S,Φ_ω), with hyper-parameter λ_D.
We now sample three minibatches for each critic update: a labeled batch for the CE term, and for the IPM a real unlabeled + generated batch.

The generator loss (with hyper-param λ_G) becomes: 
ℒ_G = ℒ̂_σ  + λ_G 1/N∑_z_i ∼ p_z, y_i ∼ p_y CE(g_θ(z_i,y_i),y_i;S,Φ_ω)
which still only requires a single minibatch to compute.

We confirm the improved stability and sample quality of objectives including covariance matching with inception scores <cit.> in Table <ref>.
Samples corresponding to the best inception score (Sigma) are given in Figure <ref>.
Using the code released with WGAN <cit.>, these scores come from the DCGAN model with  (deeper generator and discriminator) .
More samples are in appendix with combinations of Mean and Covariance Matching.

Notice rows corresponding to recognizable classes, while the noise z (shared within each column) clearly determines other elements of the visual style like dominant color, across label conditioning.



§ DISCUSSION

We noticed the influence of clipping on the capacity of the critic: a higher number of feature maps was needed to compensate for clipping.

The question remains what alternatives to clipping of Φ_ω can ensure the boundedness.
For example, we succesfully used an ℓ_2 penalty on the weights of Φ_ω. Other directions are to explore geodesic distances between the covariances <cit.>, and extensions of the IPM framework to the multimodal setting <cit.>.



icml2017




Supplementary Material for McGan: Mean and Covariance Feature Matching GAN


1em

Youssef  Mrouehequal,aif,to
Tom Sercuequal,aif,to
Vaibhava Goelto





§ SUBSPACE MATCHING INTERPRETATION OF COVARIANCE MATCHING GAN 

Let Δ_ω= Σ_ω(ℙ)-Σ_ω(ℚ). Δ_ω  is a symmetric matrix but not PSD, which has the property that its eigenvalues λ_j are related to its singular values as given by: σ_j=|λ_j| and its left and right singular vectors coincides with its eigenvectors and  satisfy the following equality u_j= sign(λ_j)v_j.
One can ask here if we can avoid having both U,V in the definition of IPM_Σ since at the optimum u_j=± v_j. One could consider δ E_ω(ℙ_r,ℙ_θ) defined as follows: 

    ω∈Ω,U ∈O_m,kmax  Energy in the subspace
    of real datax∼ℙ_r𝔼UΦ_ω(x)^2-Energy in the subspace
    of fake dataz∼ p_z𝔼U Φ_ω(g_θ(z))^2,

and then solve for min_g_θδ E_ω(ℙ_r,ℙ_θ). 
Note that:

    δ E_ω(ℙ_r,ℙ_θ)    =ω∈Ω, U∈O_m,kmax Trace(U^⊤(Σ_ω(ℙ_r)-Σ_ω(ℙ_θ))U)
       = max_ω∈Ω∑_i=1^k λ_i(Δ_ω)

δ E_ω is not symmetric furthermore the sum of those eigenvalues is not guaranteed to be positive and hence δ E_ω  is not guaranteed to be non negative, and hence does not define an IPM. Noting that  σ_i(Δ_ω)=|λ_i(Δ_ω)|,we have that:

    IPM_Σ(ℙ_r,ℙ_θ)=∑_i=1^k σ_i(Δ_ω)≥∑_i=1^k λ_i(Δ_ω)=δ E_ω(ℙ_r,ℙ_θ).

Hence δ E is not an IPM but can be optimized as a lower bound of the IPM_Σ. This would have an energy interpretation as in the energy based GAN introduced recently <cit.>: the discriminator defines a  subspace that has higher energy on real data than fake data, and the generator maximizes his energy in this subspace. 


§ MEAN AND COVARIANCE MATCHING LOSS COMBINATIONS

We report below samples for McGan, with different IPM_μ,q and IPM_Σ combinations. All results are reported for the same architecture choice for generator and discriminator, which produced qualitatively good samples with IPM_Σ (Same one reported in Section 6 in the main paper).
Note that in Figure <ref> with the same hyper-parameters and architecture choice, WGAN failed to produce good sample. In other configurations training converged.











