Evaluating Generative Adversarial Networks on Explicitly Parameterized Distributions
Shayne O'Brien Matt Groh Abhimanyu Dubey




 Evaluating Generative Adversarial Networks on Explicitly Parameterized Distributions 
    Shayne O'Brien, Matt Groh, Abhimanyu Dubey

{, , }@mit.edu 

Massachusetts Institute of Technology


    December 30, 2023
========================================================================================================



The true distribution parameterizations of commonly used image datasets are inaccessible. Rather than designing metrics for feature spaces with unknown characteristics, we propose to measure GAN performance by evaluating on explicitly parameterized, synthetic data distributions. As a case study, we examine the performance of 16 GAN variants on six multivariate distributions of varying dimensionalities and training set sizes. In this learning environment, we observe that: GANs exhibit similar performance trends across dimensionalities; learning depends on the underlying distribution and its complexity;  the number of training samples can have a large impact on performance; evaluation and relative comparisons are metric-dependent; diverse sets of hyperparameters can produce a “best” result; and some GANs are more robust to hyperparameter changes than others. These observations both corroborate findings of previous GAN evaluation studies and make novel contributions regarding the relationship between size, complexity, and GAN performance.





§ INTRODUCTION






Generative adversarial network (GAN) optimization stability and convergence properties remain poorly understood despite the introduction of hundreds of GAN variants since their conception <cit.>. While GAN learning and performance behavior has been studied <cit.>, most existing work examining this relationship focuses on image datasets for which the underlying distribution parameterization is inaccessible <cit.>. This is problematic since claims of behavior that are made by modeling an unknown target distribution require a strong assumption for generalizability.

The goal of generative modeling is to approximate a distribution p_d by learning a parameterized distribution p_g, where both p_d and p_g are defined over samples. If we do not have full access to p_d, generalizability requires us to assume that the modeled dataset is a reasonable proxy for the family of distributions from which it was sampled. Without this assumption that is often only implicitly made, using images to understand GAN behavior limits conclusions to the data context being modeled.

We seek to address a gap in the literature by investigating GAN variant performance on datasets for which we have full access to the distribution parameterization. This allows us to study empirical performance on data where we can make claims of model behavior that generalize to the full distribution, as opposed to on image datasets for which this is not necessarily true. To this end, we examine the performance of 16 GAN variants on six explicitly parameterized multivariate distributions of four different dimensionalities and three different training set sizes. 


Across 20 grid search trials, we observe that: (1) GANs exhibit similar performance trends across dimensionalities, (2) learning depends on the underlying distribution and its complexity, (3) the number of training samples can have a large impact on performance, (4) evaluation and relative comparisons are metric-dependent, (5) diverse sets of hyperparameters can produce a “best” result, and (6) some GANs are more robust to hyperparameter changes than others. These findings corroborate those of previous GAN evaluation studies as well as contribute novel insights regarding the relationship between size, complexity, and GAN performance.[All code is publicly available at <https://github.com/shayneobrien/explicit-gan-eval>.]



§ RELATED WORK


One notable work in this area by <cit.> compares seven GAN variants in terms of modeling ability and optimization stability. The authors find that as computational budget increases, all tested models reach similar Frechét Inception Distance on the MNIST, Fashion-MNIST, CIFAR10, and CelebA datasets; and F1, precision, and recall on a synthetic dataset of convex polygons. They also discuss the difficulties of comparing GANs due to multiple valid ways to analyze performance. 

<cit.> measure Inception Score and classification accuracy and report that the five GAN variants they train do not succeed at capturing distributional properties of the training set on the CelebA and LSUN datasets. The authors observe that the GAN distributions exhibit significantly less diversity at test time compared to the evaluation dataset, suggesting p_g is far from p_d.

In another study, <cit.> evaluate GAN variant performance based on the original GAN criterion, least squares, maximum mean discrepancy, and improved Wasserstein distance. They show that for the three GAN variants they consider, test-time metrics do not favor networks that use the same training-time criterion on the MNIST, CIFAR10, LSUN, and Fashion-MNIST image datasets. The authors also examine performance as a function of sample size and show that some GANs exhibit faster performance increases than others as the number of training samples increases.



Lastly, <cit.> provide a thorough discussion of the strengths and weaknesses of 26 quantitative and qualitative measures used for evaluating GANs trained on image datasets. They  conclude that there is no single, best GAN evaluation measure. The authors suggest benchmarking models under identical architectures and computational budgets, and using more than a single metric to make comparisons.





§ EXPERIMENTAL SETUP

In GANs, we define a prior probability distribution on input noise variables p_z(𝐳) and represent a mapping to the target data space p_d(𝐱) as G(𝐳, θ_G), where G is a fully differentiable neural network called the generator and θ_G are its parameters. We train G by simultaneously learning a fully differentiable network D, called the discriminator or critic and defined by D(𝐱, θ_D), that helps G during training. Whereas G is trained to mimic p_d, the learning objective, output, and precise task of D vary depending upon the GAN variant.


 §.§ Models


As a case study, we examine the same seven GAN variants evaluated by <cit.> and nine additional GAN variants that have been popularly discussed since their study was published. The primary difference between considered variants is whether the discriminator output can be interpreted as a probability (MMGAN, NSGAN <cit.>, RaGAN <cit.>, DRAGAN <cit.>, FisherGAN <cit.>, InfoGAN <cit.>, ForwGAN, RevGAN, HellingerGAN, PearsonGAN, JSGAN <cit.>)
or is unbounded (WGAN <cit.>, WGANGP <cit.>, LSGAN <cit.>, BEGAN <cit.>).

We summarize these models in Table <ref>.



In our implementations, both D and G consist of two feedforward network layers each; the full architecture has four layers total. We apply a ReLU activation function to the output of each layer and sample the noise prior 𝐳 from 𝒩(0,h/4I), where h is the hidden dimension size. All models have the same number of trainable parameters except InfoGAN and BEGAN due to their use of latent variables as inputs to D and formation of D as an autoencoder, respectively. <cit.> argue that this difference is negligible for InfoGAN and we do not observe that it gives BEGAN any tangible advantage over other models. Trainable parameter counts can be found in Table <ref>.



 §.§ Data


We train each of these variants by randomly sampling 1,000, 10,000, and 100,000 data points from the following six explicitly parameterized multivariate distributions: Gaussian with mean μ and symmetric, full rank covariance Σ both from [0,1]; exponential with inverse mean shape λ from [0,1]; beta with shape parameters a and β both from [0,1]; gamma with shape k from [0,10] and scale θ from [0,2]; Gumbel with location μ and scale β both from [0,1]; and Laplace with location μ and scale β both from [0,1]. For each of these distributions and numbers of samples, we generate datasets of 16, 32, 64, and 128 dimensions. We note that by the Universal Approximation Theorem, our proposed network architecture should be able to model each of these distributions without exception.





 §.§ Hyperparameters

For all models and data distributions, we conduct 20 grid search trials with random network initializations for learning rates γ∈ [2e^-1, 2e^-2, 2e^-3], hidden dimension sizes h ∈ [32, 64, 128, 256, 512], and batch size b = 1024.[We also ran full experiments for b ∈ [128, 256, 512], but limit our analyses to b = 1024 as results across different batch sizes are not comparable due to greater noise in the data generation process at lower values of b.] For models with introduced hyperparameters, we use those given in the original the paper. We use the Adam optimizer with default settings <cit.> and train for 25 epochs.



 §.§ Measuring Divergence

We evaluate the difference between p_d and p_g using Kullback-Leibler divergence (KL), Jensen-Shannon divergence (JS), and Wasserstein Distance (WD). KL and JS focus on the alignment of the modes of the distributions and WD emphasizes how much p_g must be modified to reach p_d. Whereas JS and WD are symmetric, KL is not. For any of these measures, a value of 0 can be interpreted as indicating the two distributions being compared are identical <cit.>. We report results as the divergence between a generated batch and a test batch of size b=1024 at the end of every epoch.



 §.§ Estimating p_g

Although we have access to the true data distribution p_d, we must estimate the probability distribution of p_g. Since the data dimensionality is low, we construct a dimension-wise histogram for each data point.[Kernel density estimation was found to give similar outputs while being more computationally expensive.] In doing so, we assume that each dimension is independent from the others. This assumption is valid in the case of all experiments involving non-Gaussian data, which follows from the multivariate model being a product of the marginal distributions. To select the optimal bin width B_w in the histogram, we follow the Freedman-Diaconis rule: B_w = 2 · IQR(x)/√(M), where IQR is the inter-quartile range of the M samples x = {x_1, ..., x_M} from the distribution being approximated.
This initialization minimizes the difference between the areas under the empirical and theoretical probability distributions <cit.>. 













































































































































§ RESULTS








In our analyses, we take the same approach as <cit.> and <cit.>: we let the “best” hyperparameter setting be the one that achieved the lowest minimum performance on average across all trials for each distribution, metric, and number of training samples, respectively. We include results, visualizations, and evidence to support all conclusions in the appendices. For the best hyperparameter settings in our learning environment, we find that:




  * GANs exhibit similar learning trends across dimensionalities: For many of the models, performance under the best hyperparameter setting consistently follows a trend across dimensionalities for all three tested metrics. At the same time, performance generally worsens with increased dimensionality. See Figures <ref>, <ref>, <ref>.


  * Learning depends on the underlying distribution and its complexity: Models which do well on some distributions perform poorly on the same distribution with higher dimensionality, or on other distributions of the same dimensionality. It is not immediately apparent that these differences are due to model design. <cit.> make a similar finding in the case of image datasets with varying complexities. See Figures <ref>, <ref>, <ref>.


  * Number of training samples can have a large impact on performance: Some GAN variants are able to achieve the same performance learning from 1,000 samples as 10,000 or 100,000 samples, while others show large performance jumps with increased amounts of data. At the same time, almost all GAN variants begin to worsen in performance within five epochs for 1,000 training samples. The number of training samples seems to be critical to some models' performances, which was also noted by <cit.>. See Figures <ref>, <ref>, and <ref>.



  * Evaluation and comparison are metric-dependent: Relative ranking of GAN variants according to performance varies depending on the evaluation metric used to rank them. No single GAN performed best across all metrics for any dataset or dimensionality. We concur with previous studies that GANs generally perform the same, although there are variants that perform worse than others on some distributions <cit.>. We warn against ranking models as relative differences can be marginal. See Tables <ref>, <ref>, <ref>, and <ref>.


  * Diverse sets of hyperparameters can produce a “best” result: Many, diverse hyperparameter settings yielded superior performances to the best average minimum performance, but these models did not achieve those minima with tight confidence bounds. Furthermore, we see that even on the best performing hyperparameter settings, our tested models preferred widely different hidden dimensionalities and learning rates; some variants with less parameters outperformed others that had more. We agree with previous work that it is important to present results that are able to be consistently reproduced <cit.>. See Table <ref>.


  * Some GANs are more robust to hyperparameter changes than others: With respect to the distribution, dimensionality, and training set size being approximated, some models yielded average minimum performances for more hyperparameters than others that fell within the confidence interval of the best average minimum performance under consideration. This is an indication that some GANs can perform well under a greater range of hyperparameter settings than others. See Table <ref>.




§ FUTURE WORK


In future work, we plan to analyze cases where GAN variants underperform relative to others and relate the characteristics of the distribution being modeled to the assumptions made in designing the variant, e.g. by empirically considering whether a normally distributed prior hurts performance on non-normal distributions. We would also like to use longer training times and more complex models to evaluate additional synthetic datasets such as multivariate mixture models, colored circles, and autoencoded image datasets.



§ ACKNOWLEDGEMENTS

We would like to thank Deb Roy, Iyad Rahwan, Manolis Zampetakis, and the Media Lab Consortium for their support of this research. Aleksander Mądry and Costis Daskalakis provided us with inspiration to pursue this project. Shayne O'Brien is partially funded through a National Science Foundation Graduate Research Fellowship.

26


urlstyle

[Arjovsky et al.(2017)Arjovsky, Chintala, and
  Bottou]arjovsky2017wasserstein
Martín Arjovsky, Soumith Chintala, and Léon Bottou.
Wasserstein generative adversarial networks.
In International Conference on Machine Learning, pages
  214–223, 2017.

[Arora et al.(2017)Arora, Ge, Liang, Ma, and
  Zhang]arora2017generalization
Sanjeev Arora, Rong Ge, Yingyu Liang, Tengyu Ma, and Yi Zhang.
Generalization and equilibrium in generative adversarial nets
  (GANs).
In Proceedings of the 34th International Conference on Machine
  Learning, pages 224–232, 2017.

[Bethelot et al.(2017)Bethelot, Schumm, and Metz]berthelot2017began
David Bethelot, Thomas Schumm, and Luke Metz.
BEGAN: Boundary equilibrium generative adversarial networks.
arXiv preprint arXiv:1703.10717, 2017.

[Borji(2018)]borji2018pros
Ali Borji.
Pros and cons of GAN evaluation measures.
CoRR, abs/1802.03446, 2018.
URL <http://arxiv.org/abs/1802.03446>.

[Chen et al.(2016)Chen, Duan, Houthooft, Schulman, Sutskever, and
  Abbeel]infogan
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter
  Abbeel.
InfoGAN: Interpretable representation learning by information
  maximizing generative adversarial nets.
In Advances in Neural Information Processing Systems, pages
  2172–2180, 2016.

[Endres and Schindelin(2003)]jensenshannon
Dominik Endres and Johannes Schindelin.
A new metric for probability distributions.
In IEEE Transactions on Information Theory, pages 1858–1860,
  2003.

[Freedman and Diaconis(1981)]diaconis
David Freedman and Persi Diaconis.
On the histogram as a density estimator: L2 theory.
Zeitschrift für Wahrscheinlichkeitstheorie und Verwandte
  Gebiete, pages 453–476, 1981.

[Goodfellow et al.(2014)Goodfellow, Pouget-Abadie, Mirza, Xu,
  Warde-Farley, Ozair, Courville, and Bengio]goodfellow2014generative
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley,
  Sherjil Ozair, Aaron Courville, and Yoshua Bengio.
Generative adversarial nets.
In Advances in neural information processing systems, pages
  2672–2680, 2014.

[Gulrajani et al.(2017)Gulrajani, Ahmed, Arjovsky, Dumoulin, and
  Courville]gulrakani2017wassersteingp
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C
  Courville.
Improved training of wasserstein GANs.
In Advances in Neural Information Processing Systems, pages
  5767–5777. 2017.

[Heusel et al.(2017)Heusel, Ramsauer, Unterthiner, Nessler, Klambauer,
  and Hochreiter]heusel2017gans
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler,
  Günter Klambauer, and Sepp Hochreiter.
GANs trained by a two time-scale update rule converge to a Nash
  equilibrium.
arXiv preprint arXiv:1706.08500, 2017.

[Hindupur(2017)]theganzoo
Avinash Hindupur.
The GAN zoo.
GitHub Repository, 2017.
URL <https://github.com/hindupuravinash/the-gan-zoo>.

[Huang et al.(2018)Huang, Yuan, Xu, Guo, Sun, Wu, and
  Weinberger]huang2018an
Gao Huang, Yang Yuan, Qiantong Xu, Chuan Guo, Yu Sun, Felix Wu, and Kilian
  Weinberger.
An empirical study on evaluation metrics of generative adversarial
  networks.
2018.
URL <https://openreview.net/forum?id=Sy1f0e-R->.

[Im et al.(2018)Im, Ma, Taylor, and Branson]divergencetraining
Daniel Jiwoong Im, Alllan He Ma, Graham W. Taylor, and Kristin Branson.
Quantitatively evaluating GANs with divergences proposed for
  training.
In International Conference on Learning Representations, 2018.
URL <https://openreview.net/forum?id=SJQHjzZ0->.

[Jolicoeur-Martineau(2018)]ragan
Alexia Jolicoeur-Martineau.
The relativistic discriminator: A key element missing from standard
  GAN.
arXiv preprint arXiv:1807.00734, 2018.

[Kingma and Ba(2015)]adam
Diederik Kingma and Jimmy Ba.
Adam: A method for stochastic optimization.
In International Conference on Learning Representations, 2015.

[Kodali et al.(2017)Kodali, Abernethy, Hays, and
  Zsolt]kodali2017dragan
Naveen Kodali, Jacob Abernethy, James Hays, and Kira Zsolt.
On convergence and stability of GANs.
arXiv preprint arXiv:1705.07215, 2017.

[Kullback and Leibler(1951)]kullbackleibler
Soloman Kullback and Richard Leibler.
On information and sufficiency.
pages 79–86, 1951.

[Lee et al.(2017)Lee, Ge, Ma, Risteski, and Arora]lee2017ability
Holden Lee, Rong Ge, Tengyu Ma, Andrej Risteski, and Sanjeev Arora.
On the ability of neural nets to express distributions.
In Journal of Machine Learning Research, pages 1–26, 2017.

[Lucic et al.(2017)Lucic, Kurach, Michalski, Gelly, and
  Bousquet]lucic2017gans
Mario Lucic, Karol Kurach, Marcin Michalski, Sylvain Gelly, and Olivier
  Bousquet.
Are GANs created equal? A large-scale study.
arXiv preprint arXiv:1711.10337, 2017.

[Mao et al.(2017)Mao, Li, Xie, Lau, Wang, and Smolley]lsgan
Xudong Mao, Qing Li, Haoran Xie, Raymond YK Lau, Zhen Wang, and Stephen Paul
  Smolley.
Least squares generative adversarial networks.
In 2017 IEEE International Conference on Computer Vision
  (ICCV), pages 2813–2821, 2017.

[Mroueh and Sercu(2017)]fishergan
Youssef Mroueh and Tom Sercu.
Fisher GAN.
In Advances in Neural Information Processing Systems, pages
  3310–3320, 2017.

[Nowozin et al.(2016)Nowozin, Cseke, and Tomioka]fgan
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka.
f-GAN: Training generative neural samplers using variational
  divergence minimization.
In Advances in Neural Information Processing Systems, pages
  271–279. 2016.

[Salimans et al.(2016)Salimans, Goodfellow, Zaremba, Cheung, Radford,
  and Chen]salimans2016improved
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and
  Xi Chen.
Improved techniques for training GANs.
In Advances in Neural Information Processing Systems, pages
  2234–2242, 2016.

[Santurkar et al.(2017)Santurkar, Schmidt, and Madry]santurkar
Shibani Santurkar, Ludwig Schmidt, and Aleksander Madry.
A classification-based perspective on GAN distributions.
arXiv preprint arXiv: 1711.00970, 2017.

[Theis et al.(2015)Theis, van den Oord, and Bethge]theis
Lucas Theis, Aäron van den Oord, and Matthias Bethge.
A note on the evaluation of generative models.
arXiv preprint arXiv:1511.01844, 2015.

[Villani(2008)]villani2008optimal
Cédric Villani.
Optimal Transport: Old and New, volume 338.
Springer Science & Business Media, 2008.


plainnat





§ GANS EXHIBIT SIMILAR LEARNING TRENDS ACROSS DIMENSIONALITIES, AND LEARNING DEPENDS ON THE UNDERLYING DISTRIBUTION AND ITS COMPLEXITY













§ NUMBER OF TRAINING SAMPLES CAN HAVE A LARGE IMPACT ON PERFORMANCE














§ EVALUATION AND COMPARISONS ARE METRIC-DEPENDENT
 















































































































§ DIVERSE SETS OF HYPERPARAMETERS CAN PRODUCE A BEST RESULT








§ SOME GANS ARE MORE ROBUST TO HYPERPARAMETER CHANGES THAN OTHERS






§ MODEL SPECIFICATIONS





