Under review as a conference paper at ICLR 2018

A C LASSIFICATION –BASED P ERSPECTIVE ON
GAN D ISTRIBUTIONS

arXiv:1711.00970v3 [cs.LG] 19 Nov 2017

Shibani Santurkar, Ludwig Schmidt, Aleksander Madry
˛
Massachusetts Institute of Technology
{shibani,ludwigs,madry}@mit.edu

A BSTRACT
A fundamental, and still largely unanswered, question in the context of Generative Adversarial Networks (GANs) is whether GANs are actually able to capture
the key characteristics of the datasets they are trained on. The current approaches
to examining this issue require significant human supervision, such as visual inspection of sampled images, and often offer only fairly limited scalability. In this
paper, we propose new techniques that employ a classification–based perspective
to evaluate synthetic GAN distributions and their capability to accurately reflect
the essential properties of the training data. These techniques require only minimal human supervision and can easily be scaled and adapted to evaluate a variety
of state-of-the-art GANs on large, popular datasets. Our analysis indicates that
GANs have significant problems in reproducing the more distributional properties
of the training dataset. In particular, when seen through the lens of classification,
the diversity of GAN data is orders of magnitude less than that of the original data.

1

I NTRODUCTION

Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) have garnered a significant
amount of attention due to their ability to learn generative models of multiple natural image datasets
(Radford et al., 2015; Denton et al., 2015; Zhang et al., 2016). Since their conception, a fundamental
question regarding GANs is to what extent they truly learn the underlying data distribution. This
is a key issue for multiple reasons. From a scientific perspective, understanding the capabilities of
common GANs can shed light on what precisely the adversarial training setup allows the GAN to
learn. From an engineering standpoint, it is important to grasp the power and limitations of the
GAN framework when applying it in concrete applications. Due to the broad potential applicability
of GANs, researchers have investigated this question in a variety of ways.
When we evaluate the quality of a GAN, an obvious first check is to establish that the generated
samples lie in the support of the true distribution. In the case of images, this corresponds to checking
if the generated samples look realistic. Indeed, visual inspection of generated images is currently
the most common way of assessing the quality of a given GAN. Individual humans can performs
this task quickly and reliably, and various GANs have achieved impressive results for generating
realistic-looking images of faces and indoor scenes (Salimans et al., 2016; Denton et al., 2015).
Once we have established that GANs produce realistic-looking images, the next concern is that the
GAN might simply be memorizing the training dataset. While this hypothesis cannot be ruled out
entirely, there is evidence that GANs perform at least some non-trivial modeling of the unknown
distribution. Previous studies show that interpolations in the latent space of the generator produce
novel and meaningful image variations (Radford et al., 2015), and that there is a clear disparity
between generated samples and their nearest neighbors in the true dataset (Arora & Zhang, 2017).
Taken together, these results provide evidence that GANs could constitute successful distribution
learning algorithms, which motivates studying their distributions in more detail. The direct approach
is to compare the probability density assigned by the generator with estimates of the true distribution
(Wu et al., 2016). However, in the context of GANs for high-dimensional image distributions, this
is complicated by two factors. First, GANs do not naturally provide probability estimates for their
samples. Second, estimating the probability density of the true distribution is a challenging problem
1

Under review as a conference paper at ICLR 2018

itself (the adversarial training framework specifically avoids this issue). Hence prior work has only
investigated the probability density of GANs on simple datasets such as MNIST (Wu et al., 2016).
Since reliably computing probability densities in high dimensions is challenging, we can instead
study the behavior of GANs in low-dimensional problems such as two-dimensional Gaussian mixtures. Here, a common failure of GANs is mode collapse, wherein the generator assigns a disproportionately large mass to a subset of modes from the true distribution (Goodfellow, 2016). This raises
concerns about a lack of diversity in the synthetic GAN distributions, and recent work shows that
the learned distributions of two common GANs indeed have (moderately) low support size for the
CelebA dataset (Arora & Zhang, 2017). However, the approach of Arora & Zhang (2017) heavily
relies on a human annotator in order to identify duplicates. Hence it does not easily scale to comparing many variants of GANs or asking more fine-grained questions than collision statistics. Overall,
our understanding of synthetic GAN distributions remains blurry, largely due to the lack of versatile
tools for a quantitative evaluation of GANs in realistic settings. The focus of this work is precisly to
address this question:
Can we develop principled and quantitative approaches to study synthetic GAN distributions?
To this end, we propose two new evaluation techniques for GAN distributions. Our methods are
inspired by the idea of comparing moments of distributions, which is at the heart of many methods in classical statistics. Although simple moments of high-dimensional distributions are often not
semantically meaningful, we can extend this idea to distributions of realistic images by leveraging
image statistics identified using convolutional neural networks. In particular, we train image classifiers in order to construct test functions corresponding to semantically meaningful properties of the
distributions. An important feature of our approach is that it requires only light human supervision
and can easily be scaled to evaluating many GANs and large synthetic datasets.
Using our new evaluation techniques, we study five state-of-the-art GANs on the CelebA and LSUN
datasets, arguably the two most common testbeds for advanced GANs. We find that most of the
GANs significantly distort the relative frequency of even basic image attributes, such as the hair
style of a person or the type of room in an indoor scene. This clearly indicates a mismatch between
the true and synthetic distributions. Moreover, we conduct experiments to explore the diversity of
GAN distributions. We use synthetic GAN data to train image classifiers and find that these have
significantly lower accuracy than classifiers trained on the true data set. This points towards a lack
of diversity in the GAN data, and again towards a discrepancy between the true and synthetic distributions. In fact, our additional examinations show that the diversity in GANs is only comparable to
a subset of the true data that is 100× smaller.

2

U NDERSTANDING GAN S THROUGH THE L ENS OF C LASSIFICATION

When comparing two distributions, a common first test is to compute low-order moments such as
the mean and the variance. If the distributions are simple enough, these quantities provide a good
understanding for how similar they are. Moreover, low-order moments have a precise definition and
are usually quick to compute. On the other hand, low-order moments can also be misleading for
more complicated, high-dimensional distributions. As a concrete example, consider a generative
model of digits (such as MNIST). If a generator produces digits that are shifted by a significant
amount yet otherwise perfect, we will probably still consider this as a good approximation of the
true distribution. However, the expectation (mean moment) of the generator distribution can be very
different from the expectation of the true data distribution. This raises the question of what other
properties of high-dimensional image distributions are easy to test yet semantically meaningful.
In the next two subsections, we describe two concrete approaches to evaluate synthetic GAN data
that are easy to compute yet capture relevant information about the distribution. The common theme
is that we employ convolutional neural networks in order to capture properties of the distributions
that are hard to describe in a mathematically precise way, but usually well-defined for a human (e.g.,
what fraction of the images shows a smiling person?). Automating the process of annotating images
with such high-level information will allow us to study various aspects of synthetic GAN data.
2.1

Q UANTIFYING M ODE C OLLAPSE

Mode collapse refers to the tendency of the generator to concentrate a large probability mass on a
few modes of the true distribution. While there is ample evidence for the presence of mode-collapse
2

Under review as a conference paper at ICLR 2018

in GANs (Goodfellow, 2016; Arora & Zhang, 2017; Metz et al., 2016), elegant visualizations of this
phenomena are somewhat restricted to toy problems on low-dimensional distributions (Goodfellow,
2016; Metz et al., 2016). For image datasets, it is common to rely on human annotators and therein
derived heuristics (see Section 2.3). While these methods have their merits, they are restrictive both
in the scale and granularity of testing. Here we propose a classification-based tool to assess how
good GANs are at assigning the right mass across broad concepts/modes. To do so, we use a trained
classifier as an expert “annotator” that labels important features in synthetic data, and then analyze
the resulting distribution. Specifically, our goal is to investigate if a GAN trained on a well-balanced
dataset (i.e., contains equal number of samples from each class) can learn to reproduce this balanced
N
structure. Let D = (X, Y ) = {(xi , yi )}i=1 represent a dataset of size N with C classes, where
(xi , yi ) denote an image-label pair drawn from true data. If the dataset D is balanced, it contains
N/C images per class. The procedure for computing class distribution in synthetic data is:
1. Train an annotator (a multi-class classifier) using the dataset D.
2. Train an unconditional GAN on the images X from dataset D, without using class labels.
3. Create a synthetic dataset by sampling N images from a GAN and labeling them using the
annotator from Step 1.
The annotated data generated via the above procedure can provide insight into the GAN’s class
distribution at the scale of the entire dataset. Moreover, we can vary the granularity of mode analysis
by choosing richer classification tasks, i.e., more challenging classes or a larger number of them. In
Section 3.3, we use this technique to visualize mode collapse in several state-of-the-art GANs on the
CelebA and LSUN datasets. All the studied GANs show significant mode collapse and the effect
becomes more pronounced when the granularity of the annotator is increased (larger number of
classes). We also investigate the temporal aspect of the GAN setup and find that the dominant mode
varies widely over the course of training. Our approach also enables us to benchmark and compare
GANs on different datasets based on the extent of mode collapse in the learned distributions.
2.2

M EASURING DIVERSITY

Our above method for inspecting distribution of modes in synthetic data provides a coarse look at the
statistics of the underlying distribution. While the resulting quantities are semantically meaningful,
they capture only simple notions of diversity. To get a more holistic view on the sample diversity
in the synthetic distribution, we now describe a second classification-based approach for evaluating
GAN distributions. The main question that motivates it is: Can GANs recover the key aspects of
real data to enable training a good classifier? We believe that this is an interesting measure of sample diversity for two reasons. First, classification of high-dimensional image data is a challenging
problem, so a good training dataset will require a sufficiently diverse sample from the distribution.
Second, augmenting data for classification problems is one of the proposed use cases of GANs (e.g.,
see the recent work of Shrivastava et al. (2017)).
If GANs are truly able to capture the quality and diversity of the underlying data distribution, we
expect almost no gap between classifiers trained on true data and synthetic data from a GAN. A
generic method to produce data from GANs for classification is to train separate GANs for each
class in the dataset D.1 Samples from these class-wise GANs can then be pooled together to get a
labeled synthetic dataset. Note that the labels are trivially determined based on the class modeled
by the particular GAN from which a sample is drawn. We perform the following steps to assess the
classification performance of synthetic data vs. true data:
1. Train a classifier on the true data D (from Section 2.1) as a benchmark for comparison.
2. Train C separate unconditional GANs, one per class in dataset D.
3. Generate a balanced synthetic labeled dataset of size N by consolidating an equal number of
samples drawn from each of these C GANs. The labels obtained by aggregating samples from
per-class GANs are designated as “default" labels for the synthetic dataset. Note that by design,
both true and synthetic datasets have N samples, with N/C examples per class.
4. Use synthetic labeled data from Step 3 to train classifier with the same architecture as Step 1.
Comparing the classifiers from Steps 1 and 4 can then shed light on the disparity between the two
distributions. Radford et al. (2015) conducted an experiment similar to Step 2 on the MNIST dataset
1
We tried the alternate approach of using class-conditional GANs to get labeled datasets. This method did
not yield good samples since most common GANs have not been designed with a conditional structure in place.

3

Under review as a conference paper at ICLR 2018

using a conditional GAN. They found that samples from their DCGAN performed comparably to
true data on nearest neighbor classification. We obtained similar good results on MNIST, which
could be due to the efficacy of GANs in learning the MNIST distribution or due to the ease of
getting good accuracy on MNIST even with a small training set (Rolnick et al., 2017). To clarify
this question, we restrict our analysis to more complex datasets, specifically CelebA and LSUN.
We evaluate the two following properties in our classification task:
(i) How well can the GANs recover nuances of the decision boundary, which is reflected by how
easily the classifier can fit the training data?
(ii) How does the diversity of synthetic data compare to that of true data when measured by classification accuracy on a hold-out set of true data?
We observe that all the studied GANs have very low diversity in this metric. In particular, the
accuracy achieved by a classifier trained on GAN data is comparable only to the accuracy of a
classifier trained on a 100× (or more) subsampled version of the true dataset. Even if we draw more
samples from the GANs to produce a training set several times larger than the true dataset, there is
no improvement in performance. Looking at the classification accuracy gives us a way to compare
different models on a potential downstream application of GANs. Interestingly, we find that visual
quality of samples does not necessarily correlate with good classification performance.
2.3

R ELATED W ORK

In GAN literature, it is common to investigate performance using metrics that involve human supervision. Arora & Zhang (2017) proposed a measure based on manually counting duplicates in GAN
samples as a heuristic for the support or diversity of the learned distribution. In Wu et al. (2016),
manual classification of a small sample (100 images) of GAN generated MNIST images is used as
a test for the GAN is missing certain modes. Such annotator-based metrics have clear advantages
in identifying relevant failure-modes of synthetic samples, which explains why visual inspection
(eyeballing) is still the most popular approach to assess GAN samples.
There have also been various attempts to build good metrics for GANs that are not based on manual
heuristics. Parzen window estimation can be used to approximate the log-likelihood of the distribution, though it is known to work poorly for high-dimensional data (Theis et al., 2016). Wu et al.
(2016) develop a method to get a better estimate for log-likelihood using annealed importance sampling. Salimans et al. (2016) propose a metric known as Inception Score, where the entropy in the
labels predicted by a pre-trained Inception network is used to assess the diversity in GAN samples.

3

E XPERIMENTS

In the following sub-sections we describe the setup and results for our classification-based GAN
benchmarks. Additional details can be found in Section 5 in the Appendix.
3.1

DATASETS

GANs have shown promise in generating realistic samples, resulting in efforts to apply them to a
broad spectrum of datasets. However, the Large-scale CelebFaces Attributes (CelebA) (Liu et al.,
2015) and Large-Scale Scene Understanding (LSUN) (Yu et al., 2015) datasets remain the most
popular and canonical ones in developing and evaluating GAN variants. Conveniently, these datasets
also have rich annotations, making them particularly suited for our classification–based evaluations.
Details on the setup for classification tasks for these datasets are given in the Appendix.
3.2

M ODELS

Using our framework, we perform a comparative study of several popular variants of GANs:
1. Deep Convolutional GAN (DCGAN): Convolutional GAN trained using a Jensen–Shannon
divergence–based objective (Goodfellow et al., 2014; Radford et al., 2015).
2. Wasserstein GAN (WGAN): GAN that uses a Wasserstein distance–based objective (Arjovsky
et al., 2017).
3. Adversarially Learned Inference (ALI): GAN that uses joint adversarial training of generative
and inference networks (Dumoulin et al., 2017).
4

Under review as a conference paper at ICLR 2018

Female
+
Mouth Closed

True Data

DCGAN

WGAN

ALI

BEGAN

Improved GAN

Female
+
Mouth Open
Male
+
Mouth Closed
Male
+
Mouth Open

4-class dataset from CelebA for attributes Male, Mouth open.
Figure 1: Illustration of a dataset used in the proposed classification benchmarks. Shown alongside
are images sampled from various unconditional GANs trained on this dataset. Labels for the GAN
samples are obtained using a pre-trained classifier as an annotator.
4. Boundary Equilibrium GAN (BEGAN): Auto-encoder style GAN trained using Wasserstein distance objective (Berthelot et al., 2017).
5. Improved GAN (ImGAN): GAN that uses semi-supervised learning (labels are part of GAN
training), with various other architectural and procedural improvements (Salimans et al., 2016).
All the aforementioned GANs are unconditional, however, ImGAN has access to class labels as a
part of the semi-supervised training process. We use standard implementations for each of these
models, details of which are provided in the Appendix (Section 5). We also used the prescribed
hyper-parameter settings for each GAN, including number of iterations we train them for. Our
analysis is based on 64 × 64 samples, which is a size at which GAN generated samples tend to be of
high quality. We also use visual inspection to ascertain that the perceptual quality of GAN samples
in our experiments is comparable to those reported in previous studies. We demonstrate sample
images in Figures 1, 4 and 5. BEGAN did not converge in our experiments on the LSUN dataset
and hence is excluded from the corresponding analysis.
In our study, we use two types of classification models:
1. ResNet: 32-Layer Residual network He et al. (2016). This model is chosen as it is a standard
classifier in vision and yields high accuracy on various datasets, making it a reliable baseline.
2. Linear Model: This is a network with one-fully connected layer between the input and output (no
hidden layers) with a softmax non-linearity. If the dimensions of input x and output ŷ, are D and
C (number of classes) respectively, then linear models implement the function ŷ = σ(W T x+b),
where W is a D × C matrix, b is a C × 1 vector and σ(·) is the softmax function. Due to it’s
simplicity, this model will serve as a useful baseline in some of our experiments.
We always train the classifiers to convergence, with decaying learning rate and no data augmentation.
3.3

E XAMINATION OF M ODE C OLLAPSE

Experimental results for quantifying mode collapse through classification tasks, described in Section 2.1, are presented below. Table 1 gives details on datasets (subsets of CelebA and LSUN)
used in our analysis, such as size (N ), number of classes (C), and accuracy of the annotator, i.e.,
a classifier pre-trained on true data, which is then used to label the synthetic, GAN-generated data.
Figure 2 presents class distribution in the synthetic data, as determined by using these annotators.
The left panel compares the relative distribution of modes in true data (uniform) with that in various
GAN-generated datasets. Each of these datasets is created by drawing N samples from the GAN
after it was trained on the corresponding true dataset. The right panel illustrates the evolution of
class distributions in various GANs over the course of training2 .
Results: These visualization lead to the following findings:
2
Temporal evaluation of ALI class distribution is absent in the analysis due to absence of periodic checkpointing provisions in the code.

5

Under review as a conference paper at ICLR 2018

Dataset
CelebA: Makeup, Smiling
CelebA: Male, Mouth Open
CelebA: Bangs, Smiling
LSUN: Bedroom, Kitchen, Classroom
LSUN: Bedroom, Conference Room, Dining Room,
Kitchen, Living Room

N
102,436
115,660
45,196
150,000
250,000

C
4
4
4
3
5

Annotator’s Accuracy (%)
90.9, 92.4
97.9, 93.5
93.9, 92.4
98.7
93.7

Table 1: Details of CelebA and LSUN subsets used for the studies on mode collapse in Section 3.3.
Here, we use a classifier trained on true data as an annotator that let’s us infer label distribution for
the synthetic, GAN-generated data. N is the training set size and C is the number of classes for both
the true and synthetic datasets. Annotator’s accuracy refers to the accuracy of the classifier on a test
set of true data. For CelebA, we use a combination of attribute-wise binary classifiers as annotators
due their higher accuracy compared to a single classifier trained jointly on all the four classes.
CelebA: Makeup, Smiling

CelebA: Male, Mouth Open

CelebA: Bangs, Smiling

LSUN: Bedroom, Kitchen, Classroom

LSUN: Bedroom, Conference Room, Dining Room, Kitchen, Living Room

Figure 2: Visualizations of mode collapse in the synthetic, GAN-generated data produced after training on our chosen subsets of CelebA and LSUN datasets. Left panel shows the relative distribution
of classes in samples drawn from synthetic datasets extracted at the end of the training process, and
compares is to the true data distribution (leftmost plots). On the right, shown is the evolution of analogous class distribution for different GANs over the course of training. BEGAN did not converge
on the LSUN tasks and hence is excluded from the corresponding analysis.

6

Under review as a conference paper at ICLR 2018

• All GANs seem to suffer from significant mode-collapse. This becomes more apparent when the
annotator granularity is increased, by considering a larger set of classes. For instance, one should
compare the relatively balanced class distributions in the 3-class LSUN task to the near-absence
of some modes in the 5-class task.
• Mode collapse is prevalent in GANs throughout the training process, and does not seem to recede
over time. Instead the dominant mode(s) often fluctuate wildly over the course of the training.
• For each task, often there is a common set of modes onto which distinct GANs exhibit collapse.
In addition to viewing our method as an approach to analyze the mode collapse, we can also use it
as a benchmark for GAN comparison. From this perspective, we can observe that ALI consistently
shows lowest mode collapse amongst all the studied GANs. The behavior of the other GANs appears
to vary based on the dataset – on CelebA, DCGAN learns a somewhat balanced distributions, while
WGAN, BEGAN and ImGAN show prominent mode collapse. This is in contrast to the results
obtained LSUN, where, for example, WGAN exhibit relatively small mode collapse, while DCGAN
and ImGAN show significant mode collapse. This points to a general challenge in real world applications of GANs: they often perform well on the datasets they were designed for (e.g. WGAN on
LSUN), but extension to new datasets is not straightforward. Temporal analysis of mode-collapse
shows that there is wide variation in the dominant mode for WGAN and Improved GAN, whereas
for BEGAN, the same mode(s) often dominates the entire training process.
3.4

D IVERSITY E XPERIMENTS

Using the procedure outlined in Section 2.2, we perform a quantitative assessment of sample diversity in GANs on the CelebA and LSUN datasets. We restrict our experiments to binary classification
as we find they have sufficient complexity to highlight the disparity between true and synthetic data.
Results for classification-based evaluation of GANs are presented in Table 2 and Figure 3.
As a preliminary check, we inspect the quality of our labeled GAN datasets. For this, we use
high-accuracy annotators from Section 2.1 to predict labels for GAN generated data and measure
consistency between the predicted and default labels (label correctness). We also inspect confidence
scores, defined as the softmax probabilities for predicted class, of the annotator. The motivation
behind these metrics is that if the classifier can correctly and with high-confidence predict labels
for labeled GAN samples, then it is likely that they are convincing examples of that class, and
hence of good “quality". Empirical results for label agreement and annotator confidence of GAN
generated datasets are shown in Table 2, and Figure 6 in the Appendix. In Table 2, we also report
an equivalent Inception Score (Salimans et al., 2016), similar to that described in Section 2.3. Using
the Inception network to get the label distribution may not be meaningful for face or scene images.
Instead, we compute the Inception Score using the label distribution predicted from the annotator
networks. Score is computed as exp(Ex [KL(p(y|x))||p(y)]), where y refers to label predictions
from the annotators 3 .
Next, we train classifiers using the true and labeled GAN-generated datasets and study their performance in terms of accuracy on a hold-out set of true data. ResNets (and other deep variants) yield
good classification performance on true data, but suffer from severe overfitting on the synthetic data,
leading to poor test accuracy (see Table 2). This already indicates a possible problem with GANs
and the diversity of the data they generate. But to highlight this problem better and avoid the issues
that stem from overfitting, we also look for a classifier which does not always overfit on the synthetic
data. We, however, observed that even training simple networks, such as one fully connected layer
with few hidden units, led to overfitting on synthetic data. Hence, we resorted to a very basic linear
model described in Section 3.2. Table 2 shows results from binary classification experiments using
linear models, with the training and test accuracies of the classifier on various datasets.
Finally, to get a better understanding of the underlying "diversity" of synthetic datasets, we train
linear models using down-sampled versions of true data (no augmentation), and compare this to
the performance of synthetic data, as shown in Table 2. Down-sampling the data by a factor of
M , denoted as ↓M implies selecting a random N/M subset of the data D. Visualizations of how
GAN classification performance compares with (down-sampled) true data are in Figure 3. Here,
the bold curve shows test accuracy of a classifier trained on true data as a function of true dataset
3

Code from https://github.com/openai/improved-gan

7

Under review as a conference paper at ICLR 2018

Task

CelebA
Male (Y/N)
# Images: 136522

CelebA
Smiling (Y/N)
# Images: 156160

CelebA
Black Hair (Y/N)
# Images: 77812

LSUN
Bedroom/Kitchen
# Images: 200000

Data Source

True
True ↓64
True ↓256
True ↓512
True ↓1024
DCGAN
WGAN
ALI
BEGAN
Improved GAN
True
True ↓64
True ↓256
True ↓512
True ↓1024
DCGAN
WGAN
ALI
BEGAN
Improved GAN
True
True ↓64
True ↓256
True ↓512
True ↓1024
DCGAN
WGAN
ALI
BEGAN
Improved GAN
True
True↓512
True ↓1024
True ↓2048
True ↓4096
DCGAN
WGAN
ALI
Improved GAN

Classification Performance
Label
Accuracy (%)
Inception Score
Correctness
Linear model
(%)
(µ ± σ)
↑1
↑10
Train Test Train Test
88.1 88.8
89.6 88.7
97.9
1.98 ± 0.0033 91.6 86.9
96.3 83.8
100.0 83.1
98.2
1.97 ± 0.0013 100.0 79.2 100.0 79.6
98.3
1.97 ± 0.0013 96.7 84.0 96.7 83.9
99.2
1.99 ± 0.0008 95.8 86.7 95.8 86.7
99.3
1.99 ± 0.0006 97.9 78.0 98.0 78.2
99.8
1.99 ± 0.0004 100.0 75.6 100.0 71.0
85.7 85.6
87.6 85.0
92.4
1.69 ± 0.0074 91.5 82.4
93.7 80.0
95.0 76.2
96.1
1.67 ± 0.0028 100.0 77.1 100.0 77.1
98.2
1.68 ± 0.0031 96.8 83.4 96.8 83.5
93.3
1.71 ± 0.0027 94.5 80.1 95.0 82.4
93.5
1.74 ± 0.0028 98.5 69.5 98.5 69.6
98.4
1.88 ± 0.0021 100.0 70.2 100.0 70.1
76.4 76.5
79.7 75.4
84.5
1.68 ± 0.0112 86.3 72.6
89 68.7
100.0 65.4
86.7
1.68 ± 0.0040 100.0 70.9 100.0 70.5
76.0
1.60 ± 0.0055 94.4 73.7 94.3 73.4
79.4
1.63 ± 0.0028 94.9 71.0 94.9 70.2
87.6
1.74 ± 0.0028 94.1 67.6 94.1 67.7
86.7
1.64 ± 0.0045 100.0 70.3 100.0 69.1
64.7 64.1
64.7 64.0
98.2
1.94 ± 0.0217 65.2 64.0
98.7 56.2
100.0 55.1
92.7
1.85 ± 0.0036 90.8 56.5 91.2 56.3
87.8
1.70 ± 0.0023 86.2 58.2 96.3 54.1
94.1
1.86 ± 0.0021 82.5 60.0 82.0 59.7
84.2
1.68 ± 0.0030 91.6 55.9 90.8 56.5

ResNet
↑1
Test
97.9
92.9
89.8
82.6
81.4
56.4
50.0
58.9
55.4
71.7
92.4
87.8
82.1
77.8
71.2
63.3
65.3
55.8
64.1
61.6
84.5
80.0
75.8
73.9
72.7
53.4
58.5
55.7
67.2
70.2
99.1
76.4
66.9
56.5
55.1
51.2
55.7
56.2
51.2

Table 2: Results from a comparative study on classification performance of true data vs. GAN generated synthetic data for the CelebA and LSUN datasets. Label correctness measures the agreement
between default labels for the synthetic datasets, and those predicted by the annotator, a classifier
trained on the true data. Shown alongside are the equivalent inception scores computed using labels
predicted by the annotator (instead of the Inception Network). Training and test accuracies for a
linear model classifier on the various true and synthetic datasets are reported. Also presented are the
corresponding accuracies for a linear model trained on down-sampled true data (↓M ) and oversampled synthetic data (↑L ). Test accuracy for ResNets trained on these datasets is also shown (training
accuracy was always 100%), though it is noticeable that deep networks suffer from issues when
trained on synthetic datasets.

8

Under review as a conference paper at ICLR 2018

CelebA: Male/Female

CelebA: Smiling/Not Smiling

LSUN: Bedroom/Kitchen

CelebA: Black Hair/Not Black Hair

Figure 3: Illustration of the classification performance of true data vs. GAN-generated synthetic
data. Classification is performed using a basic linear model, and performance is reported in terms of
accuracy on the test set of true data. The bold curve shows variation of the classification performance
of models trained on true data with the size of the training set (maximum size is N ). Here, training
sets of various sizes are obtained by down-sampling the true dataset at random. Dashed lines show
the performance of classifiers trained on various GAN-generated datasets of size N .
size (maximum size being N ). The dashed curves show test accuracy of classifiers trained on GAN
datasets, obtained by drawing N samples from GANs at the culmination of the training process.
A natural argument in the defense of GANs is that we can oversample them, i.e. generate datasets
much larger than the size of training data. Results for linear models trained using a 10-fold oversampling of GANs (drawing 10N samples), denoted by ↑10 , are show in Table 2.
Results: The major findings from these experiments are:
• Based on Table 2, and Figure 6, we see strong agreement between annotator labels and true
labels for synthetic data, on par with the scores for the test set of true data. It is thus apparent that
the GAN images are of high-quality, as expected based on the visual inspection. These scores
are lower for LSUN than CelebA, potentially due to lower quality of generated LSUN images.
From these results, we can get a broad understanding of how good GANs are at producing
convincing/representative samples from different classes across datasets. This also shows that
simple classification-based benchmarks can highlight relevant properties of synthetic datasets.
• The equivalent inception score is not very informative and is similar for the true (test set) and
synthetic datasets. This is not surprising given the simple nature of our binary classification task
and the fact that the true and synthetic datasets have almost a uniform distribution over labels.
• It is evident from Table 2 that there is a large performance gap between true and synthetic data
on classification tasks. Inspection of training accuracies shows that linear models are able to
nearly fit the synthetic datasets, but are grossly underfitting on true data. Given the high scores
of synthetic data on the previous experiments to assess dataset ‘quality’ (Table 2 and Figure 6),
it is likely that the poor classification performance is more indicative of lack of ‘diversity’.
• Comparing GAN performance to that of down-sampled true data reveals that the learned distribution, which was trained on datasets that have around hundred thousand data points exhibits
diversity that is on par with what only mere couple of hundreds of true data samples constitute! This shows that, at least from the point of view of classification, the diversity of the GAN
generated data is severely lacking.
9

Under review as a conference paper at ICLR 2018

• Oversampling GANs by 10-fold to produce larger datasets does not improve classification performance. The disparity between true and synthetic data remains nearly unchanged even after
this significant oversampling, further highlighting the lack of diversity in GANs.
In terms of the conclusions of relative performance of various GANs, we observe that WGAN and
ALI perform consistently better than the other GANs on both the CelebA and LSUN datasets. While
BEGAN samples have good perceptual quality (see Figures 1 and 4), it performs badly on our
classification tasks. On the other hand, WGAN samples have relatively poor visual quality but seem
to outperform most other GANs in classification tasks. This is a strong indicator of the need to
consider other metrics, such as the ones proposed in this paper, in addition to visual inspection to
study GANs. For LSUN, the gap between true and synthetic data is much larger, with the classifiers
getting near random performance on all the synthetic datasets. Note that these classifiers get poor
test accuracy on LSUN but are not overfitting on the training data. In this case, we speculate the
lower performance could be due to both lower quality and diversity of LSUN samples.
In summary, our key experimental finding is that even simple classification–based tests can hold
tremendous potential to shed insight on the learned distribution in GANs. This not only helps us
to get a deeper understanding of many of the underlying issues, but also provides with a more
quantitative and rigorous platform on which to compare different GANs. Our techniques could,
in principle, be also applied to assess other generative models such as Variational Auto-Encoders
(VAEs) Kingma & Welling (2014). However, VAEs have significant problems in generating realistic
samples on the datasets used in our analysis in the first place – see Arora & Zhang (2017).

4

C ONCLUSIONS

In this paper, we put forth techniques for examining the ability of GANs to capture key characteristics of the training data, through the lens of classification. Our tools are scalable, quantitative and
automatic (no need for visual inspection of images). They thus are capable of studying state-ofthe-art GANs on realistic, large-scale image datasets. Further, they serve as a mean to perform a
nuanced comparison of GANs and to identify their relative merits, including properties that cannot
be discerned from mere visual inspection.
We used the developed techniques to perform empirical studies on popular GANs on the CelebA and
LSUN datasets. Our examination showed that mode collapse is indeed a prevalent issue for GANs.
Also, we observed that synthetic GAN-generated datasets have significantly reduced diversity, at
least when examined from a classification perspective. In fact, the diversity of such synthetic data is
often few orders of magnitude smaller than that of the true data. Furthermore, this gap in diversity
does not seem to be bridged by simply producing much larger datasets by oversampling GANs.
Also, our methods can be viewed as benchmarks for GANs. From this perspective, we observed
that ALI consistently outperforms all the other studied GANs, both in terms of the extent of modecollapse, as well as, the diversity of the learned distribution when assessed from the classification
point-of-view. Finally, we noticed that good perceptual quality of samples does not necessarily correlate – and might sometimes even anti-correlate – with distribution diversity. These findings suggest
that we need to go beyond the visual inspection–based evaluations and look for more quantitative
tools for assessing quality of GANs, such as the ones presented in this paper.

10

Under review as a conference paper at ICLR 2018

R EFERENCES
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks.
In International Conference on Machine Learning (ICML), pp. 214–223, 2017.
Sanjeev Arora and Yi Zhang. Do GANs actually learn the distribution? An empirical study. arXiv
preprint arXiv:1706.08224, 2017.
David Berthelot, Tom Schumm, and Luke Metz. BEGAN: Boundary equilibrium generative adversarial networks. arXiv preprint arXiv:1703.10717, 2017.
Emily L Denton, Soumith Chintala, Rob Fergus, et al. Deep generative image models using a Laplacian pyramid of adversarial networks. In Advances in Neural Information Processing Systems
(NIPS), pp. 1486–1494, 2015.
Vincent Dumoulin, Ishmael Belghazi, Ben Poole, Alex Lamb, Martin Arjovsky, Olivier Mastropietro, and Aaron Courville. Adversarially learned inference. In International Conference on Learning Representations (ICLR), 2017.
Ian Goodfellow.
Nips 2016 tutorial:
arXiv:1701.00160, 2016.

Generative adversarial networks.

arXiv preprint

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair,
Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems (NIPS), pp. 2672–2680, 2014.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Computer Vision and Pattern Recognition (CVPR), pp. 770–778, 2016.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In International Conference
on Learning Representations (ICLR), 2014.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild.
In International Conference on Computer Vision (ICCV), 2015.
Luke Metz, Ben Poole, David Pfau, and Jascha Sohl-Dickstein. Unrolled generative adversarial
networks. arXiv preprint arXiv:1611.02163, 2016.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep
convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434, 2015.
David Rolnick, Andreas Veit, Serge Belongie, and Nir Shavit. Deep learning is robust to massive
label noise. arXiv preprint arXiv:1705.10694, 2017.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen.
Improved techniques for training GANs. In Advances in Neural Information Processing Systems
(NIPS), pp. 2234–2242, 2016.
Ashish Shrivastava, Tomas Pfister, Oncel Tuzel, Joshua Susskind, Wenda Wang, and Russell Webb.
Learning from simulated and unsupervised images through adversarial training. In Computer
Vision and Pattern Recognition (CVPR), July 2017.
L. Theis, A. van den Oord, and M. Bethge. A note on the evaluation of generative models. In
International Conference on Learning Representations (ICLR), 2016.
Yuhuai Wu, Yuri Burda, Ruslan Salakhutdinov, and Roger Grosse. On the quantitative analysis of
decoder-based generative models. arXiv preprint arXiv:1611.04273, 2016.
Fisher Yu, Yinda Zhang, Shuran Song, Ari Seff, and Jianxiong Xiao. LSUN: Construction
of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint
arXiv:1506.03365, 2015.
Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaolei Huang, Xiaogang Wang, and Dimitris
Metaxas. Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial
networks. arXiv preprint arXiv:1612.03242, 2016.
11

Under review as a conference paper at ICLR 2018

5

A PPENDIX

5.1

E XPERIMENTAL S ETUP

5.1.1

DATASETS FOR C LASSIFICATION TASKS

To assess GAN performance from the perspective of classification, we construct a set of classification tasks on the CelebA and LSUN datasets. In the case of the LSUN dataset, images are annotated
with scene category labels, which makes it straightforward to use this data for binary and multiclass classification. On the other hand, each image in the CelebA dataset is labeled with 40 binary
attributes. As a result, a single image has multiple associated attribute labels. Here, we can construct classification tasks by considering binary combinations of an attribute(s) (examples are shown
in Figures 1 and 4). Attributes used in our experiments were chosen such that the resulting dataset
was large, and classifiers trained on true data got high-accuracy so as to be good annotators for the
synthetic data.
True Data

DCGAN

WGAN

ALI

BEGAN

Improved GAN

No Makeup
+
Not Smiling
No Makeup
+
Smiling
Makeup
+
Not Smiling
Makeup
+

Smiling

(a) 4-class dataset from CelebA for attributes Makeup, Smiling.
True Data

DCGAN

WGAN

ALI

BEGAN

Improved GAN

No Bangs
+
Not Smiling
No Bangs
+
Smiling
Bangs
+
Not Smiling
Bangs
+

Smiling

(b) 4-class dataset from CelebA for attributes Bangs, Smiling.

Figure 4: Illustration of datasets from CelebA used in proposed classification-based benchmarks to
evaluate GANs. Shown alongside are images sampled from various unconditional GANs trained on
this dataset. Labels for the GAN samples are obtained using a pre-trained classifier as an annotator.
5.1.2

M ODELS

Benchmarks were performed on standard implementations •
•
•
•
•

DCGAN: https://github.com/carpedm20/DCGAN-tensorflow
WGAN: https://github.com/martinarjovsky/WassersteinGAN
ALI: https://github.com/IshmaelBelghazi/ALI
BEGAN :https://github.com/carpedm20/BEGAN-tensorflow
Improved GAN: https://github.com/openai/improved-gan
12

Under review as a conference paper at ICLR 2018

True Data

DCGAN

WGAN

ALI

Improved GAN

Bedroom

Classroom

Kitchen

(a) 3-class dataset from LSUN for Bedroom, Classroom, Kitchen.
True Data

DCGAN

WGAN

ALI

Improved GAN

Bedroom

Conference
Room
Dining
Room
Kitchen

Living
Room

(b) 5-class dataset from LSUN for Bedroom, Conference Room, Dining Room, Kitchen, Living Room.

Figure 5: Illustration of datasets from LSUN used in proposed classification-based benchmarks to
evaluate GANs. Shown alongside are images sampled from various unconditional GANs trained on
this dataset. Labels for the GAN samples are obtained using a pre-trained classifier as an annotator.

• ResNet Classifier: Variation of the standard TensorFlow ResNet https://github.com/
tensorflow/models/blob/master/research/resnet/resnet_model.py
5.2
5.2.1

A DDITIONAL E XPERIMENTAL R ESULTS
S AMPLE Q UALITY

For each of our benchmark experiments, we ascertain that the visual quality of samples produced
by the GANs is comparable to that reported in prior work. Examples of random samples drawn for
multi-class datasets from both true and synthetic data are shown in Figures 1 and 4 for the CelebA
dataset, and in Figure!5 for the LSUN dataset.
5.2.2

D IVERSITY E XPERIMENTS

Sections 2.2 and 3.4 describe techniques to study classification performance of labeled GANgenerated datasets against that of true data. In order to assess the quality of our GAN-generated
synthetic datasets, we evaluate label agreement between the default labels of the synthetic datasets,
13

Under review as a conference paper at ICLR 2018

Figure 6: Histograms of annotator confidence (softmax probability) during label prediction on true
data (test set) and synthetic data for tasks on the CelebA and LSUN datasets (see Section 3.4).
and that obtained using the annotators described in Section 2.1 as shown in Table 2. In Figure 6,
we inspect the confidence scores, defined as the softmax probabilities for predicted class, of the annotator while making these predictions. As can be seen from these results, the annotator confidence
for the synthetic data is comparable to that on the test set of true data. Thus, it seems likely that the
GAN generated samples are of good quality and are truly representative examples of their respective
classes, as expected based on visual inspection.

14

