arXiv:1709.01215v2 [stat.ML] 5 Nov 2017

ALICE: Towards Understanding Adversarial
Learning for Joint Distribution Matching

Chunyuan Li1 , Hao Liu2 , Changyou Chen3 , Yunchen Pu1 , Liqun Chen1 ,
Ricardo Henao1 and Lawrence Carin1
1
Duke University 2 Nanjing University 3 University at Buffalo
cl319@duke.edu

Abstract
We investigate the non-identifiability issues associated with bidirectional adversarial training for joint distribution matching. Within a framework of conditional
entropy, we propose both adversarial and non-adversarial approaches to learn
desirable matched joint distributions for unsupervised and supervised tasks. We
unify a broad family of adversarial models as joint distribution matching problems.
Our approach stabilizes learning of unsupervised bidirectional adversarial learning
methods. Further, we introduce an extension for semi-supervised learning tasks.
Theoretical results are validated in synthetic data and real-world applications.

1

Introduction

Deep directed generative models are a powerful framework for modeling complex data distributions.
Generative Adversarial Networks (GANs) [1] can implicitly learn the data generating distribution;
more specifically, GAN can learn to sample from it. In order to do this, GAN trains a generator to
mimic real samples, by learning a mapping from a latent space (where the samples are easily drawn)
to the data space. Concurrently, a discriminator is trained to distinguish between generated and real
samples. The key idea behind GAN is that if the discriminator finds it difficult to distinguish real from
artificial samples, then the generator is likely to be a good approximation to the true data distribution.
In its standard form, GAN only yields a one-way mapping, i.e., it lacks an inverse mapping mechanism
(from data to latent space), preventing GAN from being able to do inference. The ability to compute
a posterior distribution of the latent variable conditioned on a given observation may be important
for data interpretation and for downstream applications (e.g., classification from the latent variable)
[2, 3, 4, 5, 6, 7]. Efforts have been made to simultaneously learn an efficient bidirectional model
that can produce high-quality samples for both the latent and data spaces [3, 4, 8, 9, 10, 11]. Among
them, the recently proposed Adversarially Learned Inference (ALI) [4, 10] casts the learning of such
a bidirectional model in a GAN-like adversarial framework. Specifically, a discriminator is trained to
distinguish between two joint distributions: that of the real data sample and its inferred latent code,
and that of the real latent code and its generated data sample.
While ALI is an inspiring and elegant approach, it tends to produce reconstructions that are not
necessarily faithful reproductions of the inputs [4]. This is because ALI only seeks to match two
joint distributions, but the dependency structure (correlation) between the two random variables
(conditionals) within each joint is not specified or constrained. In practice, this results in solutions
that satisfy ALI’s objective and that are able to produce real-looking samples, but have difficulties
reconstructing observed data [4]. ALI also has difficulty discovering the correct pairing relationship
in domain transformation tasks [12, 13, 14].
In this paper, (i) we first describe the non-identifiability issue of ALI. To solve this problem, we
propose to regularize ALI using the framework of Conditional Entropy (CE), hence we call the
proposed approach ALICE. (ii) Adversarial learning schemes are proposed to estimate the conditional
31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA.

entropy, for both unsupervised and supervised learning paradigms. (iii) We provide a unified view
for a family of recently proposed GAN models from the perspective of joint distribution matching,
including ALI [4, 10], CycleGAN [12, 13, 14] and Conditional GAN [15]. (iv) Extensive experiments
on synthetic and real data demonstrate that ALICE is significantly more stable to train than ALI, in
that it consistently yields more viable solutions (good generation and good reconstruction), without
being too sensitive to perturbations of the model architecture, i.e., hyperparameters. We also show
that ALICE results in more faithful image reconstructions. (v) Further, our framework can leverage
paired data (when available) for semi-supervised tasks. This is empirically demonstrated on the
discovery of relationships for cross domain tasks based on image data.

2

Background

Consider two general marginal distributions q(x) and p(z) over x ∈ X and z ∈ Z. One domain
can be inferred based on the other using conditional distributions, q(z|x) and p(x|z). Further, the
combined structure of both domains is characterized by joint distributions q(x, z) = q(x)q(z|x) and
p(x, z) = p(z)p(x|z).
To generate samples from these random variables, adversarial methods [1] provide a sampling
mechanism that only requires gradient backpropagation, without the need to specify the conditional
densities. Specifically, instead of sampling directly from the desired conditional distribution, the
random variable is generated as a deterministic transformation of two inputs, the variable in the source
domain, and an independent noise, e.g., a Gaussian distribution. Without loss of generality, we use an
universal distribution approximator specification [9], i.e., the sampling procedure for conditionals
x̃ ∼ pθ (x|z) and z̃ ∼ qφ (z|x) is carried out through the following two generating processes:
x̃ = gθ (z, ), z ∼ p(z),  ∼ N (0, I), and z̃ = gφ (x, ζ), x ∼ q(x), ζ ∼ N (0, I),
(1)
where gθ (·) and gφ (·) are two generators, specified as neural networks with parameters θ and
φ, respectively. In practice, the inputs of gθ (·) and gφ (·) are simple concatenations, [z ] and
[x ζ], respectively. Note that (1) implies that pθ (x|z) and qφ (z|x) are parameterized by θ and φ
respectively, hence the subscripts.
R
The goal of GAN [1] is to match the marginal pθ (x) = pθ (x|z)p(z)dz to q(x). Note that q(x)
denotes the true distribution of the data (from which we have samples) and p(z) is specified as a
simple parametric distribution, e.g., isotropic Gaussian. In order to do the matching, GAN trains
a ω-parameterized adversarial discriminator network, fω (x), to distinguish between samples from
pθ (x) and q(x). Formally, the minimax objective of GAN is given by the following expression:
min max LGAN (θ, ω) = Ex∼q(x) [log σ(fω (x))] + Ex̃∼pθ (x|z),z∼p(z) [log(1 − σ(fω (x̃)))], (2)
θ

ω

where σ(·) is the sigmoid function. The following lemma characterizes the solutions of (2) in terms
of marginals pθ (x) and q(x).
Lemma 1 ([1]) The optimal decoder and discriminator, parameterized by {θ ∗ , ω ∗ }, correspond to
a saddle point of the objective in (2), if and only if pθ∗ (x) = q(x).
Alternatively, ALI [4] matches the joint distributions pθ (x, z) = pθ (x|z)p(z) and qφ (x, z) =
q(x)qφ (z|x), using an adversarial discriminator network similar to (2), fω (x, z), parameterized by
ω. The minimax objective of ALI can be then written as
min max LALI (θ, φ, ω) = Ex∼q(x),z̃∼qφ (z|x) [log σ(fω (x, z̃))]
θ,φ ω
(3)
+ Ex̃∼pθ (x|z),z∼p(z) [log(1−σ(fω (x̃, z)))].
Lemma 2 ([4]) The optimum of the two generators and the discriminator with parameters
{θ ∗ , φ∗ , ω ∗ } form a saddle point of the objective in (3), if and only if pθ∗ (x, z) = qφ∗ (x, z).
From Lemma 2, if a solution of (3) is achieved, it is guaranteed that all marginals and conditional
distributions of the pair {x, z} match. Note that this implies that qφ (z|x) and pθ (z|x) match;
however, (3) imposes no restrictions on these two conditionals. This is key for the identifiability
issues of ALI described below.

3

Adversarial Learning with Information Measures

The relationship (mapping) between random variables x and z is not specified or constrained by
ALI. As a result, it is possible that the matched distribution π(x, z) , pθ∗ (x, z) = qφ∗ (x, z) is
undesirable for a given application.
2

To illustrate this issue, Figure 1 shows all solutions
z1
z2
z1 z2
z1 z2
(saddle points) to the ALI objective on a simple toy
problem. The data and latent random variables can take
two possible values, X = {x1 , x2 } and Z = {z1 , z2 },
x1 x2
x1
x2
x1 x2
respectively. In this case, their marginals q(x) and p(z)
z1
z2
z1
z2
z1 z2
are known, i.e., q(x = x1 ) = 0.5 and p(z = z1 ) = 0.5.
x1 0
1/2
The matched joint distribution, π(x, z), can be repre- x1 /2 (1 )/2 x1 1/2 0
1/2
x2 1/2 0
x2 0
sented as a 2 × 2 contingency table. Figure 1(a) repre- x2 (1 )/2 /2
sents all possible solutions of the ALI objective in (3),
(c)
(b)
(a)
for any δ ∈ [0, 1]. Figures 1(b) and 1(c) represent opposite extreme solutions when δ = 1 and δ = 0, respec- Figure 1: Illustration of possible solutions to
tively. Note that although we can generate “realistic” the ALI objective. The first row shows the mapvalues of x from any sample of p(z), for 0 < δ < 1, we pings between two domains, The second row
will have poor reconstruction ability since the sequence shows matched joint distribution, π(x, z), as
x ∼ q(x), z̃ ∼ qφ (z|x), x̃ ∼ pθ (x|z̃), can easily contingency tables parameterized by δ = [0, 1].
result in x̃ 6= x. The two (trivial) exceptions where the model can achieve perfect reconstruction
correspond to δ = {1, 0}, and are illustrated in Figures 1(b) and 1(c), respectively. From this simple
example, we see that due to the flexibility of the joint distribution, π(x, z), it is quite likely to obtain
an undesirable solution to the ALI objective. For instance, i) one with poor reconstruction ability or
ii) one where a single instance of z can potentially map to any possible value in X , e.g., in Figure 1(a)
with δ = 0.5, z1 can generate either x1 or x2 with equal probability.
Many applications require meaningful mappings. Consider two scenarios:
• A1: In unsupervised learning, one desirable property is cycle-consistency [12], meaning that the
inferred z of a corresponding x, can reconstruct x itself with high probability. In Figure 1 this
corresponds to either δ → 1 or δ → 0, as in Figures 1(b) and 1(c).
• A2: In supervised learning, the pre-specified correspondence between samples imposes restrictions
on the mapping between x and z, e.g., in image tagging, x are images and z are tags. In this case,
paired samples from the desired joint distribution are usually available, thus we can leverage this
supervised information to resolve the ambiguity between Figure 1(b) and (c).
From our simple example in Figure 1, we see that in order to alleviate the identifiability issues
associated with the solutions to the ALI objective, we have to impose constraints on the conditionals
qφ (z|x) and pθ (z|x). Furthermore, to fully mitigate the identifiability issues we require supervision,
i.e., paired samples from domains X and Z.
To deal with the problem of undesirable but matched joint distributions, below we propose to use
an information-theoretic measure to regularize ALI. This is done by controlling the “uncertainty”
between pairs of random variables, i.e., x and z, using conditional entropies.
3.1 Conditional Entropy
Conditional Entropy (CE) is an information-theoretic measure that quantifies the uncertainty of
random variable x when conditioned on z (or the other way around), under joint distribution π(x, z):
H π (x|z) , −Eπ(x,z) [log π(x|z)], and H π (z|x) , −Eπ(x,z) [log π(z|x)].

(4)

min max LALICE (θ, φ, ω) = LALI (θ, φ, ω) + LπCE (θ, φ).

(5)

π

π

The uncertainty of x given z is linked with H (x|z); in fact, H (x|z) = 0 if only if x is a
deterministic mapping of z. Intuitively, by controlling the uncertainty of qφ (z|x) and pθ (z|x), we
can restrict the solutions of the ALI objective to joint distributions whose mappings result in better
reconstruction ability. Therefore, we propose to use the CE in (4), denoted as LπCE (θ, φ) = H π (x|z)
or H π (z|x) (depending on the task; see below), as a regularization term in our framework, termed
ALI with Conditional Entropy (ALICE), and defined as the following minimax objective:
θ,φ

ω

LπCE (θ, φ) is dependent on the underlying distributions for the random variables, parametrized by
(θ, φ), as made clearer below. Ideally, we could select the desirable solutions of (5) by evaluating
their CE, once all the saddle points of the ALI objective have been identified. However, in practice,
LπCE (θ, φ) is intractable because we do not have access to the saddle points beforehand. Below, we
propose to approximate the CE in (5) during training for both unsupervised and supervised tasks.
Since x and z are symmetric in terms of CE according to (4), we use x to derive our theoretical
results. Similar arguments hold for z, as discussed in the Supplementary Material (SM).
3

3.2 Unsupervised Learning
In the absence of explicit probability distributions needed for computing the CE, we can bound
the CE using the criterion of cycle-consistency [12]. We denote the reconstruction of x as x̂, via
generating procedure (cycle) x̂ ∼ pθ (x̂|z), z ∼ qφ (z|x), x ∼ q(x). We desire that pθ (x̂|z) have
high likelihood for x̂ = x, for the x ∼ q(x) that begins the cycle x → z → x̂, and hence that x̂
be similar to the original x. Lemma 3 below shows that cycle-consistency is an upper bound of the
conditional entropy in (4).
Lemma 3 For joint distributions pθ (x, z) or qφ (x, z), we have
H qφ (x|z) , −Eqφ (x,z) [log qφ (x|z)] = −Eqφ (x,z) [log pθ (x|z)] − Eqφ (z) [KL(qφ (x|z)kpθ (x|z))]
≤ −Eqφ (x,z) [log pθ (x|z)] , LCycle (θ, φ).

(6)

R

where qφ (z) = dxqφ (x, z). The proof is in the SM. Note that latent z is implicitly involved
in LCycle (θ, φ) via Eqφ (x,z) [·]. For the unsupervised case we want to leverage (6) to optimize the
following upper bound of (5):
min max LALI (θ, φ, ω) + LCycle (θ, φ) .
θ,φ

ω

(7)

Note that as ALI reaches its optimum, pθ (x, z) and qφ (x, z) reach saddle point π(x, z), then
LCycle (θ, φ) → H qφ (x|z) → H π (x|z) in (4) accordingly, thus (7) effectively approaches (5)
(ALICE). Unlike LπCE (θ, φ) in (4), its upper bound, LCycle (θ, φ), can be easily approximated via
Monte Carlo simulation. Importantly, (7) can be readily added to ALI’s objective without additional
changes to the original training procedure.
The cycle-consistency property has been previously leveraged in CycleGAN [12], DiscoGAN [13]
and DualGAN [14]. However, in [12, 13, 14], cycle-consistency, LCycle (θ, φ), is implemented via `k
losses, for k = 1, 2, and real-valued data such as images. As a consequence of an `2 -based pixel-wise
loss, the generated samples tend to be blurry [8]. Recognizing this limitation, we further suggest
to enforce cycle-consistency (for better reconstruction) using fully adversarial training (for better
generation), as an alternative to LCycle (θ, φ) in (7). Specifically, to reconstruct x, we specify an
η-parameterized discriminator fη (x, x̂) to distinguish between x and its reconstruction x̂:
min max LA
Cycle (θ, φ, η) = Ex∼q(x) [log σ(fη (x, x))]
θ,φ

η

+ Ex̂∼pθ (x̂|z),z∼qφ (z|x) log(1 − σ(fη (x, x̂)))].

(8)

Finally, the fully adversarial training algorithm for unsupervised learning using the ALICE framework
is the result of replacing LCycle (θ, φ) with LA
Cycle (θ, φ, η) in (7); thus, for fixed (θ, φ), we maximize
wrt {ω, η}.

The use of paired samples {x, x̂} in (8) is critical. It encourages the generators to mimic the
reconstruction relationship implied in the first joint; on the contrary, the model may reduce to the
basic GAN discussed in Section 3, and generate any realistic sample in X . The objective in (8)
enjoys many theoretical properties of GAN. Particularly, Proposition 1 guarantees the existence of
the optimal generator and discriminator.
Proposition 1 The optimal generators and discriminator {θ ∗ , φ∗ , η ∗ } of the objective in (8) is
achieved, if and only if Eqφ∗ (z|x) pθ∗ (x̂|z) = δ(x − x̂).
The proof is provided in the SM. Together with Lemma 2 and 3, we can also show that:
Corollary 1 When cycle-consistency is satisfied (the optimum in (8) is achieved), (i) a deterministic mapping enforces Eqφ (z) [KL(qφ (x|z)kpθ (x|z))] = 0, which indicates the conditionals are
matched. (ii) On the contrary, the matched conditionals enforce H qφ (x|z) = 0, which indicates the
corresponding mapping becomes deterministic.

3.3 Semi-supervised Learning
When the objective in (7) is optimized in an unsupervised way, the identifiability issues associated
with ALI are largely reduced due to the cycle-consistency-enforcing bound in Lemma 3. This
means that samples in the training data have been probabilistically “paired” with high certainty,
by conditionals pθ (x|z) and pφ (z|x), though perhaps not in the desired configuration. In realworld applications, obtaining correctly paired data samples for the entire dataset is expensive or
4

even impossible. However, in some situations obtaining paired data for a very small subset of the
observations may be feasible. In such a case, we can leverage the small set of empirically paired
samples, to further provide guidance on selecting the correct configuration. This suggests that ALICE
is suitable for semi-supervised classification.
For a paired sample drawn from empirical distribution π̃(x, z), its desirable joint distribution is well
specified. Thus, one can directly approximate the CE as
H π̃ (x|z) ≈ Eπ̃(x,z) [log pθ (x|z)] , LMap (θ) ,

(9)

where the approximation (≈) arises from the fact that pθ (x|z) is an approximation to π̃(x|z). For
the supervised case we leverage (9) to approximate (5) using the following minimax objective:
min max LALI (θ, φ, ω) + LMap (θ).
θ,φ

ω

(10)

Note that as ALI reaches its optimum, pθ (x, z) and qφ (x, z) reach saddle point π(x, z), then
LMap (θ) → H π̃ (x|z) → H π (x|z) in (4) accordingly, thus (10) approaches (5) (ALICE).

We can employ standard losses for supervised learning objectives to approximate LMap (θ) in (10),
such as cross-entropy or `k loss in (9). Alternatively, to also improve generation ability, we propose
an adversarial learning scheme to directly match pθ (x|z) to the paired empirical conditional π̃(x|z),
using conditional GAN [15] as an alternative to LMap (θ) in (10). The χ-parameterized discriminator
fχ is used to distinguish the true pair {x, z} from the artificially generated one {x̂, z} (conditioned
on z), using
min max LA
Map (θ, χ) = Ex,z∼π̃(x,z) [log σ(fχ (x, z)) + Ex̂∼pθ (x̂|z) log(1 − σ(fχ (x̂, z)))]. (11)
θ

χ

The fully adversarial training algorithm for supervised learning using the ALICE in (11) is the result
of replacing LMap (θ) with LA
Map (θ, χ) in (10), thus for fixed (θ, φ) we maximize wrt {ω, χ}.
Proposition 2 The optimum of generators and discriminator {θ ∗ , χ∗ } form saddle points of objective
in (11), if and only if π̃(x|z) = pθ∗ (x|z) and π̃(x, z) = pθ∗ (x, z).
The proof is provided in the SM. Proposition 2 enforces that the generator will map to the correctly
paired sample in the other space. Together with the theoretical result for ALI in Lemma 2, we have
Corollary 2 When the optimum in (10) is achieved, π̃(x, z) = pθ∗ (x, z) = qφ∗ (x, z).
Corollary 2 indicates that ALI’s drawbacks associated with identifiability issues can be alleviated for
the fully supervised learning scenario. Two conditional GANs can be used to boost the perfomance,
each for one direction mapping. When tying the weights of discriminators of two conditional GANs,
ALICE recovers Triangle GAN [16]. In practice, samples from the paired set π̃(x, z) often contain
enough information to readily approximate the sufficient statistics of the entire dataset. In such case,
we may use the following objective for semi-supervised learning:
min max LALI (θ, φ, ω) + LCycle (θ, φ) + LMap (θ) .
θ,φ

ω

(12)

The first two terms operate on the entire set, while the last term only applies to the paired subset. Note
that we can train (12) fully adversarially by replacing LCycle (θ, φ) and LMap (θ) with LA
Cycle (θ, φ, η)
A
and LMap (θ, χ) in (8) and (11), respectively. In (12) each of the three terms are treated with equal
weighting in the experiments if not specificially mentioned, but of course one may introduce additional
hyperparameters to adjust the relative emphasis of each term.

4

Related Work: A Unified Perspective for Joint Distribution Matching

Connecting ALI and CycleGAN. We provide an information theoretical interpretation for cycleconsistency, and show that it is equivalent to controlling conditional entropies and matching conditional distributions. When cycle-consistency is satisfied, Corollary 1 shows that the conditionals
are matched in CycleGAN. They also train additional discriminators to guarantee the matching of
marginals for x and z using the original GAN objective in (2). This reveals the equivalence between
ALI and CycleGAN, as the latter can also guarantee the matching of joint distributions pθ (x, z) and
qφ (x, z). In practice, CycleGAN is easier to train, as it decomposes the joint distribution matching
objective (as in ALI) into four subproblems. Our approach leverages a similar idea, and further
improves it with adversarially learned cycle-consistency, when high quality samples are of interest.
5

(a) True x

(b) True z

(c) Inception Score

(d) MSE

Figure 2: Quantitative evaluation of generation (c) and reconstruction (d) results on toy data (a,b).
Stochastic Mapping vs. Deterministic Mapping. We propose to enforce the cycle-consistency in
ALI for the case when two stochastic mappings are specified as in (1). When cycle-consistency is
achieved, Corollary 1 shows that the bounded conditional entropy vanishes, and thus the corresponding
mapping reduces to be deterministic. In the literture, one deterministic mapping has been empirically
tested in ALI’s framework [4], without explicitly specifying cycle-consistency. BiGAN [10] uses
two deterministic mappings. In theory, deterministic mappings guarantee cycle-consistency in ALI’s
framework. However, to achieve this, the model has to fit a delta distribution (deterministic mapping)
to another distribution in the sense of KL divergence (see Lemma 3). Due to the asymmetry of
KL, the cost function will pay extremely low cost for generating fake-looking samples [17]. This
explains the underfitting reasoning in [4] behind the subpar reconstruction ability of ALI. Therefore,
in ALICE, we explicitly add a cycle-consistency regularization to accelerate and stabilize training.
Conditional GANs as Joint Distribution Matching. Conditional GAN and its variants [15, 18, 19,
20] have been widely used in supervised tasks. Our scheme to learn conditional entropy borrows the
formulation of conditional GAN [15]. To the authors’ knowledge, this is the first attempt to study the
conditional GAN formulation as joint distribution matching problem. Moreover, we add the potential
to leverage the well-defined distribution implied by paired data, to resolve the ambiguity issues of
unsupervised ALI variants [4, 10, 12, 13, 14].

5

Experimental Results

The code to reproduce these experiments is at https://github.com/ChunyuanLI/ALICE
5.1 Effectiveness and Stability of Cycle-Consistency
To highlight the role of the CE regularization for unsupervised learning, we perform an experiment
on a toy dataset. q(x) is a 2D Gaussian Mixture Model (GMM) with 5 mixture components, and
p(z) is chosen as a standard Gaussian, N (0, I). Following [4], the covariance matrices and centroids
are chosen such that the distribution exhibits severely separated modes, which makes it a relatively
hard task despite its 2D nature. Following [21], to study stability, we run an exhaustive grid search
over a set of architectural choices and hyper-parameters, 576 experiments for each method. We report
Mean Squared Error (MSE) and inception score (denoted as ICP) [22] to quantitatively evaluate the
performance of generative models. MSE is a proxy for reconstruction quality, while ICP reflects the
plausibility and variety of sample generation. Lower MSE and higher ICP indicate better results. See
SM for the details of the grid search and the calculation of ICP.
We train on 2048 samples, and test on 1024 samples. The ground-truth test samples for x and z are
shown in Figure 2(a) and (b), respectively. We compare ALICE, ALI and Denoising Auto-Encoders
(DAEs) [23], and report the distribution of ICP and MSE values, for all (576) experiments in Figure 2
(c) and (d), respectively. For reference, samples drawn from the “oracle” (ground-truth) GMM yield
ICP=4.977±0.016. ALICE yields an ICP larger than 4.5 in 77% of experiments, while ALI’s ICP
wildly varies across different runs. These results demonstrate that ALICE is more consistent and
quantitatively reliable than ALI. The DAE yields the lowest MSE, as expected, but it also results in
the weakest generation ability. The comparatively low MSE of ALICE demonstrates its acceptable
reconstruction ability compared to DAE, though a very significantly improvement over ALI.
Figure 3 shows the qualitative results on the test set. Since ALI’s results vary largely from trial to
trial, we present the one with highest ICP. In the figure, we color samples from different mixture
components to highlight their correspondance between the ground truth, in Figure 2(a), and their
reconstructions, in Figure 3 (first row, columns 2, 4 and 6, for ALICE, ALI and DAE, respectively).
Importantly, though the reconstruction of ALI can recover the shape of manifold in x (Gaussian
mixture), each individual reconstructed sample can be substantially far away from its “original”
mixture component (note the highly mixed coloring), hence the poor MSE. This occurs because the
adversarial training in ALI only requires that the generated samples look realistic, i.e., to be located
6

(a) ALICE

(b) ALI

(c) DAEs

Figure 3: Qualitative results on toy data. Two-column blocks represent the results of each method, with left for
z and right for x. For the first row, left is sampling of z, and right is reconstruction of x. Colors indicate mixture
component membership. The second row shows reconstructions, x, from linearly interpolated samples in z.

near true samples in X , but the mapping between observed and latent spaces (x → z and z → x) is
not specified. In the SM we also consider ALI with various combinations of stochastic/deterministic
mappings, and conclude that models with deterministic mappings tend to have lower reconstruction
ability but higher generation ability. In terms of the estimated latent space, z, in Figure 3 (first row,
columns 1, 3 and 5, for ALICE, ALI and DAE, respectively), we see that ALICE results in a better
latent representation, in the sense of mapping consistency (samples from different mixture components
remain clustered) and distribution consistency (samples approximate a Gaussian distribution). The
results for reconstruction of z and sampling of x are shown in the SM.
In Figure 3 (second row), we also investigate latent space interpolation between a pair of test set
examples. We use x1 = [−2.2, −2.2] and x9 = [2.2, 2.2], map them into z 1 and z 9 , linearly
interpolate between z 1 and z 9 to get intermediate points z 2 , . . . , z 8 , and then map them back to the
original space as x2 , . . . , x8 . We only show the index of the samples for better visualization. Figure 3
shows that ALICE’s interpolation is smooth and consistent with the ground-truth distributions.
Interpolation using ALI results in realistic samples (within mixture components), but the transition is
not order-wise consistent. DAEs provides smooth transitions, but the samples in the original space
look unrealistic as some of them are located in low probability density regions of the true model.
We investigate the impact of different amount of regularization on three datasets, including the toy
dataset, MNIST and CIFAR-10 in SM Section D. The results show that our regularizer can improve
image generation and reconstruction of ALI for a large range of weighting hyperparameter values.
5.2 Reconstruction and Cross-Domain Transformation on Real Datasets
Two image-to-image translation tasks are considered. (i) Car-to-Car [24]: each domain (x and z)
includes car images in 11 different angles, on which we seek to demonstrate the power of adversarially
learned reconstruction and weak supervision. (ii) Edge-to-Shoe [25]: x domain consists of shoe
photos and z domain consists of edge images, on which we report extensive quantitative comparisons.
Cycle-consistency is applied on both domains. The goal is to discover the cross-domain relationship
(i.e., cross-domain prediction), while maintaining reconstruction ability on each domain.

Classification Accuracy (%)

Adversarially learned reconstruction To demonstrate the effectiveness of our fully adversarial
scheme in (8) (Joint A.) on real datasets, we use it in place of the `2 losses in DiscoGAN [13]. In
practice, feature matching [22] is used to help the adversarial objective in (8) to reach its optimum.
We also compared with a baseline scheme (Marginal A.) in [12], which adversarially discriminates
between x and its reconstruction x̂.
80
The results are shown in Figure 4 (a). From Inputs
60
top to bottom, each row shows ground-truth Joint A.
ALICE (10% sup.)
ALICE (1% sup.)
images, DiscoGAN (with Joint A., `2 loss
DiscoGAN
40
BiGAN
`
loss
and Marginal A. schemes, respectively) and 2
20
BiGAN [10]. Note that BiGAN is the best Marginal A.
ALI variant in our grid search compasion.
BiGAN
2
4
6
8
10
The proposed Joint A. scheme can retain the
Number of Paired Angles
same crispness characteristic to adversarially(a) Reconstruction
(b) Prediction
trained models, while `2 tends to be blurry.
Figure 4: Results on Car-to-Car task.
Marginal A. provides realistic car images, but not faithful reproductions of the inputs. This explains
7

Lcycle+`A

Lcycle+`2
BiGAN

BiGAN

(a) Cross-domain transformation
(b) Reconstruction
(c) Generated edges
Figure 5: SSIM and generated images on Edge-to-Shoe dataset.

the observations in [12] in terms of no performance gain. The BiGAN learns the shapes of cars, but
misses the textures. This is a sign of underfitting, thus indicating BiGAN is not easy to train.
Weak supervision The DiscoGAN and BiGAN are unsupervised methods, and exhibit very different
cross-domain pairing configurations during different training epochs, which is indicative of nonidentifiability issues. We leverage very weak supervision to help with convergence and guide the
pairing. The results on shown in Figure 4 (b). We run each methods 5 times, the width of the
colored lines reflect the standard deviation. We start with 1% true pairs for supervision, which yields
significantly higher accuracy than DiscoGAN/BiGAN. We then provided 10% supervison in only 2
or 6 angles (of 11 total angles), which yields comparable angle prediction accuracy with full angle
supervison in testing. This shows ALICE’s ability in terms of zero-shot learning, i.e., predicting
unseen pairs. In the SM, we show that enforcing different weak supervision strategies affects the final
pairing configurations, i.e., we can leverage supervision to obtain the desirable joint distribution.
Quantitative comparison To quantitatively assess the generated images, we use structural similarity
(SSIM) [26], which is an established image quality metric that correlates well with human visual
perception. SSIM values are between [0, 1]; higher is better. The SSIM of ALICE on prediction and
reconstruction is shown in Figure 5 (a)(b) for the edge-to-shoe task. As a baseline, we set DiscoGAN
with `2 -based supervision (`2 -sup). BiGAN/ALI, highlighted with a circle is outperformed by
ALICE in two aspects: (i) In the unpaired setting (0% supervision), cycle-consistency regularization
(LCycle ) shows significant performance gains, particularly on reconstruction. (ii) When supervision
is leveraged (10%), SSIM is significantly increased on prediction. The adversarial-based supervision
(`A -sup) shows higher prediction than `2 -sup. ALICE achieves very similar performance with the
50% and full supervision setup, indicating its advantage of in semi-supervised learning. Several
generated edge images (with 50% supervision) are shown in Figure 5(c), `A -sup tends to provide
more details than `2 -sup. Both methods generate correct paired edges, and quality is higher than
BiGAN and DiscoGAN. In the SM, we also report MSE metrics, and results on edge domain only,
which are consistent with the results presented here.
One-side cycle-consistency When uncertainty in one domain is desirable, we consider one-side
cycle-consistency. This is demonstrated on the CelebA face dataset [27]. Each face is associated with
a 40-dimensional attribute vector. The results are in the Figure 13 of SM. In the first task, we consider
the images x are generated from a 128-dimensional Gaussian latent space z, and apply LCycle on x.
We compare ALICE and ALI on reconstruction in (a)(b). ALICE shows more faithful reproduction
of the input subjects. In the second task, we consider z as the attribute space, from which the images
x are generated. The mapping from x to z is then attribute classification. We only apply LCycle on
the attribute domain, and LA
Map on both domains. When 10% paired samples are considered, the
predicted attributes still reach 86% accuracy, which is comparable with the fully supervised case. To
test the diversity on x, we first predict the attributes of a true face image, and then generated multiple
images conditioned on the predicted attributes. Four examples are shown in (c).

6

Conclusion

We have studied the problem of non-identifiability in bidirectional adversarial networks. A unified
perspective of understanding various GAN models as joint matching is provided to tackle this problem.
This insight enables us to propose ALICE (with both adversarial and non-adversarial solutions) to
reduce the ambiguity and control the conditionals in unsupervised and semi-supervised learning. For
future work, the proposed view can provide opportunities to leverage the advantages of each model,
to advance joint-distribution modeling.
8

Acknowledgements We acknowledge Shuyang Dai, Chenyang Tao and Zihang Dai for helpful
feedback/editing. This research was supported in part by ARO, DARPA, DOE, NGA, ONR and NSF.

References
[1] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio.
Generative adversarial nets. In NIPS, 2014.
[2] D. P. Kingma and M. Welling. Auto-encoding variational Bayes. In ICLR, 2014.
[3] X. Chen, Y. Duan, R. Houthooft, J. Schulman, I. Sutskever, and P. Abbeel. InfoGAN: Interpretable
representation learning by information maximizing generative adversarial nets. In NIPS, 2016.
[4] V. Dumoulin, I. Belghazi, B. Poole, A. Lamb, M. A., O. Mastropietro, and A. Courville. Adversarially
learned inference. ICLR, 2017.
[5] L. Mescheder, S. Nowozin, and A. Geiger. Adversarial variational bayes: Unifying variational autoencoders
and generative adversarial networks. ICML, 2017.
[6] Y. Pu, Z. Gan, R. Henao, X. Yuan, C. Li, A. Stevens, and L. Carin. Variational autoencoder for deep
learning of images, labels and captions. In NIPS, 2016.
[7] Y. Pu, Z. Gan, R. Henao, C. Li, S. Han, and L. Carin. Vae learning via Stein variational gradient descent.
NIPS, 2017.
[8] A. B. L. Larsen, S. K. Sønderby, H. Larochelle, and O. Winther. Autoencoding beyond pixels using a
learned similarity metric. ICML, 2016.
[9] A. Makhzani, J. Shlens, N. Jaitly, I. Goodfellow, and B. Frey. Adversarial autoencoders. arXiv preprint
arXiv:1511.05644, 2015.
[10] J. Donahue, K. Philipp, and T. Darrell. Adversarial feature learning. ICLR, 2017.
[11] Y. Pu, W. Wang, R. Henao, L. Chen, Z. Gan, C. Li, and L. Carin. Adversarial symmetric variational
autoencoder. NIPS, 2017.
[12] J. Zhu, T. Park, P. Isola, and A. Efros. Unpaired image-to-image translation using cycle-consistent
adversarial networks. ICCV, 2017.
[13] T. Kim, M. Cha, H. Kim, J. Lee, and J. Kim. Learning to discover cross-domain relations with generative
adversarial networks. ICML, 2017.
[14] Z. Yi, H. Zhang, and P. Tan. DualGAN: Unsupervised dual learning for image-to-image translation. ICCV,
2017.
[15] M. Mirza and S. Osindero. Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784, 2014.
[16] Z. Gan, L. Chen, W. Wang, Y. Pu, Y. Zhang, H. Liu, C. Li, and L. Carin. Triangle generative adversarial
networks. NIPS, 2017.
[17] M. Arjovsky and L. Bottou. Towards principled methods for training generative adversarial networks. In
ICLR, 2017.
[18] S. Reed, Z. Akata, X. Yan, L. Logeswaran, B. Schiele, and H. Lee. Generative adversarial text to image
synthesis. In ICML, 2016.
[19] P. Isola, J. Zhu, T. Zhou, and A. Efros. Image-to-image translation with conditional adversarial networks.
CVPR, 2017.
[20] C. Li, K. Xu, J. Zhu, and B. Zhang. Triple generative adversarial nets. NIPS, 2017.
[21] J. Zhao, M. Mathieu, and Y. LeCun. Energy-based generative adversarial network. ICLR, 2017.
[22] T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen. Improved techniques for
training GANs. In NIPS, 2016.
[23] P. Vincent, H. Larochelle, Y. Bengio, and P. Manzagol. Extracting and composing robust features with
denoising autoencoders. In ICML, 2008.
[24] S. Fidler, S. Dickinson, and R. Urtasun. 3D object detection and viewpoint estimation with a deformable
3D cuboid model. In NIPS, 2012.
[25] A. Yu and K. Grauman. Fine-grained visual comparisons with local learning. In CVPR, 2014.
[26] Z. Wang, A. C Bovik, H. R Sheikh, and E. P Simoncelli. Image quality assessment: from error visibility to
structural similarity. IEEE trans. on Image Processing, 2004.
[27] Z. Liu, P. Luo, X. Wang, and X. Tang. Deep learning face attributes in the wild. In ICCV, 2015.
[28] S. Xie and Z. Tu. Holistically-nested edge detection. In ICCV, 2015.

9

Supplementary Material of
ALICE: Towards Understanding Adversarial Learning for
Joint Distribution Matching

A

Information Measures

Since our paper constrain correlation of two random variables using information theoretical measures,
we first review the related concepts. For any probability measure π on the random variables x and
z, we have the following additive and subtractive relationships for various information measures,
including Mutual Information (MI), Variation of Information (VI) and the Conditional Entropy (CE).
VI(x, z) = − Eπ(z,x) [log π(x|z)] − Eπ(x,z) [log π(z|x)]

(13)

π(x, z)
+ log π(x, z)]
π(x)π(z)
= − Iπ (x, z) + Hπ (x, z)
π(x, z)
+ log π(x, z)]
= − Eπ(z,x) [log
π(x)π(z)
= − 2Iπ (x, z) + Hπ (x) + Hπ (z)
= − Eπ(z,x) [log

A.1

(14)
(15)
(16)
(17)

Relationship between Mutual Information, Conditional Entropy and the Negative Log
Likelihood of Reconstruction

The following shows how the negative log probability (NLL) of the reconstruction is related to
variation of information and mutual information. On the support of (x, z), we denote q as the encoder
probability measure, and p as the decoder probability measure. Note that the reconstruction loss for
z can be writen as its log likelihood form as LR = −Ez∼p(z),x∼p(x|z) [log q(z|x)].
Lemma 4 For random variables x and z with two different probability measures, p(x, z) and
q(x, z), we have
Hp (z|x) = −Ez∼p(z),x∼p(x|z) [log p(z|x)]

(18)


= −Ez∼p(z),x∼p(x|z) [log q(z|x)] − Ez∼p(z),x∼p(x|z) log p(z|x) − log q(z|x)

= −Ez∼p(z),x∼p(x|z) [log q(z|x)] − Ep(x) (KL(p(z|x)kq(z|x)))
≤ −Ez∼p(z),x∼p(x|z) [log q(z|x)]



(19)
(20)
(21)

where Hp (z|x) is the conditional entropy. From lemma 4, we have
Corollary 3 For random variables x and z with probability measure p(x, z), the mutual information
between x and z can be written as
Ip (x, z) = Hp (z) − Hp (z|x) ≥ Hp (z) + Ez∼p(z),x∼p(x|z) [log q(z|x)].

(22)

Given a simple prior p(z) such as isotropic Gaussian, H(z) is a constant.
Corollary 4 For random variables x and z with probability measure p(x, z), the variation of
information between x and z can be written as
VIp (x, z) = Hp (x|z) + Hp (z|x) ≥ Hp (x|z) − Ez∼p(z),x∼p(x|z) [log q(z|x)].
10

(23)

B

Proof for Adversarial Learning Schemes

The proof for cycle-consistency and conditional GAN using adversarial traning is shown below. It
follows the proof of the original GAN paper: we first show the implication of optimal discriminator,
and then show the corresponding optimal generator.
B.1

Proof of Proposition 1: Adversarially Learned Cycle-Consistency for Unpair Data

In the unsupervised case, given data sample x, one desirable property is reconstruction. The following
game learns to reconstruct:
min max L(θ, φ, ω) = Ex∼q(x) [log σ(fω (x, x)) + Ez∼qφ (z|x),x̂∼pθ (x̂|z) log(1 − σ(fω (x, x̂)))]
ω

θ,φ

(24)
Proposition 3 For fixed (θ, φ), the optimal ω in (24) yields fω∗ (x, x̂) = Eqφ (z|x) pθ (x̂|z) =
δ(x̂ − x).
Proof We start from a simple observation
Ex∼q(x) log σ(fω (x, x)) = Ex∼q(x),x̂∼q̃(x̂|x) log σ(fω (x, x̂))

(25)

when q̃(x̂|x) , δ(x̂ − x). Therefore, the objective in (24) can be expressed as

Ex∼q(x),x̂∼q̃(x̂|x) log σ(fω (x, x̂)) + Ex∼q(x),z∼qφ (z|x),x̂∼pθ (x̂|z) log(1 − σ(fω (x, x̂)))
(26)

Z Z 
Z
=
q(x)q̃(x̂|x) log σ(fω (x, x̂)) + q(x)qφ (z|x)pθ (x̂|z) log(1 − σ(fω (x, x̂)))dz dxdx̂
x

x̂

z

(27)
Note that
Z

q(x)qφ (z|x)pθ (x̂|z) log(1 − σ(fω (x, x̂)))dz
Z
=q(x) log(1 − σ(fω (x, x̂))) qφ (z|x)pθ (x̂|z)dz

(28)

=q(x)[Eqφ (z|x) pθ (x̂|z)] log[1 − σ(fω (x, x̂))]

(30)

z

(29)

z

The expression in (26) is maximal as a function of fω (x, x̂) if and only if the integrand is maximal
a
for every (x, x̂). However, the problem maxt a log(t) + b log(1 − t) attains its maximum at t = a+b
,
showing that
σ(fω∗ (x, x̂)) =

q(x)q̃(x̂|x)
q̃(x̂|x)
=
q(x)q̃(x̂|x) + q(x)Eqφ (z|x) pθ (x̂|z)
q̃(x̂|x) + Eqφ (z|x) pθ (x̂|z)

(31)

For the game in (24), for which (θ, φ) are optimized as to most confuse the discriminator, the optimal
solution for the distribution parameters (θ ∗ , φ∗ ) yield σ(fω∗ (x, x̂)) = 1/2 [1], and therefore from
(31)
Eqφ∗ (z|x) pθ∗ (x̂|z) = δ(x − x̂).

(32)


Similarly, we can show the cycle consistency property for reconstructing z as Epθ∗ (x|z) qφ∗ (ẑ|x) =
δ(z − ẑ).
B.2

Proof of Proposition 2: Adversarially Learned Conditional Generation for Paired Data

In the supervised case, given the paired data sample π(x, z), the following game is used to conditionally generate x [15]:
min max L(θ, ω) = Ex,z∼π(x,z) [log σ(fω (x, z)) + Ex̃∼pθ (x̃|z) log(1 − σ(fω (x̃, z)))]
θ

ω

To show the results, we need the following Lemma:
11

(33)

Lemma 5 The optimial generator and discriminator, with parameters (θ ∗ , ω ∗ ), forms the saddle
points of game in (33), if and only if pθ∗ (x|z) = π(x|z). Further, pθ∗ (x, z) = π(x, z)
Proof For the observed paired data π(x, z), we have p(z) = π(z), where π(z) is marginal empirical
distribution of z for the paired data.
Also, π(x̃|z) = δ(x̃ − x) when x̃ is paired with z in the dataset. We start from the observation
Ex,z∼π(x,z) log σ(fω (x, z)) = Ez∼p(z),x̃∼π(x̃|z) log σ(fω (x̃, z))

(34)

Therefore, the objective in (33) can be expressed as
Ex∼p(z),x̃∼π(x̃|z) log σ(fω (x̃, z)) + Ez∼p(z),x̃∼pθ (x̃|z) log(1 − σ(fω (x̃, z)))

(35)

This integral is maximal as a function of fω (x, z) if and only if the integrand is maximal for every
a
(x, z). However, the problem maxt a log(t) + b log(1 − t) attains its maximum at t = a+b
, showing
that
σ(fω∗ (x, z)) =

p(x)π(x|z)
π(x|z)
=
p(x)π(x|z) + p(z)pθ (x|z)
π(x|z) + pθ (x|z)

(36)

or equivalently, the optimum generator is pθ∗ (x|z) = π(x|z). Since q(x) = π(x), we further have
pθ∗ (x, z) = π(x, z). Similarly, for conditional GAN of z, we can show that is qφ∗ (z|x) = π(z|x)
and qφ∗ (x, z) = π(x, z) for the Combining them, we show that pθ∗ (x, z) = π(x, z) = qφ∗ (x, z).


C
C.1

More Results on the Toy Data
The detailed setup

The 5-component Gaussian mixture model (GMM) in x is set with the means
(0, 0), (2, 2), (−2, 2), (2, −2), (−2, −2), and standard derivation 0.2. The Isotropic Gaussian
in z is set with mean (0, 0) and standard derivation 1.0.
We consider various network architectures to compare the stability of the methods. The hyperparameters includes: the number of layers and the number of neurons of the discriminator and two
generators, and the update frenquency for discriminator and generator. The grid search specification
is summarized in Table 1. Hence, the total number of experiments is 23 × 23 × 32 = 576.

A generalized version of the inception score is calculated, ICP = Ex KL(p(y)||p(y|x)), where
x denotes a generated sample and y is the label predicted by a classifier that is trained off-line
using the entire training set. It is also worth noting that although we inherit the name “inception
score” from [22], our evaluation is not related to the “inception” model trained on ImageNet dataset.
Our classifier is a regular 3-layer neural nets trained on the dataset of interest, which yields 100%
classification accuracy on this toy dataset.

(a) ALICE

(b) ALI

(c) DAEs

Figure 6: Qualitative results on toy data. Every two columns indicate the results of a method, with
left space as reconstruction of z and right space as sampling in x, respectively.
C.2 Reconstruction of z and sampling for x
We show the additional results for the econstruction of z and sampling for x in Figure 6. ALICE
shows good sampling ability, as it reflects the Guassian characteristics for each of 5 components,
while ALI’s samples tends to be concentrated, reflected by the shrinked Guassian components. DAE
learns an indentity mapping, and thus show weak generation ability.
12

C.3 Summary of the four variants of ALICE
ALICE is a general CE-based framework to regularize the objectives of bidiretional adversarial
training, in order to obtain desirable solutions. To clearly show the versatility of ALICE, we
summarize its four variants, and test their effectivenss on toy datasets.
In unsupervised learning, two forms of cycle-consistency/reconstruction are considered to bound CE:
• Explicit cycle-consistency: Explicitly specified `k -norm for reconstruction;
• Implicit cycle-consistency: Implicitly learned reconstruction via adversarial training
In semi-supervised learning, the pairwise information is leveraged in two forms to approximate CE:
• Explicit mapping: Explicitly specified `k -norm mapping (e.g., standard supervised losses);
• Implicit mapping: Implicitly learned mapping via adversarial training
Disucssion (i) Explicit methods such as `k losses (k = 1, 2): The similarity/quality of the reconstruction to the original sample is measured in terms of `k metric. This is easy to implement
and optimize. However, it may lead to visually low quality reconstruction in high dimensions. (ii)
Implicit methods via adversarial training: it essentially requires the reconstruction to be close to
the original sample in terms of `0 metric (see Section 3.3 of [10]: Adversarial feature learning). It
theoretically guarantees perfect reconstruction, however, this is hard to achieve in practice, espcially
in high dimension spaces.
Results The effectivenss of these algorithms are demonstrated on toy data of low dimension in Figure 7. The unsupervised variants are tested in the same toy dataset described above, the results are in
Figure 7 (a)(b). For the supervised variants, we create a toy dataset, where z-domain is 2-component
GMM, and x-domain is 5-component GMM. Since each domain is symmtric, ambiguity exists when
Cycle-GAN variants attempt to discover the relationship of the two domains in pure unsupervised
setting. Indeed, we observed random switching of the discoverd corresponded components in different
runs of Cycle-GAN. By adding a tiny fraction of pairwise information (a cheap way to specify the
desirable relationship ), we can easily learn the correct correspondences for the entire datasets. In Figure 7 (c)(d), 5 pairs (out of 2048) are pre-specified: the points [0, 0], [1, 1], [−1, −1], [1, −1], [−1, 1]
in x-domain are paired with the points in z-domain with opposite signs. Both explicit and implicit
ALICE find the correct pairing configurations for other unlabeled samples. This inspires us to
manually labeling the relations for a few samples between domains, and use ALICE to automatically
control the full datasets pairing for the real datasets. One example is shown on Car2Car dataset.
C.4 Comparisons of ALI with stochastic/deterministic mappings
We investigate the ALI model with different mappings:
• ALI: two stochastic mappings;
• ALI− : one stochastic mapping and one deterministic mapping;
• BiGAN: two deterministic mappings.
We plot the histogram of ICP and MSE in Fig. 8, and report the mean and standard derivation in
Table 2. In Fig. 9, we compare their reconstruction and generation ability. Models with deterministic
mapping have higher recontruction ability, while show lower sampling ability.
Comparison on Reconstruction Please see row 1 and 2 in Fig. 9. For reconstruction, we start from
one sample (red dot), and pass it through the cycle formed by the two mappings 100 times. The
resulted reconstructions are shown as blue dots. The reconstructed samples tends to be concentrated
with more deterministic mappings.
Comparison on Sampling Please see row 3 and 4 in Fig. 9. For sampling, we first draw 1024
samples in each domain, and pass them through the mappings. The generated samples are colored as
the index of Gaussian component it comes from in the original domain.

13

(b) Implicit Cycle-Consistency (Adversarial loss)

(a) Explicit Cycle-Consistency (L2 loss)

(c) Explicit Mapping (L2 loss)

(d) Implicit Mapping (Adversarial loss)

Figure 7: Results of four variants of ALICE on toy datasets.

Table 2: Testing MSE and ICP on toy dataset..
Table 1: Grid search specification.
Settings

Values

Number of layers
Number of neurons
Update frenquency

80.0%

[2, 3]
[256, 512]
[1, 3, 5]

ALICE
ALI
ALI−
BiGAN
DAEs

0.022 ± 0.029
4.856 ± 2.920
3.888 ± 7.343
2.399 ± 3.605
0.003 ± 0.004

4.595 ± 0.604
2.776 ± 1.516
3.420 ± 1.299
3.712 ± 1.278
2.913 ± 0.004

ALICE
ALI
ALIBiGAN
DAEs

80.0%

20.0%
0.0%1.0

ICP

Percentage

Percentage

40.0%

MSE

100.0%

ALICE
ALI
ALIBiGAN
DAEs

60.0%

Method

60.0%
40.0%
20.0%

1.5

2.0

2.5 3.0 3.5 4.0
Inception Score
(a) Inception Score

4.5

5.0

0.0%0.0

0.5

1.0

Figure 8: Quantitative results on toy data.

14

1.5

2.0
2.5
MSE
(b) MSE

3.0

3.5

4.0

3

2

2

1

1

1

1

0

0

0

0

1

1

1

1

2

2

2

2

3

3

3

1

0
z1

1

2

3

3

2

1

0
z1

1

2

3

3
3

2

1

0
z1

1

2

3

3

3

2

2

2

1

1

1

1

0

0

0

0

x2

3

2

x2

3

1

1

1

1

2

2

2

2

3

3

3

3

2

1

0
x1

1

2

3

3

2

1

0
x1

1

2

3

2

1

0
x1

1

2

3

3

2

2

1

1

1

1

0

0

0

0

z2

3

2

z2

3

2

1

1

1

1

2

2

2

2

3

3

3

3

2

1

0
z1

1

2

3

3

2

1

0
z1

1

2

3

2

1

0
z1

1

2

3

2

1

1

1

1

0

0

0

0

x2

3

2

x2

3

2

x2

3

2

1

1

1

1

2

2

2

2

3

3

3

0
x1

1

2

3

3

2

1

0
x1

1

2

3

2

1

0
x1

1

2

3

3

3

2

2

1

1

0

0

1

1

2

2

3
2

1

0
x1

1

(c) ALI−

(b) ALI

0
z1

1

2

3

3

2

1

0
x1

1

2

3

3

2

1

0
z1

1

2

3

3

2

1

0
x1

1

2

3

3

2

1

0
x1

1

2

3

3
3

(a) ALICE

1

3
3

x2

1

x2

2

2

3
3

3

3

3

3
3

3

z2

z2

2

x2

x2

3

x2

z2

3

2

z2

3

2

z2

z2

3

2

3

(c) BiGAN

Figure 9: Comparison with bidirectional GAN models with different stochastic or deterministic
mappings. The 1st row is the reconstruction of z, and the 2nd row is the reconstruction of x. In these
two rows, the red dot is the original data point, the blue dots are the reconstruction. The 3rd row is
the sampling of z, and 4th row is the sampling of x. and 5th row is the reconstruction for x. In the
3rd row, colors of the generated z indicate the component of x that z conditions on.

15

D

More Results on the Effectiveness of CE Regularizers

We investigate the effectiveness and impact of the proposed cycle-consistency regularizer (explicit `2
norm) on 3 datasets, including the toy dataset, MNIST and CIFAR-10. A large range of weighting
hyperparameter λ is tested. The inception scores on toy and MNIST datasets are evaluted by the
pre-trained “perfect” classifiers of these datasets, respectively, while inception scores on CIFAR is
based on ImageNet. The results for different λ are shown in Figure 10, and the best performance is
sumarized in Table 3.

Table 3: Compariso on real datasets.
Image generation (ICP ↑)

Image reconstruction (MSE ↓)

Settings

ALI

ALICE (λ = 1)

ALI

ALICE (λ = 10−6 )

MNIST
CIFAR

8.749 ± 0.09
5.93 ± 0.0437

9.279 ± 0.07
6.015 ± 0.0284

0.4803 ± 0.100
0.672 ± 0.1129

0.0803 ± 0.007
0.4155 ± 0.2015

6
Mean Square Error

Inception Score

5
4
3
true samples
ALI (w/o CE regularizer)

2
10−2

4

denoising auto-encoders
ALI (w/o CE regularizer)

2
0

10−1
100
Weighting hyperparameter

10−2

101

(a) Toy dataset: image generation
0.6
Mean Square Error

ALI (w/o CE regularizer)

Inception Score

8
6

2

true samples
ALI (w/o CE regularizer)
10−4

10−2
100
Weighting hyperparameter

102

0.4

0.2

10−4

104

(c) MNIST: image generation

10−2
100
Weighting hyperparameter

102

104

(d) MNIST: image reconstruction
0.8
Mean Square Error

6
Inception Score

101

(b) Toy dataset: image reconstruction

10

4

10−1
100
Weighting hyperparameter

4

2

0.6
ALI (w/o CE regularizer)

0.4
0.2

ALI (w/o CE regularizer)
10−5

10−3
10−1
Weighting hyperparameter

101

0.0

103

(e) CIFAR: image generation

10−5

10−3
10−1
Weighting hyperparameter

101

103

(f) CIFAR: image reconstruction

Figure 10: Impact of the proposed cycle-consistency regularizer. The “perfect” performance is shown
as a solid line, the ALI (i.e., without CE regularizer) performance is a dash line. ALICE with different
levels of regularization are shown as light blue dots, and best performance of ALICE is shown as the
dot with a dark blue circle.

16

E

More Details on Real Data Experiments

E.1 Car to Car Experiment
Setup The dataset [24] consists of rendered images of 3D car models with varying azimuth angles
at 15◦ intervals. 11 views of each car are used. The dataset is split into train set ( 169×11 = 1859
images) and test set ( 14×11 = 154 images), and further split the train set into two groups, each of
which is used as A domain and B domain samples. To evaluate, we trained a regressor and a classifier
that predict the azimuth angle using the train set. We map the car image from one domain to the
other, and then reconstruct to the original domain. The cycle-consistency is evaluted as the prediction
accuracy of the reconstructed images.
Table 4 shows the MSE and prediction accuracy by leverage the supervision in different number
of angles. To further demonstrate that we can easily control the correspondence configuration
by designing the proper supervision, we use ALICE to enforce coherent supervsion and opposite
supervision, respectively. Only 1% supervison information is used in each angle. We translated
images in the test set using each of the three trained models, and azimuth angles were predicted using
the regressor for both input and translated images. In Table 5, we show the cross domain relationship
discovered by each method. X and Y axis indicates predicted angles of original and transformed cars,
respectively. All three plots are results at the 10th epoch. Scatter points with supervision are more
concentrated on the diagnals in the plots, which indicates higher prediction/correlation. The learning
curves are shown in Table 5(d). The Y axis indicate the RMSE in angle prediction. We see that very
weak supervision can largely imporve the convergence results and speed. Example and comparison
arre shown in Figure11.
Table 5: The scatter plots on car2car.

Table 4: ACC and MSE in prediction
on car translation. The top four methods are our methods reported in the
format of #Angle (supervison%).

80

80

60

60

40

40

20

20

0

0

20

20

40

40

60

60

80

Methods

MSE

ACC (%)

11 (1%)
11 (10%)
6 (10%)
2 (10%)

438.71±5.43
366.74±0.38
380.61±4.94
656.28±20.9

80.32±5.30
84.83±2.68
83.27±3.37
16.20±3.50

DiscoGAN
BiGAN

712.20±14.6
790.13±15.0

13.86±3.00
12.07±4.03

80
80

60

40

20

0

20

40

60

80

(a) DiscoGAN

80

60

40

20

0

20

40

60

80

(b) ALICE: coherent sup.

80
60
40
20
0
20
40
60
80
80

60

40

20

0

20

40

60

(c) ALICE: opposite sup.

80

(d) Learning curves

Inputs
ALICE
(coherent.sup. )
ALICE
(opposite.sup. )
DiscoGAN
BiGAN

Figure 11: Cross-domain relationship discovery with weakly supervised information using ALICE.
17

E.2

Edge-to-Shoe Dataset

The MSE results on cross-domain prediction and one-domain reconstruction are shown in Figure 12.
1600
1400
1200

ALICE ( rec + A-sup)
ALICE ( rec+ 2-sup)
ALICE (only 2-sup)
DiscoGAN ( 2-sup)

1200
1000
MSE

MSE

1400

ALICE ( rec + A-sup)
ALICE ( rec+ 2-sup)
ALICE (only 2-sup)
DiscoGAN ( 2-sup)

1000

800
600

800

400

600
0

20

40
60
80
100
Supervision (%)
(a) Cross-domain transformation on both sides

0

40
60
80
100
Supervision (%)
(b) Reconstruction on both sides

BiGAN

BiGAN

ALICE ( rec + A-sup)
ALICE ( rec+ 2-sup)
ALICE (only 2-sup)
DiscoGAN ( 2-sup)

MSE

MSE

(c) Cross-domain transformation on both sides

1400
1300
1200
1100
1000
900
800

(d) Reconstruction on on both sides
1400
ALICE ( rec + A-sup)
ALICE ( rec+ 2-sup)
1200
ALICE (only 2-sup)
DiscoGAN ( 2-sup)
1000

800
600

0

400

40
60
80
100
Supervision (%)
(e) Cross-domain transformation on edges
0.70

20

0

20

40
60
80
Supervision (%)
(f) Reconstruction on edges

100

0.70
ALICE ( rec + A-sup)
ALICE ( rec+ 2-sup)
ALICE (only 2-sup)
DiscoGAN ( 2-sup)

0.60
0.55

0.65
SSIM

0.65
SSIM

20

0.60
0.55

0.50

ALICE ( rec + A-sup)
ALICE ( rec+ 2-sup)
ALICE (only 2-sup)
DiscoGAN ( 2-sup)

0.50
40
60
80
100
0
20
40
60
80
100
Supervision (%)
Supervision (%)
(g) Cross-domain transformation on edges
(h) Reconstruction on edges
Figure 12: SSIM and MSE on Edge-to-Shoe dataset. Top 2 rows are results reported for both domains, and the
bottom 2 rows are results for edge domain only.
0

20

18

E.3

Celeba Face Dataset

Reconstruction results on the validation dataset of Celeba dataset are shown in Figure 13. ALI results
are from the paper [4]. ALICE provides more faithful reconstruction to the input subjects. As a
trade-off between theoretical optimum and practical convergence, we employ feature matching, and
thus our results exhibits slight bluriness characteristic.

(a) ALICE

(b) ALI

(c) Generated faces.
Figure 13: Reconstruction of (a) ALICE and (b) ALI. Odd columns are original samples from the validation set
and even columns are corresponding reconstructions. (c) Generated faces (even rows), based on the predicted
attributes of the real face image (odd row).

E.4

Real applications to Edges2Cartoon: ALICE for Alice in Wonderland

We demonstrate the potential real applications of ALICE algorithms on the task of sketch to cartoon.
We built a dataset by collecting frames from Disney’s film Alice in Wonderland. A large image
size 256 × 256 is considered. The training dataset consists of two domains: cartoon images and
edges images, where the edges are created via holistically-nested edge detection [28] on their true
cartoon images. The image content is about either of two characters in the film: Alice or White
Rabbit. Therefore, each domain exhibits two modes. 52 images are collected in each domain. The
one-to-one image correspondence between two domain is unknown, the goal is to efficiently generate
realistic cartoon images for animation, based on the edges.
CycleGAN is an unsupervised learning algorithm. Since we have shown its equivalence to
ALI/BiGAN (see Related Work), and its superiority in terms of stability. We derive our weaklysupervised ALICE algorithm on this dataset as: (i) CycleGAN for the unpaired data, and (ii) explicit
`1 loss and/or implicit conditional GAN loss for the paired samples. Note that only one pair is
randomly chosen for each character. The total training iteration is 10K. We observed the results while
training, and summarize in the following:
19

• ALICE converges faster than CycleGAN: . The generated images after 6K iterations are
show in Figure 14. CycleGAN generates images with mixed colors (e.g., , the clothes of
Rabbit), while ALICE clearly paint colors in different regions.
• ALICE enables desirable image generation (with better generalization): The generated images after 10K iterations are show in Figure 15. We generated image based on
slightly different edges: more background and details on the character. CycleGAN gets
confused when identifying the character, thus inconsistently paint the dfferent colors to
Rabbit, while explicit ALICE can generate coherent color.
• ALICE improves image quality via weak supervision: Two sets of the generated image
after 10K iterations are compared in Figure 16. ALICE can capture more visual details (e.g.,
the background flowers in (a)) and colors (e.g., the ambarella in (b)).

(a) Real Cartoon

(b) Real Edges (training set)

(c) Generated cartoon images via explicit ALICE

(d) Generated cartoon images via implicit ALICE

(e) Generated cartoon images via CycleGAN
Figure 14: Generaed cartoon images (conditioned on training edges in (b)) after 6K iterations of different
algorithms: (c) explicit ALICE, (d) implicit ALICE and (e) CycleGAN.

20

(a) Real Cartoon

(b) Real Edges (more detailed, and slightly different from the training set)

(c) Generated cartoon images via explicit ALICE

(d) Generated cartoon images via implicit ALICE

(e) Generated cartoon images via CycleGAN
Figure 15: Generaed cartoon images (conditioned on more detailed edges in (b)) after 10K iterations of different
algorithms: (c) explicit ALICE, (d) implicit ALICE and (e) CycleGAN.

21

(a) Alice

(b) Rabbit

Figure 16: Comparison of different algorithms after 10K iterations. The
bounding boxes highlight the different regions. From top to bottom: real
cartoon, edges, explicit ALICE, implicit ALICE, CycleGAN.
22

