TRANSPORT MAPS FOR DISTRIBUTION PRE SERVING OPERATIONS ON LATENT SPACES OF G ENER ATIVE M ODELS

arXiv:1711.01970v2 [cs.LG] 24 Jan 2018

O PTIMAL

Eirikur Agustsson
D-ITET, ETH Zurich
Switzerland
aeirikur@vision.ee.ethz.ch

Alexander Sage
D-ITET, ETH Zurich
Switzerland
sagea@student.ethz.ch

Radu Timofte
D-ITET, ETH Zurich
Merantix GmbH
radu.timofte@vision.ee.ethz.ch

Luc Van Gool
D-ITET, ETH Zurich
ESAT, KU Leuven
vangool@vision.ee.ethz.ch

A BSTRACT
Generative models such as Variational Auto Encoders (VAEs) and Generative Adversarial Networks (GANs) are typically trained for a fixed prior distribution in
the latent space, such as uniform or Gaussian. After a trained model is obtained,
one can sample the Generator in various forms for exploration and understanding,
such as interpolating between two samples, sampling in the vicinity of a sample
or exploring differences between a pair of samples applied to a third sample. In
this paper, we show that the latent space operations used in the literature so far
induce a distribution mismatch between the resulting outputs and the prior distribution the model was trained on. To address this, we propose to use distribution
matching transport maps to ensure that such latent space operations preserve the
prior distribution, while minimally modifying the original operation. Our experimental results validate that the proposed operations give higher quality samples
compared to the original operations.

1

I NTRODUCTION & R ELATED W ORK

Generative models such as Variational Autoencoders (VAEs) (Kingma & Welling, 2013) and Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) have emerged as popular techniques
for unsupervised learning of intractable distributions. In the framework of Generative Adversarial
Networks (GANs) (Goodfellow et al., 2014), the generative model is obtained by jointly training a
generator G and a discriminator D in an adversarial manner. The discriminator is trained to classify synthetic samples from real ones, whereas the generator is trained to map samples drawn from
a fixed prior distribution to synthetic examples which fool the discriminator. Variational Autoencoders (VAEs) (Kingma & Welling, 2013) are also trained for a fixed prior distribution, but this is
done through the loss of an Autoencoder that minimizes the variational lower bound of the data
likelihood. For both VAEs and GANs, using some data X we end up with a trained generator G,
that is supposed to map latent samples z from the fixed prior distribution to output samples G(z)
which (hopefully) have the same distribution as the data.
In order to understand and visualize the learned model G(z), it is a common practice in the literature
of generative models to explore how the output G(z) behaves under various arithmetic operations on
the latent samples z. In this paper, we show that the operations typically used so far, such as linear
interpolation (Goodfellow et al., 2014), spherical interpolation (White, 2016), vicinity sampling and
vector arithmetic (Radford et al., 2015), cause a distribution mismatch between the latent prior distribution and the results of the operations. This is problematic, since the generator G was trained on
a fixed prior and expects to see inputs with statistics consistent with that distribution. We show that
this, somewhat paradoxically, is also a problem if the support of resulting (mismatched) distribution
is within the support of a uniformly distributed prior, whose points all have equal likelihood during
training.
1

samples from prior
linear
matched (ours)
spherical

(a) Uniform prior: Trajectories of linear interpolation, our matched interpolation and the spherical interpolation (White, 2016).

(b) Uniform prior distribution.

(c) Linear midpoint distribution

(d) Matched midpoint distribution (ours)

(e) Spherical midpoint
distribution (White, 2016)

Figure 1: We show examples of distribution mismatches induced by the previous interpolation
schemes when using a uniform prior in two dimensions. Our matched interpolation avoids this with
a minimal modification to the linear trajectory, traversing through the space such that all points along
the path are distributed identically to the prior.

To address this, we propose to use distribution matching transport maps, to obtain analogous latent
space operations (e.g. interpolation, vicinity sampling) which preserve the prior distribution of
the latent space, while minimally changing the original operation. In Figure 1 we showcase how
our proposed technique gives an interpolation operator which avoids distribution mismatch when
interpolating between samples of a uniform distribution. The points of the (red) matched trajectories
are obtained as minimal deviations (in expectation of l1 distance) from the the points of the (blue)
linear trajectory.
1.1

G ENERATIVE M ODELS AND S AMPLE O PERATIONS

In the literature there are dozens of papers that use sample operations to explore the learned models.
Bengio et al. (2013) use linear interpolation between neighbors in the latent space to study how
well deep vs shallow representations can disentangle the latent space of Contractive Auto Encoders
(CAEs) (Rifai et al., 2011).
In the seminal GAN paper of Goodfellow et al. (2014), the authors use linear interpolation between
latent samples to visualize the transition between outputs of a GAN trained on MNIST. Dosovitskiy
et al. (2015) linearly interpolate the latent codes of an auto encoder trained on a synthetic chair
dataset.
Radford et al. (2015) also linearly interpolate between samples to evaluate the quality of the learned
representation. Furthermore, motivated by the semantic word vectors of Mikolov et al. (2013), they
explore using vector arithmetic on the samples to change semantics such as adding a smile to a
generated face.
Reed et al. (2016) use linear interpolation to explore their proposed GAN model which operates
jointly in the visual and textual domain. Brock et al. (2016) combine GANs and VAEs for a neural
photo editor, using masked interpolations to edit an embedded photo in the latent space.
1.2

D ISTRIBUTION M ISMATCH AND R ELATED A PPROACHES

While there are numerous works performing operations on samples, most of them have ignored the
problem of distribution mismatch, such as the one presented in Figure 1d. Kingma & Welling (2013)
and Makhzani et al. (2015) sidestep the problem when visualizing their models, by not performing
operations on latent samples, but instead restrict the latent space to 2-d and uniformly sample the
percentiles of the distribution on a 2-d grid. This way, the samples have statistics that are consistent
with the prior distribution. However, this approach does not scale up to higher dimensions - whereas
the latent spaces used in the literature can have hundreds of dimensions.
2

Operation

Expression

2-point interpolation
n-point interpolation
Vicinity sampling
Analogies

y = tz1 + (1 − t)z2 , t ∈ [0, 1]
Pn
P
y = i=1 ti zi with i ti = 1
yj = z1 + uj for j = 1, · · · , k
y = z3 + (z2 − z1 )

(Gaussian) Matched Operation
p
ỹ = y/pt2 + (1 − t)2
Pn
ỹ = y/ √ i=1 t2i
y˜j = yj√
/ 1 + 2
ỹ = y/ 3

Table 1: Examples of interesting sample operations which need to be adapted if we want the distribution of the result y to match the prior distribution. If the prior is Gaussian, our proposed matched
operation simplifies to a proper re-scaling factor (see third column) for additive operations.
Related to our work, White (2016) experimentally observe that there is a distribution mismatch
between the distance to origin for points drawn from uniform or Gaussian distribution and points
obtained with linear interpolation, and propose to use a so-called spherical linear interpolation to reduce the mismatch, obtaining higher quality interpolated samples. However, the proposed approach
has no theoretical guarantees.
In this work, we propose a generic method to fully preserve the desired prior distribution when
using sample operations. The approach works as follows: we are given a ‘desired’ operation, such
as linear interpolation y = tz1 + (1 − t)z2 , t ∈ [0, 1]. Since the distribution of y does not match
the prior distribution of z, we search for a warping f : Rd → Rd , such that ỹ = f (y) has the
same distribution as z. In order to have the modification ỹ as faithful as possible to the original
operation y, we use optimal transform maps (Santambrogio, 2015; Villani, 2003; 2008) to find a
minimal modification of y which recovers the prior distribution z.
This is illustrated in Figure 1a, where each point ỹ of the matched curve is obtained by warping a
corresponding point y of the linear trajectory, while not deviating too far from the line.

2

F ROM DISTRIBUTION MISMATCH TO OPTIMAL TRANSPORT

With implicit models such as GANs (Goodfellow et al., 2014) and VAEs (Kingma & Welling, 2013),
0
we use the data X , drawn from an unknown random variable x, to learn a generator G : Rd 7→ Rd
with respect to a fixed prior distribution pz , such that G(z) approximates x. Once the model is
trained, we can sample from it by feeding latent samples z through G.
We now bring our attention to operations on latent samples z1 , · · · , zk from pz , i.e. mappings
κ : Rd × · · · × R d → R d .
We give a few examples of such operations in Table 1.

(1)

Since the inputs to the operations are random variables, their output y = κ(z1 , · · · , zk ) is also a
random variable (commonly referred to as a statistic). While we typically perform these operations
on realized (i.e. observed) samples, our analysis is done through the underlying random variable y.
The same treatment is typically used to analyze other statistics over random variables, such as the
sample mean, sample variance and test statistics.
In Table 1 we show example operations which have been commonly used in the literature. As
discussed in the Introduction, such operations can provide valuable insight into how the trained
generator G changes as one creates related samples y from some source samples. The most common
such operation is the linear interpolation, which we can view as an operation
yt = tz1 + (1 − t)z2 ,
(2)
where z1 , z2 are latent samples from the prior pz and yt is parameterized by t ∈ [0, 1].
Now, assume z1 and z2 are i.i.d, and let Z1 , Z2 be their (scalar) first components with distribution
pZ . Then the first component of yt is Yt = tZ1 + (1 − t)Z2 , and we can compute:
Var[Yt ] = Var[tZ1 + (1 − t)Z2 ] = t2 Var[Z1 ] + (1 − t)2 Var[Z2 ] = (1 + 2t(t − 1))Var[Z]. (3)
Since (1 + 2t(t − 1)) 6= 1 for all t ∈ [0, 1] \ {0, 1}, it is in general impossible for yt to have
the same distribution as z, which means that distribution mismatch is inevitable when using linear
interpolation. A similar analysis reveals the same for all of the operations in Table 1.
This leaves us with a dilemma: we have various intuitive operations (see Table 1) which we would
want to be able to perform on samples, but their resulting distribution pyt is inconsistent with the
distribution pz we trained G for.
3

·10−2
0.25

6
prior
midpoint linear
midpoint matched (ours)
midpoint spherical

0.2

prior
midpoint linear
midpoint matched (ours)
midpoint spherical

5

4

pkyk2

pkyk2

0.15
3

0.1
2
5 · 10−2

0

1

5

10

15

20

25

30

35

40

45

50

0
20

40

60

kyk2

80

100

120

140

160

180

200

kyk2

(a) Uniform distribution

(b) Gaussian distribution

Figure 2: Distribution of the squared norm kyk2 of midpoints for two prior distributions in 100
dimensions: (a) components uniform on [−1, 1] and (b) components Gaussian N (0, 1), for linear
interpolation, our proposed matched interpolation and the spherical interpolation proposed by White
(2016). Both linear and spherical interpolation introduce a distribution mismatch, whereas our proposed matched interpolation preserves the prior distribution for both priors.
Due to the curse of dimensionality, as empirically observed by White (2016), this mismatch can
be significant in high dimensions. We illustrate this in Figure 2, where we plot the distribution of
the squared norm kyt k2 for the midpoint t = 1/2 of linear interpolation, compared to the prior
distribution kzk2 . With d = 100 (a typical dimensionality for the latent space), the distributions are
dramatically different, having almost no common support. In Appendix 5.1 we expand this analysis
and show that this happens for all prior distributions with i.i.d. entries (i.e. not only Uniform and
Gaussian).
2.1

D ISTRIBUTION MATCHING WITH OPTIMAL TRANSPORT

In order to address the distribution mismatch, we propose a simple and intuitive strategy for constructing distribution preserving operators, via optimal transport:
Strategy 1 (Optimal Transport Matched Operations).
1. We construct an ’intuitive’ operator y = κ(z1 , · · · , zk ).
2. We analytically (or numerically) compute the resulting (mismatched) distribution py
3. We search for a minimal modification ỹ = f (y) (in the sense that Ey [c(ỹ, y)] is minimal
with respect to a cost c), such that distribution is brought back to the prior, i.e. pỹ = pz .
The cost function in step 3 could e.g. be the euclidean distance c(x, y) = kx − yk, and is used
to measure how faithful the modified operator, ỹ = f (κ(z1 , · · · , zk )) is to the original operator
k. Finding the map f which gives a minimal modification can be challenging, but fortunately it is
a well studied problem from optimal transport theory. We refer to the modified operation ỹ as the
matched version of y, with respect to the cost c and prior distribution pz .
For completeness, we introduce the key concept of optimal transport theory in a simplified setting, i.e. assuming probability distributions are in euclidean space and skipping measure theoretical
formalism. We refer to Villani (2003; 2008) and Santambrogio (2015) for a thorough and formal
treatment of optimal transport.
The problem of step (3) above was first posed by Monge (1781) and can more formally be stated as:
Problem 1 (Santambrogio (2015) Problem 1.1). Given probability distributions px , py , with domains X , Y respectively, and a cost function c : X × Y → R+ , we want to minimize
n
o
inf Ex∼px [c(x, f (x))] f : X → Y, f (x) ∼ py
(MP)
We refer to the minimizer f ∗ X → Y of (MP) (if it exists), as the optimal transport map from px to
py with respect to the cost c.
4

However, the problem remained unsolved until a relaxed problem was studied by Kantorovich
(1942):
Problem 2 (Santambrogio (2015) Problem 1.2). Given probability distributions px , py , with domains X , Y respectively, and a cost function c : X × Y → R+ , we want to minimize
o
n
(KP)
inf E(x,y)∼px,y [c(x, y)] (x, y) ∼ px,y , x ∼ px , y ∼ py ,
where (x, y) ∼ px,y , x ∼ px , y ∼ py denotes that (x, y) have a joint distribution px,y which has
(previously specified) marginals px and py .
We refer to the joint px,y which minimizes (KP) as the optimal transport plan from px to py with
respect to the cost c.
The key difference is to relax the deterministic relationship between x and f (x) to a joint probability distribution px,y with marginals px and py for x and y. In the case of Problem 1, the
minimization might be over the empty set since it is not guaranteed that there exists a mapping
f such that f (x) ∼ y. In contrast, for Problem 2, one can always construct a joint density px,y
with px and py as marginals, such as the trivial construction where x and y are independent, i.e.
px,y (x, y) := px (x)py (y).
Note that given a joint density px,y (x, y) over X × Y, we can view y conditioned on x = x for
a fixed x as a stochastic function f (x) from X to Y, since given a fixed x do not get a specific
function value f (x) but instead a random variable f (x) that depends on x, with f (x) ∼ y|x = x
p
(x,y)
with density py (y|x = x) := x,y
px (x) . In this case we have (x, f (x)) ∼ px,y , so we can view the
Problem KP as a relaxation of Problem MP where f is allowed to be a stochastic mapping.
While the relaxed problem of Kantorovich (KP) is much more studied in the optimal transport literature, for our purposes of constructing operators it is desirable for the mapping f to be deterministic
as in (MP).
To this end, we will choose the cost function c such that the two problems coincide and where we
can find an analytical solution f or at least an efficient numerical solution.
In particular, we note that most operators in Table 1 are all pointwise, such that if the points zi have
i.i.d. components, then the result y will also have i.i.d. components.
If we combine this with the constraint for the cost c to be additive over the components of x, y, we
obtain the following simplification:
Theorem 1. Suppose px and py have i.i.d components and c over X × Y = Rd × Rd decomposes
as
d
X
c(x, y) =
C(x(i) , y (i) ).
(4)
i=1

Consequently, the minimization problems (MP) and (KP) turn into d identical scalar problems for
the distributions pX and pY of the components of x and y:
n
o
inf EX∼pX [C(X, T (X))] T : R → R, T (X) ∼ pY
(MP-1-D)
n
o
inf E(X,Y )∼pX,Y [C(X, Y )] (X, Y ) ∼ pX,Y , X ∼ pX , Y ∼ pY ,
(KP-1-D)
such that an optimal transport map T for (MP-1-D) gives an optimal transport map f for (MP)
by pointwise application of T , i.e. f (x)(i) := T (x(i) ), and an optimal transport plan pX,Y for
Qd
(KP-1-D) gives an optimal transport plan px,y (x, y) := i=1 pX,Y (x(i) , y (i) ) for (KP).
Proof. See Appendix.
Fortunately, under some mild constraints, the scalar problems have a known solution:
Theorem 2 (Theorem 2.9 in Santambrogio (2015)). Let h : R → R+ be convex and suppose the
cost C takes the form C(x, y) = h(x − y). Given an continuous source distribution pX and a target
distribution pY on R having a finite optimal transport cost in (KP-1-D), then
[−1]

mon
TX→Y
(x) := FY

5

(FX (x)),

(5)

3

1
0.8

t = 0.05
t = 0.25
t = 0.5

2

t = 0.05
t = 0.25
t = 0.5

0.6
0.4

1

ỹ

ỹ

0.2
0

0

−0.2
−1
−0.4
−0.6

−2

−0.8
−1
−1

−0.8

−0.6

−0.4

−0.2

0
y

0.2

0.4

0.6

0.8

−3
−3

1

−2.5

−2

(a) Uniform prior

−1.5

−1

−0.5

0
y

0.5

1

1.5

2

2.5

3

(b) Gaussian prior

Figure 3: We show the monotone transport maps for linear interpolation evaluated at t ∈
{0.05, 0.25, 0.5}, to Uniform and Gaussian priors.
defines an optimal transport map from pX to pY for (MP-1-D), where FX (x) :=

Rx
−∞

pX (x0 )dx0

[−1]

is the Cumulative Distribution Function (CDF) of X and FY (y) := inf{t ∈ R|FY (t) ≥ y} is
mon
the pseudo-inverse of FY . Furthermore, the joint distribution of (X, TX→Y
(X)) defines an optimal
transport plan for (KP-1-D).
mon
The mapping TX→Y
(x) in Theorem 2 is non-decreasing and is known as the monotone transport
mon
map from X to Y . It is easy to verify that TX→Y
(X) has the distribution of Y , in particular
[−1]
FX (X) ∼ Uniform(0, 1) and if U ∼ Uniform(0, 1) then FY (U ) ∼ Y .

Now, combining Theorems 1 and 2, we obtain a concrete realization of the Strategy 1 outlined above.
We choose the cost c such that it admits to Theorem 1, such as c(x, y) := kx − yk1 , and use an
operation that is pointwise, so we just need to compute the monotone transport map in (5). That is,
if z has i.i.d components with distribution pZ , we just need to compute the component distribution
pY of the result y of the operation, the CDFs FZ , FY and obtain
[−1]

TYmon
→Z (y) := FZ

(FY (y))

(6)

(i)
as the component-wise modification of y, i.e. ỹ (i) := TYmon
→Z (y ).

In Figure 3 we show the monotone transport map for the linear interpolation y = tz1 + (1 − t)z2
for various values of t. The detailed calculations and examples for various operations are given in
Appendix 5.3, for both Uniform and Gaussian priors. The Gaussian case has a particularly simple
resulting transport map for additive operations, where it is just a linear transformation through a
scalar multiplication, summarized in the third column of Table 1.

3
3.1

E XPERIMENTS
C OMPARISON OF DISTRIBUTIONS

To validate the correctness of the matched operators obtained above, we numerically simulate the
distributions for toy examples, as well as prior distributions typically used in the literature.
Priors vs. interpolations in 2-D For Figure 1, we sample 1 million pairs of points in two dimension, from a uniform prior (on [−1, 1]2 ), and estimate numerically the midpoint distribution of
linear interpolation, our proposed matched interpolation and the spherical interpolation of White
(2016). It is reassuring to see that the matched interpolation gives midpoints which are identically
distributed to the prior. In contrast, the linear interpolation condenses more towards the origin, forming a pyramid-shaped distribution (the result of convolving two boxes in 2-d). Since the spherical
interpolation of White (2016) follows a great circle with varying radius between the two points, we
see that the resulting distribution has a “hole” in it, “circling” around the origin for both priors.
Priors vs. interpolations in 100-D For Figure 2, we sample 1 million pairs of points in d = 100
dimensions, using either i.i.d. uniform components on [−1, 1] or Gaussian N (0, 1) and compute the
6

(a) LLD icon dataset

(b) LSUN dataset

(c) CelebA dataset

Figure 4: 2-point interpolation: Each example shows linear, SLERP and transport matched interpolation from top to bottom respectively. For LLD icon dataset (a) and LSUN (b), outputs are
produced with DCGAN using a uniform prior distribution, whereas the CelebA model (c) uses a
Gaussian prior. The output resolution for the (a) is 32 × 32, for (b) and (c) 64 × 64 pixels.

(a) Linear

(b) Spherical

(c) Distribution matched

Figure 5: Midpoint sampling for linear, SLERP and uniform-matched interpolation when using the
same pairs of sample points on LLD icon dataset with uniform prior.

distribution of the squared norm of the midpoints. We see there is a dramatic difference between
vector lengths in the prior and the midpoints of linear interpolation, with only minimal overlap. We
also show the spherical (SLERP) interpolation of White (2016) which has a matching first moment,
but otherwise also induces a distribution mismatch. In contrast, our matched interpolation, fully
preserves the prior distribution and perfectly aligns. We note that this setting (d = 100, uniform or
Gaussian) is commonly used in the literature.
3.2

Q UALITATIVE RESULTS

In this section we will present some concrete examples for the differences in generator output dependent on the exact sample operation used to traverse the latent space of a generative model. To
this end, the generator output for latent samples produced with linear interpolation, SLERP (spherical linear interpolation) of White (2016) and our proposed matched interpolation will be compared.
Please refer to Table 1 for an overview of the operators used in this Section.
Setup We used DCGAN (Radford et al., 2015) generative models trained on LSUN bedrooms (Yu
et al., 2015), CelebA (Liu et al., 2015) and LLD (Sage et al., 2017), an icon dataset, to qualitatively
evaluate. For LSUN, the model was trained for two different output resolutions, providing 64 × 64
pixel and a 128×128 pixel output images (where the latter is used in figures containing larger sample
images). The models for LSUN and the icon dataset where both trained on a uniform latent prior
distribution, while for CelebA a Gaussian prior was used. The dimensionality of the latent space is
100 for both LSUN and CelebA, and 512 for the model trained on the icon model. Furthermore we
use improved Wasserstein GAN (iWGAN) with gradient penalty (Gulrajani et al., 2017) trained on
CIFAR-10 at 32 × 32 pixels with a 128-dimensional Gaussian prior to produce the inception scores
presented in Section 3.3.
7

(a) Linear

(b) Spherical

(c) Distribution matched

Figure 6: Midpoint sampling for linear, SLERP and uniform-matched interpolation when using the
same pairs of sample points on LSUN (64 × 64) with uniform prior.

(a) Linear interpolation

(b) Spherical interpolation

(c) Distribution matched

Figure 7: 4-point interpolation between 4 sampled points (corners) from DCGAN trained on LSUN
(128 × 128) using a uniform prior. The same interpolation is shown using linear, SLERP and
distribution matched interpolation.

2-point interpolation We begin with the classic example of 2-point interpolation: Figure 4 shows
three examples per dataset for an interpolation between 2 points in latent space. Each example is
first done via linear interpolation, then SLERP and finally matched interpolation. In Figure 15 in the
Appendix we show more densely sampled examples.
It is immediately obvious in Figures 4a and 4b that linear interpolation produces inferior results with
generally more blurry, less saturated and less detailed output images. SLERP and matched interpolation are slightly different, however it is not visually obvious which one is superior. Differences
between the various interpolation methods for CelebA (Figure 4c) are much more subtle to the point
that they are virtually indistinguishable when viewed side-by-side. This is not an inconsistency

(a) Linear interpolation

(b) Spherical

(c) Distribution matched

Figure 8: 4-point interpolation between 4 sampled points (corners) from DCGAN trained on icon
dataset using a uniform prior. The same interpolation is shown using linear, SLERP and distribution
matched interpolation.
8

(a) Vicinity sampling

(b) Matched vicinity sampling

Figure 9: Vicinity sampling on LSUN dataset (128 × 128) with uniform prior. The sample in the
middle is perturbed in random directions producing the surrounding sample points.

(a) Vicinity sampling

(b) Matched vicinity sampling

Figure 10: Vicinity sampling on LLD icon dataset with uniform prior. The sample in the middle is
perturbed in random directions producing the surrounding sample points.

though: while distribution mismatch can cause large differences, it can also happen that the model
generalizes well enough that it does not matter.
Midpoint interpolation In all cases, the point where the interpolation methods diverge the most, is
at the midpoint of the interpolation where t = 0.5. Thus we provide 25 such interpolation midpoints
in Figures 5 (LLD icons) and 6 (LSUN) for direct comparison.
4-point interpolation An even stronger effect can be observed when we do 4-point interpolation,
showcased in Figure 7 (LSUN) and Figure 8 (LLD icons). The higher resolution of the LSUN output

Figure 11: Random walk for LLD, LSUN (64 x 64) and CelebA. The random walks consist of a
succession of steps in random directions, calculated for the same sequence of directions using (nonmatched) vicinity sampling in the upper rows and our proposed matched vicinity sampling in the
lower rows.
9

Dataset
Model
Prior
Inception score

CIFAR-10
iWGAN
Gaussian, 128-D
7.90 ± 0.11

LLD-icon
DCGAN
Uniform, 100-D
3.70 ± 0.09

LSUN
DCGAN
Uniform, 100-D
3.90 ± 0.08

CelebA
DCGAN
Gaussian, 100-D
2.05 ± 0.04

3.57 ± 0.07 (-8%)
3.90 ± 0.11
3.89 ± 0.08
2.95 ± 0.06 (-24%)
3.89 ± 0.10
3.91 ± 0.10

1.71 ± 0.02 (-17%)
2.04 ± 0.04
2.04 ± 0.03
1.46 ± 0.01 (-29%)
2.04 ± 0.04
2.04 ± 0.04

Inception Score on midpoints of interpolation operations:
2-point linear
2-point SLERP
2-point matched
4-point linear
4-point SLERP
4-point matched

7.12 ± 0.08 (-10%)
7.89 ± 0.09
7.89 ± 0.08
5.84 ± 0.08 (-26%)
7.87 ± 0.09
7.91 ± 0.09

3.56 ± 0.06 (-4%)
3.68 ± 0.09
3.69 ± 0.08
3.45 ± 0.08 (-7%)
3.72 ± 0.09
3.69 ± 0.10

Table 2: Inception scores on LLD-icon, LSUN, CIFAR-10 and CelebA for the midpoints of various
interpolation operations. Scores are reported as mean ± standard deviation (relative change in %).
highlights the very apparent loss of detail and increasing prevalence of artifacts towards the midpoint
in the linear version, compared to SLERP compared and our matched interpolation.
Vicinity sampling Furthermore we provide two examples for vicinity sampling in Figures 9 and
10. Analogous to the previous observations, the output under a linear operator lacks definition,
sharpness and saturation when compared to both spherical and matched operators.
Random walk An interesting property of our matched vicinity sampling is that we can obtain a
random walk in the latent space by applying it repeatedly: we start at a point y0 = z drawn from the
prior, and then obtain point yi by sampling a single point in the vicinity of yi−1 , using some fixed
’step size’ .
We show an example of such a walk in Figure 11, using  = 0.5. As a result of the repeated application of the vicinity sampling operation, the divergence from the prior distribution in the non-matched
case becomes stronger with each step, resulting in completely unrecognizable output images on the
LSUN and LLD icon models. Even for the CelebA model where differences where minimal before, they are quite apparent in this experiment. The random walk thus perfectly illustrates the need
for respecting the prior distribution when performing any operation in latent space, as the adverse
effects can cumulate through the repeated application of operators that do not comply to the prior
distribution.
3.3

Q UANTITATIVE RESULTS

We quantitatively confirm the observations of the previous section by using the Inception
score(Salimans et al., 2016). In Table 2 we compare the Inception score of our trained models (i.e.
using random samples from the prior) with the score when sampling midpoints from the 2-point and
4-point interpolations described above, reporting mean and standard deviation with 50,000 samples,
as well as relative change to the original model scores if they are significant. Compared to the
original scores of the trained models, our matched operations are statistically indistinguishable (as
expected) while the linear interpolation gives a significantly lower score in all settings (up to 29%
lower). As observed for the quality visually, the SLERP heuristic gives similar scores to the matched
operations.

4

C ONCLUSIONS

We have shown that the common latent space operations used for Generative Models induce distribution mismatch from the prior distribution the models were trained for. This problem has been
mostly ignored by the literature so far, partially due to the belief that this should not be a problem for
uniform priors. However, our statistical and experimental analysis shows that the problem is real,
with the operations used so far producing significantly lower quality samples compared to their inputs. To address the distribution mismatch, we propose to use optimal transport to minimally modify
(in l1 distance) the operations such that they fully preserve the prior distribution. We give analytical
formulas of the resulting (matched) operations for various examples, which are easily implemented.
The matched operators give a significantly higher quality samples compared to the originals, having
the potential to become standard tools for evaluating and exploring generative models.
10

R EFERENCES
Yoshua Bengio, Grégoire Mesnil, Yann Dauphin, and Salah Rifai. Better mixing via deep representations. In Proceedings of the 30th International Conference on Machine Learning (ICML-13),
pp. 552–560, 2013.
Andrew Brock, Theodore Lim, James M Ritchie, and Nick Weston. Neural photo editing with
introspective adversarial networks. arXiv preprint arXiv:1609.07093, 2016.
Alexey Dosovitskiy, Jost Tobias Springenberg, and Thomas Brox. Learning to generate chairs with
convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition, pp. 1538–1546, 2015.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair,
Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural information processing systems, pp. 2672–2680, 2014.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. Improved training of wasserstein gans. arXiv:1704.00028v2, 2017.
Leonid Vitalievich Kantorovich. On the translocation of masses. In Dokl. Akad. Nauk SSSR, volume 37, pp. 199–201, 1942.
Diederik P Kingma and Max Welling.
arXiv:1312.6114, 2013.

Auto-encoding variational bayes.

arXiv preprint

Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild.
In Proceedings of International Conference on Computer Vision (ICCV), 2015.
David JC MacKay. Information theory, inference and learning algorithms. Cambridge university
press, 2003.
Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, and Brendan Frey. Adversarial
autoencoders. arXiv preprint arXiv:1511.05644, 2015.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean.
Distributed representations of words and phrases and their compositionality.
In
C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger
(eds.), Advances in Neural Information Processing Systems 26, pp. 3111–3119.
Curran Associates, Inc., 2013.
URL http://papers.nips.cc/paper/
5021-distributed-representations-of-words-and-phrases-and-their-compositionality.
pdf.
Gaspard Monge. Mémoire sur la théorie des déblais et des remblais. Histoire de l’Académie Royale
des Sciences de Paris, 1781.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep
convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434, 2015.
Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee.
Generative adversarial text to image synthesis. arXiv preprint arXiv:1605.05396, 2016.
Salah Rifai, Pascal Vincent, Xavier Muller, Xavier Glorot, and Yoshua Bengio. Contractive autoencoders: Explicit invariance during feature extraction. In Proceedings of the 28th international
conference on machine learning (ICML-11), pp. 833–840, 2011.
Alexander Sage, Eirikur Agustsson, Radu Timofte, and Luc Van Gool. Lld: Large logo dataset.
2017. URL https://data.vision.ee.ethz.ch/cvl/lld/.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen.
Improved techniques for training gans. In Advances in Neural Information Processing Systems,
pp. 2234–2242, 2016.
Filippo Santambrogio. Optimal transport for applied mathematicians. Birkäuser, NY, 2015.
11

Cédric Villani. Topics in optimal transportation. Number 58. American Mathematical Soc., 2003.
Cédric Villani. Optimal transport: old and new, volume 338. Springer Science & Business Media,
2008.
Tom White. Sampling generative networks. arXiv preprint arXiv:1609.04468, 2016.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun:
Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv
preprint arXiv:1506.03365, 2015.

12

5
5.1

A PPENDIX
O N THE CURSE OF DIMENSIONALITY AND GEOMETRIC OUTLIERS

We note that the analysis here can bee seen as a more rigorous version of an observation made by
White (2016), who experimentally show that there is a significant difference between the average
norm of the midpoint of linear interpolation and the points of the prior, for uniform and Gaussian
distributions.
Suppose our latent space has a prior with z = [Z1 , · · · , Zd ] ∈ [−1, 1]d with i.i.d entries Zi ∼ Z. In
this case, we can look at the squared norm

kzk2 =

d
X

Zi2 .

(7)

i=1

From the Central Limit Theorem (CLT), we know that as d → ∞,
√

1
2
d( kzk2 − µZ 2 ) → N (0, σZ
2 ),
d

(8)

in distribution. Thus, assuming d is large enough such that we are close to convergence, we can
2
approximate the distribution of kzk2 as N (dµZ 2 , dσZ
2 ). In particular, this implies that almost all
points lie on a relatively thin
spherical
shell,
since
the
mean grows as O(d) whereas the standard
√
deviation grows only as O( d).
We note that this property is well known for i.i.d Gaussian entries (see e.g. Ex. 6.14 in MacKay
(2003)). For Uniform distribution on the hypercube it is also well known that the mass is concentrated in the corner points (which is consistent with the claim here since the corner points lie on a
sphere).
Now consider an operator such as the midpoint of linear interpolation, y = 12 z1 + 12 z2 , with compo(i)
(i)
nents Y (i) = 12 Z1 + 12 Z2 . Furthermore, let’s assume the component distribution pZ is symmetric
around 0, such that E[Z] = 0.
In this case, we can compute:
1 (i) 1 (i)
1
1
E[(Y (i) )2 ] = Var[ Z1 + Z2 ] = Var[Z] = µ2Z 2
2
2
2
2
1 (i) 1 (i) 2
1
1 2
(i) 2
2
Var[(Y ) ] = Var[( Z1 + Z2 ) ] = Var[Z ] = σZ 2 .
2
2
4
4

(9)
(10)

2
Thus, the distribution of kyk2 can be approximated with N ( 12 dµZ 2 , 41 dσZ
2 ).

Therefore, y also mostly lies on a spherical shell, but with a different radius than z. In fact, the
shells will intersect at regions which have a vanishing probability for large d. In other words, when
looking at the squared norm kyk2 , kyk2 is a (strong) outlier with respect to the distribution of kzk2 .

5.2

P ROOF OF T HEOREM 1

Proof. We will show it for the Kantorovich problem, the Monge version is similar.
13

Starting from (KP), we compute
o
n
inf E(x,y)∼px,y [c(x, y)] (x, y) ∼ px,y , x ∼ px , y ∼ py
(
)
d
X
(i)
(i)
= inf E(x,y)∼px,y [
C(x , y )] (x, y) ∼ px,y , x ∼ px , y ∼ py

(11)
(12)

i=1

= inf

( d
X

)
(i)

(i)

E(x,y)∼px,y [C(x , y )] (x, y) ∼ px,y , x ∼ px , y ∼ py

(13)

i=1
d
X

n
o
inf E(x,y)∼px,y [C(x(i) , y (i) )] (x, y) ∼ px,y , x ∼ px , y ∼ py

(14)

o
n
inf E(X,Y )∼pX,Y [C(X, Y )] (X, Y ) ∼ pX,Y , X ∼ pX , Y ∼ pY

(15)

o
n
= d · inf E(X,Y )∼pX,Y [C(X, Y )] (X, Y ) ∼ pX,Y , X ∼ pX , Y ∼ pY ,

(16)

≥

i=1

=

d
X
i=1

(17)
where the inequality in (14) is due to each term being minimized separately.
Qd
Now let Pd (X, Y ) be the set of joints px,y with px,y (x, y) = i=1 pX,Y (x(i) , y (i) ) where pX,Y
has marginals pX and pY . In this case Pd (X, Y ) is a subset of all joints px,y with marginals px and
py , where the pairs (x(1) , y (1) ), . . . , (x(d) , y (d) )) are constrained to be i.i.d. Starting again from
(13) can compute:
)
( d
X
(i)
(i)
inf
E(x,y)∼px,y [C(x , y )] (x, y) ∼ px,y , x ∼ px , y ∼ py
i=1

≤ inf

( d
X

)
(i)

(i)

E(x,y)∼px,y [C(x , y )] px,y ∈ Pd (X, Y )

(18)

i=1

= inf

( d
X

)
E(x,y)∼px,y [C(x(i) , y (i) )] px,y ∈ Pd (X, Y )

(19)

i=1

= inf

( d
X

)
E(X,Y )∼pX,Y [C(X, Y )] (X, Y ) ∼ pX,Y , X ∼ pX , Y ∼ pY

(20)

i=1

n
o
= d · inf E(X,Y )∼pX,Y [C(X, Y )] (X, Y ) ∼ pX,Y , X ∼ pX , Y ∼ pY ,

(21)
(22)

where the inequality in (18) is due to minimizing over a smaller set.
Since the two inequalities above are in the opposite direction, equality must hold for all of the
expressions above, in particular:
n
o
inf E(x,y)∼px,y [c(x, y)] (x, y) ∼ px,y , x ∼ px , y ∼ py
(23)
n
o
= d · inf E(X,Y )∼pX,Y [C(X, Y )] (X, Y ) ∼ pX,Y , X ∼ pX , Y ∼ pY
(24)
Thus, (KP) and (KP-1-D) equal up to a constant, and minimizing one will minimize the
other. Therefore the minimization of the former can be done over pX,Y with px,y (x, y) =
Qd
(i) (i)
i=1 pX,Y (x , y ).
5.3

C ALCULATIONS FOR E XAMPLES

In the next sections, we illustrate how to compute the matched operations for a few examples, in
particular for linear interpolation and vicinity sampling, using a uniform or a Gaussian prior. We
14

picked the examples where we can analytically compute the uniform transport map, but note that it
[−1]
is also easy to compute FZ and (FY (y)) numerically, since one only needs to estimate CDFs in
one dimension.
Since the components of all random variables in these examples are i.i.d, for such a random vector
x we will implicitly write X for a scalar random variable that has the distribution of the components
of x.
mon
When computing the monotone transport map TX→Y
, the following Lemma is helpful.
Lemma 1 (Theorem 2.5 in Santambrogio (2015)). Suppose a mapping g(x) is non-decreasing and
maps a continuous distribution pX to a distribution pY , i.e.

g(X) ∼ Y,
then g is the monotone transport map

mon
TX→Y

(25)

.

mon
According to Lemma 1, an alternative way of computing TX→Y
is to find some g that is nondecreasing and transforms pX to pY .

E XAMPLE 1:U NIFORM L INEAR I NTERPOLATION
Suppose z has uniform components Z ∼ Uniform(−1, 1). In this case, pZ (z) = 1/2 for −1 < z <
1.
Now let yt = tz1 + (1 − t)z2 denote the linear interpolation between two points z1 , z2 , with
component distribution pYt . Due to symmetry we can assume that t > 1/2, since pYt = pY1−t . We
then obtain pYt as the convolution of ptZ and p(1−t)Z , i.e. pYt = ptZ ∗ p(1−t)Z . First we note that
ptZ = 1/(2t) for −t < z < t and p(1−t)Z = 1/(2(1 − t)) for −(1 − t) < z < 1 − t. We can then
compute:
pYt (y) = (ptZ ∗ p(1−t)Z )(y)


0




y+1

1
=
2 − 2t
2(1 − t)(2t) 


−y + 1



0

(26)
if y < −1
if − 1 < y < −t + (1 − t)
if − t + (1 − t) < y < t − (1 − t)
if t − (1 − t) < y < 1
if 1 < y

(27)

(28)
The CDF FYt is then obtained by computing
Z y
FYt (y) =
pYt (y 0 )dy 0
−∞


0



1

 2 (y + 1)(y + 1)
1
=
2(1 − t)(y + t)
2(1 − t)(2t) 

2(1 − t)(3t − 1) + (− 21 y 2 + y + 12 (2t − 1)2 − (2t − 1))



2(1 − t)(2t)

Since pZ (z) = 1/2 for |z| < 1, we have FZ (z) =
2(p − 21 ).

1
2z

+

1
2

(29)
if y < −1
if − 1 < y < 1 − 2t
if 1 − 2t < y < 2t − 1
if 2t − 1 < y < 1
if 1 < y
(30)
[−1]

for |z| < 1. This gives FZ
[−1]

Now, we just compose the two mappings to obtain TYmon
(y) = FZ
t →Z

(p) =

(FYt (y)).

E XAMPLE 2: U NIFORM V ICINITY S AMPLING AND R ANDOM WALK
Let z again have uniform components on [−1, 1]. For vicinity sampling, we want to obtain new
points z10 , ·, zk0 which are close to z. We thus define
zi0 := z + ui ,
15

(31)

where ui also has uniform components, such that each coordinate of zi0 differs at most by  from z.
By identifying tZi0 = tZ + (1 − t)Ui with t = 1/(1 + ), we see that tZi0 has identical distribution
to the linear interpolation Yt in the previous example. Thus gt (Zi0 ) := TYmon
(tZi0 ) will have the
t →Z
0
distribution of Z, and by Lemma1 is then the monotone transport map from Zi to Z.
E XAMPLE 3: G AUSSIAN L INEAR I NTERPOLATION , V ICINITY S AMPLING AND A NALOGIES
Suppose z has components Z ∼ N (0, σ 2 ). In this case, we can compute linear interpolation as
before, yt = tz1 + (1 − t)z2 . Since the sum of Gaussians is Gaussian, we get, Yt ∼ N (0, t2 σ 2 +
(1 − t)2 σ 2 ). Now, it is easy to see that with a proper scaling factor, we can adjust the variance of
Yt back to σ 2 . That is, √ 2 1
Yt ∼ N (0, σ 2 ), so by Lemma 1 gt (y) := √ 2 1
y is the
2
2
t +(1−t)

t +(1−t)

monotone transport map from Yt to Z.
By adjusting the vicinity sampling operation to
zi0 := z + ei ,
where ei ∼ N (0, 1), we can similarly find the monotone transport map g (y) =

(32)
√ 1
y.
1+2

Another operation which has been used in the literature is the “analogy”, where from samples
z1 , z2 , z3 , one wants to apply the difference between z1 and z2 , to z3 . The transport map is then
g(y) = √13 y

16

5.4

A DDITIONAL EXPERIMENTS

(a) Linear interpolation

(b) Spherical linear interpolation

(c) Distribution matched

Figure 12: 4-point interpolation between 4 sampled points (corners) from DCGAN trained on
CelebA with Gaussian prior. The same interpolation is shown using linear, SLERP and distribution
matched interpolation.

(a) Linear

(b) Spherical

(c) Distribution matched

Figure 13: Midpoint sampling for linear, SLERP and uniform-matched interpolation when using the
same pairs of sample points on CelebA with Gaussian prior.

(a) Vicinity sampling

(b) Matched vicinity sampling

Figure 14: Vicinity sampling on CelebA dataset with Gaussian prior. The sample in the middle is
perturbed in random directions producing the surrounding sample points.

17

(a) LLD icon dataset

(b) LSUN dataset

Figure 15: 2-point interpolation in detail: Each example shows linear, SLERP and transport matched
interpolation from top to bottom respectively, with 16 points taken from the path, produced with
DCGAN using a uniform prior distribution.

18

