Conditional Autoencoders with Adversarial Information Factorization
Antonia Creswell
Imperial College London

Anil A Bharath
Imperial College London

ac2211@ic.ac.uk

a.bharath@ic.ac.uk

Biswa Sengupta
Noah’s Ark Lab, Huawei
London

arXiv:1711.05175v1 [cs.CV] 14 Nov 2017

biswa.sengupta@huawei.com

Abstract

shape of their face. Work by Bao et al. [4] propose latent
space generative models for synthesizing images from finegrained categories, synthesizing different celebrities’ faces
conditional on the identity of the celebrity.
Rather than considering fine-grain categories, we propose to take steps towards solving the different, but related problem of image attribute manipulation. To solve
this problem we want to be able to synthesize images and
change one element or attribute of its content. For example,
if we are synthesizing faces we would like to edit whether
the person is smiling or not. This problem is different to
the fine grain synthesis problem because we should be able
to synthesize two images that are similar, with only a single (chosen) attribute changed, rather than synthesizing two
different faces. This problem is also more difficult than
the fine-grain image synthesis problem because we need to
learn a latent space representation that separates an object
category from its attributes.
Addressing this problem is interesting because it may enable us to use latent space generative models for image editing. Currently, there are several limitations to using latent
space generative models that need to be addressed before
they may be used for image editing:

Generative models, such as variational auto-encoders
(VAE) and generative adversarial networks (GAN), have
been immensely successful in approximating image statistics in computer vision. VAEs are useful for unsupervised
feature learning, while GANs alleviate supervision by penalizing inaccurate samples using an adversarial game. In
order to utilize benefits of these two approaches, we combine the VAE under an adversarial setup with auxiliary label information. We show that factorizing the latent space
to separate the information needed for reconstruction (a
continuous space) from the information needed for image
attribute classification (a discrete space), enables the capability to edit specific attributes of an image.

1. Introduction
Latent space generative models, for example, generative
adversarial networks (GANs) [11, 19] and variational autoencoders (VAEs) [12] learn a mapping from a latent encoding space to a data space, for example, the space of natural images. It has been shown that the latent space learned
by these models is often organised in a near-linear fashion
[19, 12], where by neighbouring points in latent space map
to similar images in data space. Further, certain “directions”
in latent space correspond to changes in the intensity of certain attributes, for example, for faces, the extent to which
some one is smiling. This may be useful for image synthesis where one can use the latent space to develop new
design concepts [9, 24], edit an image [24] or synthesize
avatars [23, 22]. This is because semantically meaningful
changes may be made to images by manipulating the latent
space [19, 24].
One direction for latent space generative models has
been class conditional image synthesis [5, 18, 17], where an
image of a particular object category is synthesized. Often,
object categories may be sub-divided into finer grain subcategories. For example, the category “face” may be split
into further sub-categories depending, for example, on the

1. Fidelity and quality of reconstructions. To edit an
image, we first need to be able to accurately reconstruct the image. VAEs [12] are typically good at reconstructing inputs, however, outputs are often blurry.
While GANs achieve a superior quality of samples
they lack inverse models to map from data space, back
to image space. Various extensions of GANs [7, 10]
do have an inverse model, however, reconstructions
are poor. More recently, reconstruction has been addressed by Li et al. [14] – making vast improvements
on both quality as well as the fidelity of reconstructions.
2. Control over independent attributes. In image editing, one requires the ability to target and change single attributes of an image, independently. However,
typically when using latent space generative models to
1

change one target attribute, other attributes change too
[13], or – in some cases – the image may not change at
all.

log p(x) ≥ Eqφ (z|x) log pθ (x|z) + KL[qφ (z|x)||p(z)] (1)

Further, to these limitations, overcoming one does not
automatically avoid the other, and often the two goals can
conflict. For example, consider the case where a generative
model has a latent space with the same dimensionality as
an image, it is possible to ‘copy’ the image to the latent
space and back obtaining perfect reconstruction. However,
the latent space of this generative model is (probably) not
useful for image editing.
In this paper, we propose a method to address the second
issue and demonstrate our results using a model that combines the reconstruction properties of a VAE and the image quality of a GAN. We apply our method to the CelebA
dataset of (un-cropped) faces and control the “smile” attribute. Our model consists of several components, we evaluate the contribution of each component to the model and
demonstrate that our model may be used to successfully edit
the ‘smile’ attribute in more that 90% of the test cases.
Our contributions are as follows:

where, p(z) is a chosen prior distribution, for example
p(z) = N (0, I). The encoder predicts, µφ (x) and σφ (x)
for a given input x, and a latent sample ẑ is drawn from
qφ (z|x) as follows:  ∼ N (0, I) then z = µφ (x) + σφ (x) ∗
. By choosing a multivariate Gaussian prior, the KLdivergence may be calculated analytically [12]. The first
term in the loss function is typically approximated by calculating the reconstruction error between many samples of
x and x̂ = Dθ (Eφ (x)).
New data samples, not present in the training data, are
synthesized by first drawing latent samples from the prior,
z ∼ p(z) and then drawing data samples from pθ (x|z);
equivalent to passing the z samples through the decoder,
Dθ (z).
VAEs offer both a generative model, pθ (x|z), and an encoding model, qφ (z|x), which are useful as starting points
for image editing in the latent space. However, samples
drawn from a VAE are often blurred [19].

2.2. Conditional VAEs

• A novel architecture for combining VAE and the GAN
architecture, with an ability to edit singular attributes
of an image.

Autoencoders may be augmented in many different ways
to achieve category-conditional image synthesis [13, 4, 15].
It is common to append a one-hot label vector, y, to inputs
of the encoder and decoder [13]. However, for small label vectors (relative to the size of the inputs to the encoder
and the decoder model), it is possible for the label information, y to be ignored. A more interesting approach, for conditional (non-variational, semi-supervised) autoencoders is
presented by Makhzani et al. [15]. This an encoder that
gives both a latent vector ẑ and an attribute vector ŷ. The
encoder is updated to minimize a classification loss between
the true label y and ŷ. We incorporate a similar architecture
into our model, but we make additional modifications to the
training of the encoder for the reasons explained below.
There is a drawback to incorporating attribute information in this [15] way when the purpose of the model is to edit
specific attributes, rather than to synthesize samples from a
particular category. We observe that in this naive implementation of conditional VAEs, varying attribute (or label) ŷ for
a fixed ẑ can results in unpredictable changes in synthesized
data samples, x̂. For example, if for a fixed ẑ, changing ŷ
does not result in the (intended) attribute corresponding to ŷ
to change. This suggests that information about the attribute
one wishes to edit, y, is partially contained in ẑ rather than
ŷ. Similar problems have been discussed and addressed to
some extent in the GAN literature [5, 17, 18], where it has
been observed that label information in ŷ is often ignored
during sample synthesis.
In general, one may think that ẑ and ŷ are independent. However, if attributes, y, that should be described

• A cost function for training the encoder such that the
latent representation factorizes information about attributes such as reconstruction fidelity (continuous attribute) and label information (discrete attribute).
• Targeted attribute editing in images, using a latent variable generative model.
Code for training and evaluation is available here:
https://github.com/ToniCreswell/attribute-cVAEGAN.git.

2. Latent space generative models
Latent space generative models come in various forms.
Two state-of-art generative models include Variational Autoencoders (VAE) [12] and Generative Adversarial Autoencoders (GAN). Both models allow synthesis of novel data
samples from latent encodings. Here we explain each model
in more detail.

2.1. Variational Autoencoder (VAE)
Variational autoencoders [12] consist of an encoder
qφ (z|x) and decoder pθ (x|z); often times these can be instantiated as neural networks, Eφ (·) and Dθ (·) respectively,
with learnable parameters, φ and θ. A VAE is trained to
minimize the evidence lower bound (ELBO) on log p(x),
where p(x) is the data generating distribution. The ELBO
is given by:
2

by ŷ remain unchanged for a reconstruction where only ŷ is
changed, this suggests that ẑ contains most of the information that should be contained in ŷ. We propose to separate
the information about y from ẑ using a mini-max optimization involving y, ẑ, the encoder, Eφ and an auxiliary network, Aψ .

ble samples in the training data. If the training data consists of images from multiple categories, synthesized samples may come from anyone (or possibly a combination) of
those categories. For a vanilla VAE, one cannot choose to
synthesize samples from a particular category. Conditional
VAEs (and GANs) [5, 18, 17] provide an approach to synthesize class-specific data samples.

2.3. Generative Adversarial Networks (GAN)

2.5. Our proposed solution

An alternative generative model, which may be used to
synthesize much sharper images is the Generative Adversarial Network (GAN) [11, 19]. GANs involve two models,
a generator, Gθ (·) and a discriminator Cχ (·), which may be
implemented using convolutional neural networks [19, 6].
GAN training involves these two networks engaging in a
min-max game. The discriminator, Cχ , is trained to classify
samples from Gθ as being ‘fake’ and samples from the data
generating distribution, p(x), as being ‘real’. The generator
is trained to synthesize samples that confuse the discriminator; that is, to synthesize samples that the discriminator
cannot distinguish from the ‘real’ samples. The objective
function is given by:

If a latent encoding, ẑ contains desired attribute information, y, that should instead be in the attribute vector, ŷ,
then a classifier should be able to accurately predict y from
ẑ. Ideally, ẑ contains no information about y and so a, classifier should not be able to predict y from ẑ. We propose
to train an auxiliary network to predict y from ẑ accurately
while updating the encoder of the VAE to output ẑ values
that cause the auxiliary network to fail. If ẑ contains no
information about the desired attribute, y that we wish to
edit, then the information can be conveyed in ŷ instead, and
x̂ must contain that information in order to (still) minimize
reconstruction loss. We now formalize these ideas.

3. Methods

min max Ep(x) [log(Cχ (x)] +Epg (x) [log(1−Cχ (x))] (2)
χ

θ

In what follows, we will introduce our novel VAE-GAN
architecture. Then, we will explain our novel approach
to training the encoder to factor (separate) out information
about y from ẑ, such that H(y|ẑ) ≈ H(y).

where pg (x) is the distribution of synthesized samples, sampled by: z ∼ p(z), then x = Gθ (z), where p(z) is a chosen
prior distribution, for example a multivariate Gaussian.

3.1. Model Architecture

2.4. Best of both GAN and VAE

A diagram of our architecture is presented in Figure 1. In
addition to the encoder, Eφ , decoder, Dθ , and discriminator,
Cχ , we introduce an auxiliary network, Aψ : ẑ → ỹ, whose
purpose is described in detail in Section 3.2. We use ŷˆ to
indicate the predicted label of a reconstructed data sample.
There are multiple cost functions that are used to train
our model. We introduce these cost functions and describe
the combinations of cost functions that are used to update
each set of network parameters. We define the following
loss terms:

The vanilla GAN model does not provide an easy method
to map data samples to latent space. Although, there are
several variants on the GAN that do involve learning an encoder type model [10, 8, 14], only the approach presented
by Li et al. [14] allows data samples to be faithfully reconstructed. The approach presented by Li et al. requires adversarial training to be applied to several high dimensional
distributions. Training adversarial networks on high dimensional data samples is challenging [2] despite several proposed improvements [20, 3]. For this reason, rather than
adding a decoder to a GAN, we consider an alternative latent generative model that combines a VAE with a GAN. In
this arrangement, the VAE may be used to learn an encoding
and decoding process, and a discriminator may be placed at
the end of the decoder to ensure the quality of data samples
at the output of the decoder. Indeed, there have been several
suggestions on how to combine VAEs and GANs [4, 13, 16]
each has a different structure and a set of loss functions,
none are designed specifically for attribute editing.
The content of (image) samples synthesized from a
vanilla VAE (or GAN) depends on the latent variable z,
which is drawn from a (chosen) random distribution, p(z).
For a well-trained model, synthesized samples will resem-

Lrec = Lbce (x̂, x)
Lgan =
1
[Lbce (Cχ (x), real)
3
+ Lbce (Cχ (Dθ (Eφ (x))), f ake)

(3)

(4)

+ Lbce (Cχ (Dθ (z)), f ake)]

3

Lclass = Lbce (ŷ, y)

(5)

ˆ y)
L̂class = Lbce (ŷ,

(6)

Laux = Lbce (ỹ, y)

(7)

Lenc = Lrec + αLKL + β L̂class + ρLclass − δLgan (10)
α, β and ρ are the regularization coefficients.
The loss function in (10) is not sufficient for training an
encoder used for attribute manipulation. For this, we propose an additional network and an additional cost function,
as described below.

(a) Current work

3.2. Information Factorization
To factor label information, y out of ẑ we propose to introduce an additional, auxiliary network, Aψ that is trained
to correctly predict y from ẑ. At the same time the encoder,
Eφ , is updated to cause Aψ to make incorrect classifications. In this way, the encoder is encouraged not to place
attribute information (y) in ẑ. This may be described by the
following mini-max objective:

(b) Previous work [4]

Figure 1: IFcVAEGAN with Information Factorization
This figure shows two variants of our model. The core of
the model is shown in blue. This is a VAE with information
factorization, note that, Eφ is split in two, Ey,φ and Ez,φ ,
for obtaining both a latent encoding ẑ and the label ŷ respectively. Dθ is the decoder and Aψ the auxiliary network. The
pink blocks show how a GAN architecture may be incorporated by placing a discriminator Cχ , after the encoder, Eφ ,
and training Cχ to classify decoded samples as “fake” and
samples from the dataset as “real”. Finally, the dotted lines
show how decoded samples may be passed back through the
encoder to obtain a label, ŷˆ which may be used to obtain a
gradient, containing label information, for updating the decoder of the VAE. For simplicity, the KL regularization is
not shown in this figure. Previous work: CVAEGAN [4]
Architecture most similar to our own. Note that there is no
auxiliary network performing information factorization and
classifier is not built into the encoder and therefore does not
share features (parameters) learned by the encoder.
LKL = KL[qφ (z|x)||p(z)]

min max Lbce (Aψ (Ez,φ (x)), y)
ψ

= min max Laux
ψ

(11)

φ

where Ez,φ(x) is the latent output of the encoder.
Training is complete when the Aψ is maximally confused and cannot predict y from ẑ = Ez,φ (x), where y is
the true label of x. The encoder loss for information factorization is therefore given by:
LIF cV AEGAN = Lenc − Laux

(12)

We call the conditional VAEGAN trained in this way,
an Information Factorized cVAEGAN (IFcVAEGAN). The
training procedure is presented in Algorithm 1.

(8)

3.3. Attribute Manipulation

where costs are calculated over a batch of data
samples,{xi }M
{y }M ; {z}iM =1 ∼ p(z) =
i=1 with labels
PM i i=1
1
N (0, I); Lbce (a, b) = M i=1 a log b + (1 − a) log(1 − b);
real and f ake, are vectors of ones and zeros respectively.
The parameters, θ of the decoder are updated with gradients from the following loss functions:
Ldec = Lrec + β L̂class − δLgan

φ

To edit an image such that it has the desired attribute,
we first encode the image to obtain a ẑ (the identity), append this to our desired attribute label, ŷ = y and pass this
through the decoder. We use ŷ = 0 and ŷ = 1 to synthesize
samples in each mode of the desired attribute e.g. ‘smile’
and ‘no smile’.

(9)

4. Results

note that the L̂class is a classification loss on reconstructed
data samples. Calculating this loss provides a gradient
which contains label information to the decoder, which otherwise the decoder would not have [5]. We use β = [0, 1].
Discriminator parameters, χ are updated to minimize Lgan .
The parameters, φ of an encoder, designed to synthesize
samples from a desired category may be updated by minimizing the following cost functions:

In this section, we show both qualitative and quantitative results to evaluate our proposed model. We refer to a
cVAEGAN that is trained without a L̂class or Laux term in
the cost function as a naive cVAEGAN and a cVAEGAN
trained with the Laux term as an Information Factorization
cVAEGAN (IFcVAEGAN). Qualitatively our results illustrate that:
4

Algorithm 1 Training Information Factorization cVAEGAN (IFcVAEGAN): The prior, p(z) = N (0, I).
1:

code. We trained the model for 8 epochs, further training
lead to worse results. However, when reconstructing samples and setting ŷ = 0 for ‘no smile’ and ŷ = 1 for ‘smile’,
we found that the naive cVAEGAN failed to synthesize
samples with the desired target attribute, ‘smile’. Rather,
the model targeted an undesired attribute, ‘hair colour’, this
is shown in Figure 3. For fair comparison, we trained an
IFcVAEGAN with similar a hyper-parameter combination,
with additional parameter value, β = 1.0 and α = 0.5, to
facilitate factorization. The results are shown in Figure 2.
In our experiments, we found that smaller α values resulted both in better reconstructions and better control over
attributes in an image. Figure 4 shows an IFcVAEGAN
trained with the hyper-parameters, {ρ = 0.1, δ = 0.1, α =
0.2, β = 0.0, nz = 200}. Note that we consider this to
be our ‘best model’ in terms of the visual quality of results
given the hyper-parameters that we experimented with. We
use β = 0.0 meaning that we do not use L̂class loss in this
model.
Note that models that involve GANs are often difficult to
train and may not converge [2]. We found that training the
discriminator, Cχ , using optimization method, RMSprop,
with a small momentum (0.1), did lead to the loss functions converging, during training. However, this affected
the sharpness of samples synthesized. Rather, we used a
larger momentum, momentum = 0.5, which often did not
converge, but resulted in sharper images. We do not show
examples of the effect of different momentum values for the
sake of space and relevance.

procedure T RAINING C VAE WITH INFORMATION
FACTORIZATION

2:
3:
4:
5:
6:
7:
8:
9:
10:
11:
12:
13:
14:
15:
16:
17:
18:
19:
20:

for i in range(N ) do
. N is no. of epochs
. Forwards pass through all networks
x∼D
. D is the training data
z ∼ p(z)
ẑ, ŷ ← Eφ (x)
x̂ ← Dθ (ŷ, ẑ)
ỹ ← Aψ (ẑ) . output of the auxiliary network
. Calculate updates, u
uθ ← −∇θ Ldec
uφ ← −∇φ LIF cV AEGAN
uχ ← −∇χ Laux
uψ ← −∇ψ Lgan
# do updates
θ ← RMSprop(θ, uθ )
φ ← RMSprop(φ, uφ )
χ ← RMSprop(χ, uχ )
ψ ← RMSprop(ψ, uψ )
end for
end procedure

• A naive cVAEGAN may fail, changing the wrong attributes when ŷ is changed, while the Information Factorization cVAEGAN, (IFcVAEGAN) changes the desired attributes (with all of the other model parameters
fixed).
• By fine-tuning an IFcVAEGAN, particularly reducing
the level of regularisation, α, on the KL-term in the
VAE loss, the model achieves better reconstruction of
data samples while maintaining the ability to edit attributes.

(a) Smiling

Having fine-tuned our model, we test the contribution of
each component in our model, by adding and removing
terms from the IFcVAEGAN cost function. We also study
the effect of the weighting on the KL regularisation term.
Finally, we show examples of editing other face attributes
such as ‘wearing eyeglasses’ and ‘male’ (gender).

(b) Not Smiling

Figure 2: Our Work (Information Factorization cVAEGAN, (IFcVAEGAN)) Here we see that the IFcVAEGAN
targets the desired attribute, ‘smile’ successfully.

4.1. Qualitative Results
Here, we demonstrate how a naive cVAEGAN may fail
– changing undesired attributes, rather than the desired
one. We experimented with various different hyperparameters to obtain good quality samples for the naive cVAEGAN. We found that smaller α values led to poorer samples, while larger α samples led to the wrong attributes
being changed. We found hyper-parameter combination,
{ρ = 0.1, δ = 0.1, α = 1.0, nz = 200} to give the highest quality samples, where nz is the dimension of the latent

4.2. Contributions of each component to the final
model
We analyze the contribution of each component of our
proposed model and evaluate the effect of regularization of
the KL divergence term (Table 1). We consider reconstruction error and classification accuracy on synthesized data
5

Table 1: What are the essential parts of the model? This
table shows how novel components of the IFcVAEGAN loss
functions affect mean squared error (MSE) reconstruction
error and the ability to edit the data samples. Ability to edit
samples is quantified by a classification accuracy, (Class1,
Class0), on samples for ŷ = 1 and ŷ = 0, where the ground
truth label is y = 1 and y = 0 respectively. This may
be thought of as the proportion of samples that have the
desired attribute. Our ‘best model’ has hyper-parameters
{ρ = 0.1, δ = 0.1, α = 0.2, β = 0.0}, without L̂class and
with information factorization Laux .

(a) Smiling

(b) Not Smiling

Figure 3: Previous Work (naive cVAEGAN) Example of
a naive cVAEGAN failing to change the desired attribute
when ŷ is changed. Rather than ŷ controlling the ‘smile’
attribute as intended, ŷ controls the ‘hair colour’ attribute.

Our ‘best model’ (α = 0.2)
Regularization (α = 0.5)
W/ L̂class , (α = 0.2)
W/Out Laux , (α = 0.2)
W/Out Laux , W/L̂class , (α = 0.2)

MSE

Class0

Class1

0.028
0.0300
0.028
0.0280
0.027

0.813
0.5000
0.938
0.188
0.000

1.000
0.938
0.938
0.813
1.000

lowing observations:
• Effect of KL regularisation: Our model performs
better with lower levels of regularisation; α = 0.2
rather than α = 0.5. Results indicate that higher levels of regularisation reduce the model’s ability to reconstruction data samples well, while also hindering
the ability to edit attributes well. Only half of the
samples synthesized, with ŷ = 0, have the desired
attributes. This is likely because over regularisation
increases I(x; z), since KL divergence is directly related to I(x; z). Making it more difficult for ŷ to contribute to the information in x.

(a) Original

(b) Smiling

• Effect of L̂class : Using L̂class does not provide any
clear benefit. We explored using this term because a
similar approach had been proposed in the GAN literature [5, 18] for conditional image synthesis (rather
than attribute editing) and as far as we could tell, this
approach had not been used in the VAE literature. This
term is intended to maximize I(x, y) by providing a
gradient containing label information to the decoder.
However, it does not address the factorization of attribute information, y, from ẑ.

(c) Not Smiling

Figure 4: IFcVAEGAN reconstruction

samples. Smaller reconstruction error indicates better reconstruction, and larger classification values suggest better
control over attribute changes. The classifier is trained to
classify ‘smile’ (y = 1) vs. ‘not smile’ (y = 0). We apply
the classifier to two sets of samples synthesized using ŷ = 0
and ŷ = 1. If the desired attributes are changed the classification scores should be high for both sets of samples, if not,
the classifier is likely to perform well on only one of the
sets, suggesting that the attribute was not edited, but fixed.
Note, all original test data samples for this experiment were
from the ‘smile’ category. The results are shown in Table
1. We may interpret classification score as the proportion
of samples with the desired attributes and MSE error as the
fidelity of reconstruction. From Table 1, we make the fol-

• Effect of Information Factorization: Without our
proposed Laux term in the encoder loss function, the
model fails completely to perform attribute editing.
Since Class0+Class1= 1 this strongly suggests that
samples are synthesized independently of ŷ and that
the synthesized images are the same for ŷ = 0 and
ŷ = 1.
• Effect of L̂class on its own: For completeness, we
also evaluated our model without Laux but with L̂class
6

5. Related work and Conclusions

to test the effect of L̂class on its own. Though
similar approaches have been successful for category
conditional image synthesis, it was not as successful on the attribute editing task. Similarly, as above,
Class0+Class1= 1, suggesting that samples are synthesized independently of ŷ. Further, Class0=0.0, suggests that none of the synthesized images has the desired attribute ŷ = 0 (‘no smile’), i.e. all samples are
with attribute ‘smile’. This supports the use of Laux
when training models for attribute editing over L̂class
which is suggested in the GAN literature [5, 18] for
category specific samples synthesis.

Our work introduces a novel architecture to combine
VAEs and GANs and propose modifications to the conditional latent space generative models in order to make them
more amenable for attribute (for example, facial attribute)
manipulation. At the cost of introducing an extra neural
network (the auxiliary network), we show that factorizing
the latent space into sub-sets that encode information about
discrete labels and continuous reconstruction, separately,
enable us to attain improvements, in terms of targeted attribute editing, over the current state-of-the-art conditional
VAE-GAN architecture.
Schmidhuber [21] performs similar factorization of the
latent space, ensuring that each component of the encoding
is independent. This is achieved by learning an encoding
such that other elements in the encoding may not be predicted from a subset of remaining elements. We use related
concepts, with additional class label information, and incorporate the encoding in a generative model.
Our work has the closest resemblance to the CVAE-GAN
architecture (see Figure 1), proposed by Bao et al. [4].
CVAE-GAN is designed for synthesizing samples of a particular class, rather than manipulating a single attribute of
an image from a class. In short, their objective is to synthesize a “Hathway” face, whereas our objective would be to
make “Hathway smile” or “Hathway not smile”, which has
different demands on the type of factorization in the latent
representation. It is sufficient to have a representation that
divides object categories, rather than attributes of objects in
a category. Separating categories is an easier problem because it is possible to have distinct categories, and a change
in the category may result in larger changes in the image.
For an attribute change, the change needs to be targeted,
making minimal changes to the rest of the image.
In a similar vein to our work, Antipov et al. [1] acknowledge the need for “identity preservation” in the latent space.
They achieve this by introducing an identity classification
loss between an input data sample and a reconstructed data
sample, rather than trying to separate information in the encoding itself. Rather than limiting our objective to identity
preservation, we want to control single attributes of an image without interfering with other attributes that may not affect identity i.e., wearing a glass or altering the hair colour
may not affect identity, but we also do not want to change
them. Similar to our work, Larsen et al. [13] use a VAE
with discriminator on the end; however, unlike them we do
not use feature matching. Additionally, their architecture
does not condition on label information, instead they search
the latent space and add attribute vectors encodings. The attribute vector is the difference between mean encodings of
several samples with the same attribute. Unlike our work,
under their framework other unintended attributes change
when one attribute is targeted. Additionally, the “editing”

4.3. Editing other attributes
In this section we apply our proposed method to manipulate other facial attributes including: gender ‘male’, ‘not
male’ and whether a person is wearing eye glasses for not.
The initial samples from which the ẑ’s are obtained are test
samples whose labels are y = 1. Figure 5 shows examples
of switching between the ‘male’ and ‘not male’ attribute.
While Figure 6 shows examples of faces with and without
eye glasses.

(a) Male (y = 1)

(b) Not Male (y = 0)

Figure 5: Editing the ‘male’ attribute

(a) Wearing eye glasses (y = 1)

(b) Not wearing eye glasses (y = 0)

Figure 6: Editing the ‘eye glasses’ attribute

7

process is not done in an end to end fashion. They do introduce a conditional version of their model, but use this
to synthesize samples, rather than edit attributes of existing
samples.
All in all, our work highlights an important difference
between category conditional image synthesis and attribute
editing in images – what works for category conditional image synthesis may not work for attribute editing. Also, for
attribute editing to be successful it is necessary to factor label information out of the latent encoding.

[15] A. Makhzani, J. Shlens, N. Jaitly, I. Goodfellow, and B. Frey.
Adversarial autoencoders. arXiv preprint arXiv:1511.05644,
2015. 2
[16] L. Mescheder, S. Nowozin, and A. Geiger. Adversarial variational bayes: Unifying variational autoencoders and generative adversarial networks. arXiv preprint arXiv:1701.04722,
2017. 3
[17] M. Mirza and S. Osindero. Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784, 2014. 1, 2, 3
[18] A. Odena, C. Olah, and J. Shlens. Conditional image
synthesis with auxiliary classifier gans. arXiv preprint
arXiv:1610.09585, 2016. 1, 2, 3, 6, 7
[19] A. Radford, L. Metz, and S. Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434, 2015. 1,
2, 3
[20] T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen. Improved techniques for training gans. In
Advances in Neural Information Processing Systems, pages
2234–2242, 2016. 3
[21] J. Schmidhuber. Learning factorial codes by predictability
minimization. Learning, 4(6), 2008. 7
[22] Y. Taigman, A. Polyak, and L. Wolf. Unsupervised crossdomain image generation. arXiv preprint arXiv:1611.02200,
2016. 1
[23] L. Wolf, Y. Taigman, and A. Polyak. Unsupervised creation
of parameterized avatars. arXiv preprint arXiv:1704.05693,
2017. 1
[24] J.-Y. Zhu, P. Krähenbühl, E. Shechtman, and A. A. Efros.
Generative visual manipulation on the natural image manifold. In European Conference on Computer Vision, pages
597–613. Springer, 2016. 1

References
[1] G. Antipov, M. Baccouche, and J.-L. Dugelay. Face aging with conditional generative adversarial networks. arXiv
preprint arXiv:1702.01983, 2017. 7
[2] M. Arjovsky and L. Bottou. Towards principled methods
for training generative adversarial networks. arXiv preprint
arXiv:1701.04862, 2017. 3, 5
[3] M. Arjovsky, S. Chintala, and L. Bottou. Wasserstein gan.
arXiv preprint arXiv:1701.07875, 2017. 3
[4] J. Bao, D. Chen, F. Wen, H. Li, and G. Hua. Cvae-gan:
Fine-grained image generation through asymmetric training.
arXiv preprint arXiv:1703.10155, 2017. 1, 2, 3, 4, 7
[5] X. Chen, Y. Duan, R. Houthooft, J. Schulman, I. Sutskever,
and P. Abbeel. Infogan: Interpretable representation learning
by information maximizing generative adversarial nets. In
Advances in Neural Information Processing Systems, pages
2172–2180, 2016. 1, 2, 3, 4, 6, 7
[6] E. L. Denton, S. Chintala, R. Fergus, et al. Deep generative image models using a laplacian pyramid of adversarial
networks. In Advances in neural information processing systems, pages 1486–1494, 2015. 3
[7] J. Donahue, P. Krähenbühl, and T. Darrell. Adversarial feature learning. arXiv preprint arXiv:1605.09782, 2016. 1
[8] J. Donahue, P. Krähenbühl, and T. Darrell. Adversarial feature learning. arXiv preprint arXiv:1605.09782, 2016. 3
[9] A. Dosovitskiy, J. T. Springenberg, M. Tatarchenko, and
T. Brox. Learning to generate chairs, tables and cars with
convolutional networks. IEEE transactions on pattern analysis and machine intelligence, 39(4):692–705, 2017. 1
[10] V. Dumoulin, I. Belghazi, B. Poole, A. Lamb, M. Arjovsky,
O. Mastropietro, and A. Courville. Adversarially learned inference. arXiv preprint arXiv:1606.00704, 2016. 1, 3
[11] I. J. Goodfellow, J. Shlens, and C. Szegedy. Explaining and harnessing adversarial examples. arXiv preprint
arXiv:1412.6572, 2014. 1, 3
[12] D. P. Kingma and M. Welling. Auto-encoding variational
bayes. arXiv preprint arXiv:1312.6114, 2013. 1, 2
[13] A. B. L. Larsen, S. K. Sønderby, H. Larochelle, and
O. Winther. Autoencoding beyond pixels using a learned
similarity metric. arXiv preprint arXiv:1512.09300, 2015.
2, 3, 7
[14] C. Li, H. Liu, C. Chen, Y. Pu, L. Chen, R. Henao,
and L. Carin. Towards understanding adversarial learning for joint distribution matching.
arXiv preprint
arXiv:1709.01215, 2017. 1, 3

8

