Photo-to-Caricature Translation on Faces in the Wild

arXiv:1711.10735v1 [cs.CV] 29 Nov 2017

Ziqiang Zheng

Haiyong Zheng Zhibin Yu Zhaorui Gu
Ocean University of China

Abstract

input =

Recently, image-to-image translation has been made
much progress owing to the success of conditional Generative Adversarial Networks (cGANs). However, it’s still
very challenging for translation tasks with the requirement
of high-level visual information conversion, such as phototo-caricature translation that requires satire, exaggeration,
lifelikeness and artistry. We present an approach for learning to translate faces in the wild from the source photo domain to the target caricature domain with different styles,
which can also be used for other high-level image-to-image
translation tasks. In order to capture global structure with
local statistics while translation, we design a dual pathway model of cGAN with one global discriminator and one
patch discriminator. Beyond standard convolution (Conv),
we propose a new parallel convolution (ParConv) to construct Parallel Convolutional Neural Networks (ParCNNs)
for both global and patch discriminators, which can combine the information from previous layer with the current
layer. For generator, we provide three more extra losses in
association with adversarial loss to constrain consistency
for generated output itself and with the target. Also the
style can be controlled by the input style info vector. Experiments on photo-to-caricature translation of faces in the
wild show considerable performance gain of our proposed
method over state-of-the-art translation methods as well as
its potential real applications.

output

Bing Zheng

input =

output

Figure 1. Translating faces in the wild from photo to caricature
with different styles by our proposed method.

of its subject in an exaggerated way and usually used to
describe a politician or movie star for political or entertainment purpose. Creating caricatures can be considered
as artistic creation tracing back to the 17th century with
the profession of caricaturist. Then some efforts have
been made to produce caricatures semi-automatically using
computer graphics techniques [1], which intend to provide
warping tools specifically designed toward rapidly producing caricatures. But there are very few software programs
designed specifically for automatically creating caricatures,
and to the best of our knowledge, none can work to be comparable with caricaturist. Nowadays, besides the political
and public-figure satire, caricatures are also used as gifts or
souvenirs, and more and more museums dedicated to caricature throughout the world were opened. So it would be
very useful and meaningful if computers can create caricatures from photos automatically and intelligently.
Photo-to-caricature is a typical high-level image-toimage translation problem but with bigger challenge than
other low-level translation problems such as photo-to-label,
photo-to-map, or photo-to-sketch [7], because caricatures

1. Introduction
Image-to-image translation has been made much
progress [7, 36, 32, 2, 8] because many tasks in image
processing, computer graphics, and computer vision can
be posed as translating an input image into a corresponding output image [35, 11, 30, 10, 27, 25, 4]. And its
achievements mainly owes to the success of Generative Adversarial Nets (GANs) [5], especially conditional GANs
(cGANs) [18, 22, 7]. However, the current studies mainly
concern image-to-image translation tasks with low-level visual information conversion, e.g., photo-to-sketch.
A caricature is a rendered image showing the features

• require satire and exaggeration of photos;
• need artistry with different styles;
• must be lifelike, especially the expression of a face
photo.
Specifically, for a face photo, we want to create the face
caricatures with different styles, which exaggerate the face
1

shape or facial sense organs (i.e., ears, mouth, nose, eyes
and eyebrow) but keep the vivid expression while producing
artistry.
In this paper, we propose a GAN-based method for learning to translate faces in the wild from the source photo domain to the target caricature domain with different styles
(see Fig. 1 for translating examples and Fig. 2 for the architecture of our proposed method). Based on the model
of pix2pix [7], we design a dual pathway model of
cGAN for image-to-image translation, where one pathway
of global discriminator is in charge of caring the global
structure information, while another pathway of patch discriminator is responsible for concerning the local statistics
information. And for both global and patch discriminators,
we propose a new parallel convolution (ParConv) instead of
the standard convolution (Conv) to construct Parallel Convolutional Neural Networks (ParCNNs). Our proposed parallel convolutional layer can combine the information from
the previous layer with the information of the current layer,
so that the ParCNN will connect different channels to avoid
structure error. For generator, besides the adversarial loss,
we provide another three more extra weighted losses (realfake L1 loss, global perceptual similarity loss, and cascade
layer loss) to constrain consistency for generated output itself and with the target. By using our proposed method, the
photos of faces in the wild can be translated to caricatures
with different exaggerated artistic styles while still keeping
the original lifelike expression (Fig. 1 and 11).
We have extensively evaluated our method on IIIT-CFW
dataset [19], FEI dataset [28], ADFES dataset [29], CelebA
dataset [13], KDEF dataset [14], and WSEFEP dataset [20].
The experimental results show that our method can create
acceptable caricatures from face photos while state-of-theart image-to-image translation methods can’t. Also the designed experiments indicate the effectiveness of our proposed dual pathway of discriminators, parallel convolution
and extra losses, respectively. Besides, we tested our phototo-caricature translation method for producing caricatures
with different styles. Furthermore, the proposed method can
create caricatures from arbitrary face photos. This might be
helpful to fill aforementioned gap of automatic and intelligent caricature creation. Our source code will be published
with the paper.

image translation developed by Isola et al. [7] with the released pix2pix software have achieved reasonable results
on many translation tasks such as photo-to-label, phototo-map and photo-to-sketch. Then CycleGAN [36], DualGAN [32], and DiscoGAN [8] were proposed for unpaired or unsupervised image-to-image translation with almost comparable results to paired or supervised methods.
However, the translation tasks these work can tackle usually
concern the conversion of low-level visual information such
as line (photo→sketch), color (summer→winter), and texture (horse→zebra), but it’s still very challenging for some
translation tasks with the requirement of high-level visual
information conversion, e.g., abstraction and exaggeration
(photo→caricature).

2. Related Work

3. Method

Image-to-image translation. Owing to the success of
GANs [5], especially various conditional GANs [18, 17, 23,
31, 30, 33], image-to-image translation problems have been
made much progress recently. Earlier image-conditional
models for specific applications have achieved impressive
results on inpainting [21], de-raining [33], texture synthesis [11], style transfer [30], video prediction [17] and superresolution [10]. The general-purpose solution for image-to-

Conditional GANs are generative models that learn a
mapping from observed input image x and random noise
vector z, to target image y, G : {x, z} → y. The objective
of our conditional GAN with auxiliary style information z

Photo-to-cartoon translation. Translating photo to cartoon by computer algorithms has been studied for a long
time because of the corresponding interesting and meaningful applications. The earlier work mainly relied on the
analysis of facial features [15, 3, 12], which is hard to be
applicable for large-scale faces in the wild. Thanks to the
invention of GANs [5], automatic photo-to-cartoon translation becomes feasible. But most of the current related
work mainly focused on the generation of anime [34] or
emoji [27] with specific style1 . And these works have nothing to do with caricature creation that needs to be exaggerated, lifelike and artistic.
Unlike the prior works for image-to-image translation dealing with low-level visual information conversion,
our study mainly focuses on the translation problems of
high-level visual information conversion, e.g., photo-tocaricature. For this purpose, our method differs from the
past work in network architecture as well as the layers and
losses. We design a dual pathway model of cGAN with two
discriminators to capture global structure and local statistics
respectively, and we propose a new parallel convolution to
construct Parallel CNNs for both discriminators. For generator, we provide three more extra losses in association with
adversarial loss to constrain the consistency. Here we show
that our approach is effective on the face photo-to-caricature
translation task, which typically requires high-level visual
information conversion.

1 We consider that anime, emoji, and caricature are three types of cartoon or three subsets of cartoon.

real y
Global Discriminator
L gs

style
info

D

ParCNN

L l1

L g (G, D g )

Generator

z

g

real
?
fake

G

ParConv
U-Net
L (G, D )

input x
L cl

fake G (x, z)

L p (G, D p )

ParCNN

CNN
cascade
layer
feature maps
16 ×16

gradient
penalty

D

p

Patch Discriminator

fake
?
real

real y

Figure 2. Network architecture and data flow chart of our proposed method for face photo-to-caricature translation.

can be expressed as
Lc (G, D) = Ex,y∼Pdata (x,y) [log D(x, y)] +

Ex∼Pdata (x),z∼Pstyle (z) [log(1 − D(x, G(x, z)))] .
(1)

The goal is to learn a generator distribution over data y that
matches the real data distribution Pdata by transforming an
observed input image x ∼ Pdata (x) and a style info variable z ∼ Pstyle (z) into a sample G(x, z). This generator
is trained by playing against an adversarial discriminator
D that aims to distinguish between samples from the true
data distribution (real image pair {x, y}) and the generator’s distribution (synthesized image pair {x, G(x, z)}). To
deal with the high-level visual information conversion for
some more challenging image-to-image translation tasks,
we extend one discriminator to dual discriminators with our
proposed parallel CNNs and add three more losses to adversarial loss, so that our method can tackle conversions of
both the low-level (line, color, texture, etc.) and the highlevel (expression, exaggeration, artistry, etc.) visual information. The overall network architecture and data flow are
illustrated in Fig. 2.

3.1. Parallel convolution (ParConv)
The standard convolutional layer of CNNs [9] uses convolution operation as a filter to detect local conjunctions
of features from the previous layer and map their appearance to a feature map. So the convolutional layer captures

Conv
W
X

X

X⇤

X⇤

ParConv
W⇤

+
+

X
X⇤

Y

Figure 3. The illustration of our parallel convolution operation for
CNNs.

the local dependencies of the image (feature map) from the
previous layer. But this might lose the global structure relationships. To deal with this problem, we propose a new
parallel convolutional layer for CNNs (as shown in Fig. 3),
which uses convolution operation to detect local conjunctions of features from the previous layer and then capture
global connections of features from the current layer, so that
it could let the CNNs concern not only the local details but
also the global structure.
Our proposed ParConv function for the ith feature map

the loss of global discriminator Dg is

can be expressed as
Yi = f (

m
X

Wji Xj +

j=1

n
X

∗
Wki
Xk∗ + b),

(2)

k=1

(
Pm
Xk∗ = j=1 Wjk Xj ,
w.r.t.
∗
Wji 6= Wki
,
where m and n are the number of feature maps from the
previous layer and the current layer respectively; X represents the feature map of previous layer and X ∗ represents
the feature map of current layer convolved by feature maps
of previous layer, while W and W ∗ are their corresponding
weights; f (·) indicates the activation function and we use
Leaky ReLU [16] in our experiments.
We adopt our proposed ParConv to construct ParCNNs
for the dual discriminators of our model.

3.2. Patch discriminator
Following pix2pix [7], we also use patch discriminator as one pathway of our adversarial system. But we improve the original patch discriminator by using our ParCNNs (see Sec. 3.1) and adding gradient penalty [6] to the
loss. The loss of our patch discriminator Dp is
Lp (G, Dp ) = Lc (G, Dp ) + Lgp ,

(3)

where Lc (G, Dp ) comes from the objective loss Eq. 1 and
Lgp indicates the gradient penalty,
Lc (G, Dp ) = Ex,y∼Pdata (x,y) [log Dp (x, y)] +

Ex∼Pdata (x),z∼Pstyle (z) [log(1 − Dp (x, G(x, z)))] ,
(4)

and
Lgp



= λEx̂∼Px̂ (k∇x̂ Dp (x̂)k2 − 1)2 ,

x̂ = αG(x, z) + (1 − α)y,

(5)
(6)

where x̂ represents the mixture of the synthesized fake image G(x, z) and the target image y, ∇x̂ indicates the gradient of Dp (x̂), λ is the hyper parameter that controls how
much gradient we choose, and α is a random number between 0 and 1. We add this gradient penalty to improve the
training of GANs and the quality of synthesized samples.
We used greedy search to obtain the optimal value of λ as
λ = 1.0 for all the experiments in this paper, which we
found to work well across a variety of datasets.

3.3. Global discriminator
Besides the patch discriminator, we design a global discriminator as another pathway of our adversarial system for
high-level image-to-image translation. We also use our proposed ParCNNs (see Sec. 3.1) for this discriminator, and

Lg (G, Dg ) = Lc (G, Dg ) = Ex,y∼Pdata (x,y) [log Dg (x, y)] +

Ex∼Pdata (x),z∼Pstyle (z) [log(1 − Dg (x, G(x, z)))] .
(7)

This global discriminator mainly concerns the global structure information and provides the global perceptual similarity loss for generator (see Sec. 3.4).

3.4. Generator
Previous studies have found it beneficial to mix the GAN
objective with a more traditional loss, such as L2 [21] and
L1 [7] distance. We explore this opinion to provide three
more extra losses for the GAN objective, and our final objective is
G∗ = arg min maxL(G, D) + γLl1 + σLgs + ηLcl , (8)
G
D
(
Lp (G, Dp ) for Dp ,
w.r.t. L(G, D) =
(9)
Lg (G, Dg ) for Dg ,
where Ll1 means the real-fake L1 loss, Lgs represents the
global perceptual similarity loss, Lcl means cascade layer
loss, and γ, σ and η are hyper parameters to balance the
contribution of each loss to the objective. We used greedy
search to optimize the hyper parameters with γ = 50, σ =
10, and η = 5 for all the experiments in this paper.
Loss Ll1 is referred from pix2pix [7], which found
that L1 distance encourages less blurring than L2, and the
expression is
Ll1 = ky − G(x, z)k1 .

(10)

This loss can constrain the synthesized fake image G(x, z)
to be meaningful with the target image y (see Sec. 4.5 for
experiments).
Loss Lgs comes from our designed global discriminator
and can be expressed as
Lgs = kDg (y) − Dg (G(x, z))k1 .

(11)

This global perceptual similarity loss can constrain the synthesized fake image G(x, z) to be similar with the target
image y on perception (see Sec. 4.5 for experiments). This
is very important for image-to-image translation tasks with
high-level visual information conversion, because these
tasks usually require that the synthesized images have reasonable semantics.
Loss Lcl is inspired by the Cascaded Refinement Network (CRN) [2] with the following expression
n

Lcl = β

1X
kG(x, z)φi − yφi k1 ,
n i=1

(12)

where φ represents the output feature map of a trained visual
perception network and n is the number of feature maps, β
is a hyper parameter and we use β = 6.67 in our experiments. Like CRN, we use a pre-trained deep CNN VGG19 [26] to provide feature maps for Lcl loss. Unlike CRN
that uses both lower-layer and higher-layer activations, we
only use the higher-layer feature maps (with size 16 × 16)
for this loss. Layers in the network represent the increasing
levels of abstraction (from edges and colors to objects and
categories), therefore, for high-level visual information abstraction, the higher layers of a CNN could be more helpful,
and the experiments on Sec. 4.5 validate this conclusion.
As shown in Fig. 2, we use U-Net [24] with skip connections [7] as the generator to share the low-level and
high-level information between the input and output directly across the net. Besides, to synthesize images with
different styles, we use one-hot encoding to provide style
info vector z for style control.

4. Experiments
As a typical image-to-image translation task, phototo-caricature requires high-level visual information conversion, which is very challenging for the state-of-the-art
general-purpose solutions. To explore the effect of our proposed model, we tested the method on a variety of datasets
for translating faces in the wild from photo to caricature
with different styles, and qualitative results are shown in
Figs. 4, 10, and 11.

4.1. Dataset and training
Our proposed model is trained in a supervised fashion
on a paired face photo-caricature dataset, which was rebuilt based on IIIT-CFW [19]. The IIIT-CFW is a dataset
for the cartoon faces in the wild and contains 8928 annotated cartoon faces of famous personalities in the world with
varying profession. Also it provides 1000 real faces of the
public figure for cross modal retrieval tasks. However, it’s
not suitable for the training of photo-to-caricature translation task because the face photos and face cartoons2 are
not paired, e.g., the facial orientation and expression of the
photo and caricature for the same person are varying a lot.
So we rebuild a photo-caricature dataset with 390 paired
images by searching the IIIT-CFW dataset and internet as
the training set for our experiments. At inference time, we
run the generator net in exactly the same manner as during
the training phase. And we have extensively evaluated our
method on a variety of datasets with faces in the wild, including IIIT-CFW dataset [19], FEI dataset [28], ADFES
dataset [29], CelebA dataset [13], KDEF dataset [14], and
WSEFEP dataset [20].
2 We consider that caricature is a type of a cartoon or a subset of a
cartoon.

4.2. Comparison with state-of-the-arts
Using the IIIT-CFW dataset, we first compare our proposed method with Pix2pix [7], CycleGAN [36], DualGAN [32], and DiscoGAN [8] on photo-to-caricature translation task. All the five models were trained on the same
training dataset with 390 paired photo-caricature images
and tested on novel data from IIIT-CFW dataset that does
not overlap those for training. Fig. 4 shows the experimental results of the comparison.
From Fig. 4, it can be seen that, DiscoGAN loses the
structure information totally, Pix2pix makes structure error in almost all cases, while CycleGAN and DualGAN
can keep the structure information of the input image but
without enough conversion for caricature creation task. Although it’s still not good enough for the results of our
method compared to human caricaturists, the experiments
on photo-to-caricature translation of faces in the wild show
considerable performance gain of our proposed method
over state-of-the-art image-to-image translation methods,
especially the encouraging ability of exaggeration and abstraction. However, due to the very challenging task with
less training data but on various styles, our method also
messes some details while translation (see the mouth of the
first row and the left eye of the third row on our results in
Fig. 4).
This experiment also expresses that high-level imageto-image translation tasks like photo-to-caricature are generally more difficult than those low-level translation tasks
such as photo-to-sketch, because it not only needs to abstract the facial features, but also requires to exaggerate the
emotional expression. So that the pixel-level methods (like
Pix2pix) might fail as they force the generator to concentrate on local information rather than whole structure.

4.3. Dual discriminators
In this experiment, we verify the effectiveness of our proposed dual pathway of discriminators. We first use only one
patch discriminator (PatchD), and then use dual discriminators with one patch discriminator plus one global discriminator (PatchD+GlobalD), while keeping all other architecture and settings fixed for training and testing. The example results are shown in Fig. 5, one PatchD model almost
misses the key structure information on faces, but our dual
PatchD+GlobalD model can render the structure of facial
features well. It further proves that the PatchD model only
concerns the local statistics for tackling low-level image-toimage translation tasks (e.g., photo-to-sketch).

4.4. Parallel convolution evaluation
This section gives the comparison of results by using our
ParConv against the standard Conv for CNNs of dual discriminators. The experimental settings are similar with experiment in Sec. 4.3, and the example results are shown in

Input

Pix2pix

CycleGAN

DualGAN

DiscoGAN

Ours

Input

Conv

ParConv

Figure 6. Comparison of standard convolution (Conv) with our
parallel convolution (ParConv).

ParConv can deal with this situation owing to the convolution from both the previous layer and the current layer.
Figure 4. Comparison of state-of-the-art image-to-image translation methods with our proposed method for face photo-tocaricature translation on IIIT-CFW dataset.

Input

PatchD

PatchD+GlobalD

Figure 5. Comparison of only one discriminator (PatchD) with our
dual discriminators (PatchD+GlobalD).

Fig. 6. The cases shown in Fig. 6 illustrate that the standard
Conv only detect local conjunctions of features without caring global connections of features, so that Conv messes up
the facial features such as eyes, ears, nose, and mouth, while

4.5. Loss selection
As described in Sec. 3.4, we explore the objective loss
of generator by providing three more extra losses Ll1 , Lgs ,
and Lcl . So we designed three experiments to validate the
effectiveness of these three losses respectively.
We first consider to check if the extra loss should be provided to the GAN objective, and the second column in Fig.
7 shows the results without any extra loss. It’s easy to see
that without mixing any extra loss, the adversarial system
not only loses to capture the facial features but also falls
in mode collapse resulting in almost the same synthesized
results for different inputs. So it’s necessary to mix the extra loss for GANs dealing with image-to-image translation
tasks. The third and fourth column in Fig. 7 illustrate the
results by mixing L2 loss Ll2 and L1 loss Ll1 respectively.
It can be seen that L2 loss produces blurry results that are
hard to be recognized on the whole face as well as the facial features such as eyes, nose, and mouth. Although L1
loss makes some mistakes in details while translation, the
synthesized faces can still be identifiable.
Then we want to verify if the global perceptual similarity loss Lgs coming from our global discriminator is necessary for our adversarial system. Fig. 8 gives the example
translated results without and with Lgs under the same architecture and settings. The results on the second column,
indicating without Lgs , express some perceptual error on
some facial features such as head, eyes, and mouth. And
the results on the third column with our Lgs show better
performance.

Input

w/o L more

w/ L l2

w/ L l1

Input

w/o L cl

256+128+64+32+16

128+64+32+16

64+32+16

32+16

w/ L cl
different
cascade
layers
256

128

64

32

16

Figure 9. Comparison of extra losses for final objective of generator: without (w/o) Lcl , and with (w/) Lcl by mixing feature maps
from different cascade layers.

Figure 7. Comparison of extra losses for final objective of generator: without (w/o) Lmore , with (w/) Ll2 , and with (w/) our
Ll1 .

Input

w/o L gs

w/ L gs

Figure 10. Example results of face photo-to-caricature translation
with different styles by our method on FEI dataset.

the more meaningful the result is. It also concludes that the
higher-layer feature maps could be helpful for high-level
image-to-image translation.

4.6. Style control

Figure 8. Comparison of extra losses for final objective of generator: without (w/o) Lgs , and with (w/) our Lgs .

At last, we analyze the effect of cascade layer loss Lcl
with the different mixtures of output feature maps from different cascade layers, and Fig. 9 reports the compared results. The first row from column two to six shows the results without using Lcl and with different settings of different summing cascade layers respectively, in which the synthesized images are messed by summing more cascade layers, and the more cascade layers the method sums, the more
messes the result makes. The second row from column two
to six illustrates the results by using Lcl on a single layer,
and it is clear to see that, the higher layer the method uses,

The caricature has many different styles, such as sketch
and oil painting. So it would be useful if we can control
the style of translation. To achieve this goal, we divided
our paired photo-caricature training dataset into different
cartoon style categories, such as sketch, American comics,
simple line-drawing, and oil painting, etc., and we trained
our model with these categorized images by adding extra
one-shot vectors at the bottleneck of U-Net as the auxiliary label information for style control. Then we tested our
trained model on FEI dataset [28], and Fig. 10 shows the
example results of face photo-to-caricature translation with
different styles. It shows that column two expresses the
sketch style obviously, while the others can express different levels of exaggeration and artistry, e.g., the translated
mouth of the second person (row two). Furthermore, the
emotional expression of faces can also be translated and exaggerated by our method.

FEI
FEI
ADFES
CelebA
KDEF
WSEFEP
IIIT-CFW

Figure 11. Example paired inputs (lefts) and outputs (rights) on a variety of face datasets (FEI [28], ADFES [29], CelebA [13], KDEF [14],
WSEFEP [20], and IIIT-CFW [19]) for freestyle face caricature creation using our proposed method.

4.7. Freestyle face caricature creation
To evaluate the ability for real applications in the daily
life, we tested our method on a variety of face datasets
(FEI [28], ADFES [29], CelebA [13], KDEF [14], WSEFEP [20], and IIIT-CFW [19]) to illustrate the photo-tocaricature translation on faces in the wild, and the results are
shown in Fig. 11. These freestyle face caricature creation
results validate that our model works not bad on arbitrary
faces and show the potential value for the related applications.

5. Conclusion and Future Work
We present a novel GAN-based method to deal with
high-level image-to-image translation task, i.e., photo-tocaricature translation. The proposed method uses dual discriminators for capturing global structure and local statistics
information, and adopts our parallel convolution for ParCNNs of the dual discriminators for connecting the previous layer and the current layer to avoid structure error, also
mixes three more extra losses with the GAN objective to
constrain the consistency under exaggeration. Besides, the

style info vector can be used to control the translation for
different styles. Experimental results show that our method
not only outperforms other state-of-the-art image-to-image
translation methods, but also works well on a variety of
datasets for photo-to-caricature translation of faces in the
wild.
Limitations. Translating photo to caricature is a very
challenging high-level image-to-image translation task.
Thus, our model also fails in some cases, e.g., the third row
in Fig. 4, Fig. 5, and Fig. 8. Although our method can keep
the structure information of faces, it is still hard to render
the details for providing high-quality caricatures. Besides,
it’s also sensitive to side face with complex background,
e.g., some cases on CelebA and KDEF datasets in Fig. 11.
Future work. With regard to future work, first, it would
be interesting to investigate our method on other tasks
of high-level image-to-image translation (e.g., human-tocartoon translation for cartoon movies). Second, in the proposed method, the model still needs to be improved to provided high-quality rendered translated results. Third, we
intend to apply our method on the real applications of automatic and intelligent photo-to-caricature translation.

References
[1] E. Akleman, J. Palmer, and R. Logan. Making extreme caricatures with a new interactive 2D deformation technique
with simplicial complexes. In Proceedings of Visual, pages
165–170, 2000. 1
[2] Q. Chen and V. Koltun. Photographic image synthesis with
cascaded refinement networks. In ICCV, 2017. 1, 4
[3] Y.-L. Chen, W.-H. Tsai, et al. Automatic generation of talking cartoon faces from image sequences. In Proceedings of
Conferences on Computer Vision, Graphics & Image Processing, 2004. 2
[4] H.-Y. Fish Tung, A. W. Harley, W. Seto, and K. Fragkiadaki.
Adversarial inverse graphics networks: Learning 2d-to-3d
lifting and image-to-image translation from unpaired supervision. In ICCV, 2017. 1
[5] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu,
D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In NIPS, 2014. 1, 2
[6] I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and
A. Courville. Improved training of wasserstein gans. arXiv
preprint arXiv:1704.00028, 2017. 4
[7] P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros. Image-to-image
translation with conditional adversarial networks. In CVPR,
2017. 1, 2, 4, 5
[8] T. Kim, M. Cha, H. Kim, J. Lee, and J. Kim. Learning to
discover cross-domain relations with generative adversarial
networks. arXiv preprint arXiv:1703.05192, 2017. 1, 2, 5
[9] A. Krizhevsky, I. Sutskever, and G. E. Hinton. ImageNet
classification with deep convolutional neural networks. In
NIPS, 2012. 3
[10] C. Ledig, L. Theis, F. Huszar, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani, J. Totz, Z. Wang, and
W. Shi. Photo-realistic single image super-resolution using a
generative adversarial network. In CVPR, 2017. 1, 2
[11] C. Li and M. Wand. Precomputed real-time texture synthesis
with markovian generative adversarial networks. In ECCV,
2016. 1, 2
[12] P.-Y. C. W.-H. Liao and T.-Y. Li. Automatic caricature generation by analyzing facial features. In ACCV, 2004. 2
[13] Z. Liu, P. Luo, X. Wang, and X. Tang. Deep learning face
attributes in the wild. In ICCV, 2015. 2, 5, 8
[14] D. Lundqvist, A. Flykt, and A. Öhman. The karolinska directed emotional faces (KDEF). Department of Clinical Neuroscience, Psychology section, Karolinska Institutet, 1998. 2,
5, 8
[15] W. C. Luo, P. C. Liu, and M. Ouhyoung. Exaggeration of facial features in caricaturing. In Proceedings of International
Computer Symposium, 2002. 2
[16] A. L. Maas, A. Y. Hannun, and A. Y. Ng. Rectifier nonlinearities improve neural network acoustic models. In ICML,
2013. 4
[17] M. Mathieu, C. Couprie, and Y. LeCun. Deep multi-scale
video prediction beyond mean square error. arXiv preprint
arXiv:1511.05440, 2015. 2
[18] M. Mirza and S. Osindero. Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784, 2014. 1, 2

[19] A. Mishra, S. N. Rai, A. Mishra, and C. Jawahar. IIIT-CFW:
A benchmark database of cartoon faces in the wild. In ECCVW, 2016. 2, 5, 8
[20] M. Olszanowski, G. Pochwatko, K. Kuklinski, M. SciborRylski, P. Lewinski, and R. K. Ohme. Warsaw Set of Emotional Facial Expression Pictures: a validation study of facial
display photographs. Frontiers in Psychology, 5:1516, 2015.
2, 5, 8
[21] D. Pathak, P. Krahenbuhl, J. Donahue, T. Darrell, and A. A.
Efros. Context encoders: Feature learning by inpainting. In
CVPR, 2016. 2, 4
[22] A. Radford, L. Metz, and S. Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In ICLR, 2016. 1
[23] S. Reed, Z. Akata, X. Yan, L. Logeswaran, B. Schiele, and
H. Lee. Generative adversarial text to image synthesis. In
ICML, 2016. 2
[24] O. Ronneberger, P. Fischer, and T. Brox. U-Net: Convolutional networks for biomedical image segmentation. In MICCAI, 2015. 5
[25] M. Sela, E. Richardson, and R. Kimmel. Unrestricted facial geometry reconstruction using image-to-image translation. In ICCV, 2017. 1
[26] K. Simonyan and A. Zisserman. Very deep convolutional
networks for large-scale image recognition. In ICLR, 2015.
5
[27] Y. Taigman, A. Polyak, and L. Wolf. Unsupervised crossdomain image generation. In ICLR, 2017. 1, 2
[28] C. E. Thomaz and G. A. Giraldi. A new ranking method
for principal components analysis and its application to face
image analysis. Image and Vision Computing, 28(6):902–
913, 2010. 2, 5, 7, 8
[29] J. Van Der Schalk, S. T. Hawk, A. H. Fischer, and B. Doosje.
Moving faces, looking places: validation of the Amsterdam Dynamic Facial Expression Set (ADFES). Emotion,
11(4):907, 2011. 2, 5, 8
[30] X. Wang and A. Gupta. Generative image modeling using
style and structure adversarial networks. In ECCV, 2016. 1,
2
[31] X. Yan, J. Yang, K. Sohn, and H. Lee. Attribute2image: Conditional image generation from visual attributes. In ECCV,
2016. 2
[32] Z. Yi, H. Zhang, P. Tan, and M. Gong. DualGAN: Unsupervised dual learning for image-to-image translation. In ICCV,
2017. 1, 2, 5
[33] H. Zhang, V. Sindagi, and V. M. Patel. Image de-raining
using a conditional generative adversarial network. arXiv
preprint arXiv:1701.05957, 2017. 2
[34] L. Zhang, Y. Ji, and X. Lin. Style transfer for anime sketches
with enhanced residual u-net and auxiliary classifier gan.
arXiv preprint arXiv:1706.03319, 2017. 2
[35] J.-Y. Zhu, P. Krähenbühl, E. Shechtman, and A. A. Efros.
Generative visual manipulation on the natural image manifold. In ECCV, 2016. 1
[36] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros. Unpaired imageto-image translation using cycle-consistent adversarial networks. In ICCV, 2017. 1, 2, 5

