Sketch-to-Image Generation Using Deep Contextual Completion
Yongyi Lu†

Shangzhe Wu†

arXiv:1711.08972v1 [cs.CV] 24 Nov 2017

†

Yu-Wing Tai‡

Chi-Keung Tang†

Hong Kong University of Science and Technology
‡
Tencent Youtu

Abstract

Input

When the input to pix2pix translation [9] is a badly
drawn sketch, the output follows the input edges due to the
strict alignment imposed by the translation process. In this
paper we propose sketch-to-image generation, where the
output edges do not necessarily follow the input edges. We
address the image generation problem using a novel joint
image completion approach, where the sketch provides the
image context for completing, or generating the output image. We train a deep generative model to learn the joint
distribution of sketch and the corresponding image by using joint images. Our deep contextual completion approach
has several advantages. First, the simple joint image representation allows for simple and effective definition of losses
in the same joint image-sketch space, which avoids complicated issues in cross-domain learning. Second, while
the output is related to its input overall, the generated features exhibit more freedom in appearance and do not strictly
align with the input features. Third, from the joint image’s
point of view, image and sketch are of no difference, thus
exactly the same deep joint image completion network can
be used for image-to-sketch generation. Experiments evaluated on three different datasets show that the proposed approach can generate more realistic images than the state-ofthe-arts on challenging inputs and generalize well on common categories.

CE

pix2pix

Ours

Figure 1. Freehand sketch to image results by CE [16], pix2pix [9]
and our method. Even the sketches are badly drawn they are still
expressive in conveying facial features and emotion. Our method
does not require strict alignment with while still faithful to the
input sketch, resulting in more realistic and natural images.

features) which may be clumsy to specify in verbose text
description.
In this paper, we propose a novel approach to address
the sketch-to-image generation problem. We pose the image generation problem as an image completion problem,
Figure 2. In conventional image completion, the corrupted
part of an input image is completed using image context.
In the face completion example, the unmasked, partial facial features are the contextual information. By analogy,
in our joint-image completion, the “corrupted” part consists
of the entire image to be generated, while the “context” is
provided by the input sketch for the “completion” or generation of the photographic object. Since deep neural network
is employed to learn joint images, we call our image generation approach deep contextual completion. This novel approach has several advantages. First, there are no separate
domains for image and sketch learning; only one network
is used to understand a joint sketch-image pair which is a
single image. This is in stark contrast with cross-domain
learning where images and sketches are treated individually
and learned in separate networks. In our approach, the network is trained end-to-end, and we can simply search for
an encoding of the provided “corrupted” image in the latent
space by using back propagation. No embedding or other
intermediate steps are required. Second, while related to

1. Introduction
Image-to-image translation generates impressive photographic images in a variety of applications demonstrated
in [9]. However the translation process makes the output
align with the input edges. This can be highly problematic
in sketch-to-image generation when the input is a free-hand
sketch. Figure 1 shows casual sketches of a frontal face,
where the free-hand sketch and its natural photographic object do not strictly align to each other.
Our goal of sketch-to-image generation is to automatically generate a photographic image of the sketched object. Even a poorly drawn sketch allows non-artists to easily
specify an object’s attributes in many situations (e.g., facial
1

Real

Mask

Corrupted data

Predict

Joint Image

Mask

Corrupted data

Predict

Inpainting

DCC
Figure 2. Image generation posed as image completion. Top: Semantic image inpainting with the uncropped part as context [33]. Bottom: Joint image completion with the sketch as the “uncropped” context, with the cropped part on the right. Our joint image concatenates
a sketch and its corresponding image side-by-side.

its input edges, the generated image may exhibit different
poses and shapes beyond the input sketches which may not
correspond well to photographic objects. Third, from the
joint image’s point of view, image and sketch are of no difference, so they can be swapped to provide the context for
completion for the other. Thus, exactly the same sketchto-image generation approach/network can be used for the
reverse or image-to-sketch generation.
Our proposed framework is generic which can employ
any state-of-the-art generative model. Capitalizing on the
DCGAN for image completion [33], in this paper, we propose a two-phase recipe to learn the sketch-image correspondence inherent in a joint image. For training, the
network learns the sketch-image correspondence using uncropped joint images. In completion, we search for an
encoding of the provided corrupted image using only the
sketch to provide the context for “completing” the image. This encoding is then used to reconstruct the image
by feeding it to the generator which generates the photographic object from sketch. We evaluate our method
on three datasets: CelebA [15], Stanford Cars [13] and
Caltech-UCSD Birds [24]. Experimental results show that
our approach can generate more realistic and natural images
than the state-of-the-arts on challenging inputs, e.g. poorly
drawn sketches shown in Figure 1, while producing comparable results to state-of-the-arts on good sketches where the
edges correspond to photographic objects.

generating impressive results. We note that a simple or even
poorly drawn sketch can easily convey complex visual attributes which may require detailed textual descriptions to
specify.
Related to our deep contextual completion for joint images is deep image completion with perceptual and contextual losses [33]. Pretrained with uncorrupted data, the G
and D networks are trained to reconstruct a complete image.
Their impressive examples show that even a large region of
a facial image is cropped from the input, the generated complete facial image looks very realistic. Another impressive
work on image completion [16] is based on autoencoders
with a standard reconstruction loss and an adversarial loss.
Autoencoders have also been successfully applied to generating images from visual attributes [32]. Analogous to
image completion, where the uncropped part of the image
provides the proper context for facial image completion, in
our sketch-to-image generation, the entire input sketch is regarded as the “uncropped context” for completing the entire
natural image part of the joint image.
In recent works, the mapping between sketches and images was studied in sketch-based image retrieval, where the
sketch and the image were learned in separate networks.
In [4], several triplet CNNs were evaluated for measuring the similarity between sketches and photos. Triplet
networks are used to learn joint embeddings. In [35],
a database of sketch-photo pairs was collected, and deep
learning was used to learn a joint embedding. The Sketchy
Database [19] contributed a collection of sketch-photo pairs
which were used to train a cross-domain CNN to embed
them in the same space. While classical representations
were proposed to retrieve images from sketch queries [11, 3,
20], recent methods used sophisticated feature representations [6, 21, 5, 18]. Recent cross-domain embedding methods trained deep networks to learn a common feature space
for sketches and 3D models [25], and images and 3D models [14]. Siamese networks trained with contrastive loss [7]
and triplet or ranking loss [26] were proposed. On the other
hand, we do not regard sketch and image as two separate
domains, as they form the same joint image. This allows
for an effective encoding for the “corrupted” joint image in

2. Related Work
The rapid development of deep learning has accounted
for recent exciting progress in image generation, especially the introduction of generative adversarial networks
(GAN) [8]. GAN trains two networks, a generative model
G and a discriminative model D. When G and D are
deep convolutional networks, we have DCGAN [17]. Upon
convergence, D can reject images that looks fake and G
can produce high-quality images indistinguishable from the
training images. Conditioning variables were then introduced to GAN. Conditional PixelCNN [23] used text descriptions and object locations to generate images. Conditioned on style and structure, S 2 -GAN [27] is capable of
2

(a) Refined initialization.

(b) Completion pipeline.

Figure 3. Deep contextual completion pipeline. (a) Initialization: red box represents better initialization based on KL-divergence. (b) Given
the initialization B, we use back propagation on the loss defined in Eq. (5) to map the corrupted image A to the latent space. The mapped
vector is then passed through the G network to generate the missing image C.

pled from a prior distribution pz and the real data x from
the pdata distribution, the objective of a GAN can be formulated as:

the latent space and leads to stable training.

3. Approach
This section presents our technical approach for generating images from sketches under the deep contextual completion framework. First, we introduce the input space for
the joint sketch-image pairs (section 3.1). After a review of
GAN, the building block for image generation (section 3.2),
we proceed to present our deep contextual completion (section 3.3), where learning sketch-image correspondence requires the definition of our objective loss and two-stage
strategy.

LGAN (G, D) =Ex∼pdata (x) [log D(x)]+
Ez∼pz (z) [log(1 − D(G(z)))]

(1)

where a discriminator D is trained to maximize this objective against an adversarial generator G that is trained to minimize it, i.e. a two-player min-max game:
G∗ = arg min max LGAN (G, D)
G

3.1. Joint Sketch-Image Representation

D

(2)

Upon convergence, G manages to produce an image similar to one drawn from pdata distribution from the latent
space. Our deep model is built on the DCGAN architecture
proposed in [17] with the following technical revisions for
deep contextual completion.

Cross-domain image generation is a nontrivial task, especially for the sketch inversion or sketch-to-image generation problem we are addressing in this paper, since sketches
are often highly abstract with sparse visual content, and they
are sometimes badly drawn. Rather than following traditional ways of separating sketch and image into two different network branches, we propose to model them in a joint
input space. Specifically, based on a corpus containing samples with real images (B) and their aligned sketch styles (A),
we place them into a joint sketch-image pair (AB), as shown
in Figure 2 which is effective for learning their joint distribution and latent correspondence. This avoids the complications in training separate networks and then bridging the
two domains in some form of embedding [35, 4].
We require that the model to be sufficiently powerful
to represent the complex nonlinear relationships between
sketches and images and allows fast inference. We achieve
both of these goals by using generative adversarial networks
under an image completion framework.

3.3. Deep Contextual Completion
We do not treat sketch and image separately in the generative model. Recent work in semantic inpainting regards
inpainting as a constrained image generation problem [33],
where the generated content is supposed to be well aligned
with the surrounding pixels while maintaining semantic reality based on the observed context. Analogously, in our
framework, the joint image naturally captures the contextual
information, i.e., the correspondence between the sketch
and image portions.
Driven by these factors, we train G and D networks with
the complete joint images; the network then automatically
predicts the corrupted image portion based on the context of
the corresponding sketch portion. The G embeds the joint
images from the pdata distribution onto a non-linear manifold z, so that the closest mapping can be applied from
the held-out test image (i.e., only the sketch portion is observed) denoted by y to the reconstructed joint image G(z).

3.2. Generative Adversarial Networks
Here we give a concise review of the Generative Adversarial Networks (GANs) [8]. Given a random vector z sam3

Figure 3 illustrates this mapping which will be detailed in
section 3.3.2. Here we first introduce the objective functions of our proposed framework.
3.3.1

from the original DCGAN, readers may have already noticed that our generative network generates images of higher
perceptual quality compared to the results reported by DCGAN. The reason is that the corresponding sketches provide
higher level of structural information, which successfully
rejects a lot of “outliers” that only look plausible locally
while the global structure of generated objects is violated.

Objective Function

To get the closest mapping of the corrupted joint image and
the reconstructed joint image, we need to search for a generated joint image G(ẑ) in which the sketch portion best
resembles the input sketch.
Given the randomly sampled input z ∼ pz , we define our
loss function to comprise of two losses in our objective:

Encoding. Figure 3 illustrates the overall pipeline of mapping in this stage. In completion, our goal is to encode the
corrupted joint image input (i.e., the image portion of the
testing joint image that has been masked out) to the closest image on the manifold of G in the latent space, so that
we can use this closest joint image as our predicted results.
Deepdream proposes to update input y based on a particular
layer’s output using back-propagation. As stated in [33], in
the completion scenario where the corrupted input is not
drawn from pdata manifold, or G manifold, maximizing
D(y) may not lead to the expected reconstruction results.

Contextual Loss We use the contextual loss in [17] to
measure the context similarity between the uncorrupted portions, i.e., the input sketch portion and the reconstructed
sketch, which is defined as:
Lcontextual (z) = DKL (M

y, M

G(z))

(3)

Projection through back propagation. Instead of maximizing D(y), we compute the ẑ vector that minimizes our
objective function in Eq. (5). This means that we are projecting the corrupted input onto the z space of the generator
through the iterative back propagation. Specifically, the input is a vector z initialized with uniformly random noise,
and a joint image with only the sketch on the left with the
image on the right being masked out. We back propagate
the loss in Eq. (5) to update the randomly sampled input z of
network G. Note that in this stage only the input vector z is
updated using gradient descent, the weights of the network
G and D remain unchanged. Figure 4 provides visualization of traversing the latent space during back-propagation
(with four iterations as shown). Note that [33] also adopts
similar gradient descent optimization on inverse mapping.
After back-propagation, the corrupted input y 0 s closest
mapping vector ẑ in the latent space is then passed through
G network to generate G(ẑ). The resulting image fills in
the missing values of y (the image portion) with G(ẑ):

where M is the binary mask of the corrupted joint image and
denotes the Hadamard production. Different from [17],
since a sketch is a binary image rather than a natural image, we use the KL-divergence to measure the similarity
between the distribution of two sketches which tends to produce sharper images. Note the asymmetry of the two compared sketches using KL-divergence. Ideally, all the pixels
at the sketch portions are the same between y and G(z), i.e.,
Lcontextual (z) = 0, and we penalize G(z) for not generating a sketch similar to the observed input sketch y.
Perceptual Loss The perceptual loss maintains the semantic content of the predicted image, which is defined using the adversarial loss of the G network:
Lperceptual (z) = log(1 − D(G(z)))

(4)

The objective function for ẑ is then the weighted sum of
the two different losses:
ẑ = arg min(Lcontextual (z) + λLperceptual (z))
z

(5)

xgenerated = M

where λ is a hyperparameter to constrain the generated image with the input. A small λ will guarantee similar appearance of the input and output.
3.3.2

y + (1 − M)

G(ẑ)

(6)

Initialization. We use a uniformly sampled noise vector as
input. An obvious problem is that the generated image is
affected by the initialization. If the initialized sketch portion of G(z) perceptually exhibits a large gap from the input sketch, it will be hard for the corrupted image to be
mapped to the closest z in the latent space with gradient descent. This will result in failure samples even if we set a
very small λ in Eq. (5). To address this problem, we refine the initialization as follows: we sample N uniformly
random noise vectors, and obtain their respective initialized
sketches via the forward pass. Then we compute the pairwise KL-divergence between the input sketch and these N
initialized sketches. The one which gives the lowest KLdivergence represents the best initialization among the N

Joint Image Completion

Our deep contextual completion network consists of the
training stage and completion stage. The training stage is
the same as the traditional unconditional GAN training except that our training samples are joint images. We modify the traditional DCGAN network for better training results (next subsection). After training, we learn a generative network G that achieves the objective of reproducing
the joint image data distribution, i.e., mapping samples from
noise distribution pz to the data distribution pdata . Different
4

Generator:
512

128

256

64

3

z

G(z)

4x8

8x16

Discriminator:
1

fake
real

512

4x8

256

128

8x16

16x32
64

16x32

32x64

64x128

3

32x64

y
64x128

Figure 5. The G and D network architecture for deep contextual
completion.
Figure 4. Manifold traversing (with four iterations as shown) when
updating latent vector z using back-propagation. z(0) is random
noise picked via our initialization scheme; z(k) denotes the k-th
iteration; and ẑ the final solution.

we use the XDoG edge detector proposed in [29], and the
Photocopy effect [1] in Photoshop to produce two styles of
sketches. To better resemble hand-drawn sketches, we simplified the edge images using [22].
We generate 3 sketch styles in total. We split the data in
each style into a training set and a testing set. Pretrained
on the sketches of the XDoG style, the network is finetuned using sketches of other style sequentially, i.e., photocopy style, then the simplification style. We use only the
XDoG style in the first pretraining because XDoG are more
similar to the original photographic images compared with
other styles. In this way, we can guarantee the network is
first trained with a good local minima before augmenting
the network with other sketch styles. We show in the experimental session that the augmenting styles help generalize the sketch-image correspondence better, while allowing
some degree of freedom in appearance.

samples and will be used as the initialized sketch. This
process is illustrated in Figure 3. We set N = 10 in our
implementation.
3.3.3

Network Architecture

We present the network architecture to implement our deep
contextual completion. Figure 5 shows the complete network.
Generator. Following the architecture of DCGAN [17], a
100-D random noise vector, uniformly sampled from −1 to
1, is fed into the generator network G. Then, a 8192 × 2
linear layer reshapes the input to 4 × 8 × 512. We use
five up-convolutional layers with kernel size 5 and stride
2. We also have a batch normalization layer after each
up-convolutional layer except the last one to accelerate the
training and stabilize the learning. The leaky rectified linear
unit (LReLU) activation is used in all layers. Finally, tanh
is applied in the output layer. This series of up-convolutions
and non-linearities conduct a non-linear weighted upsampling of the latent space, and generates a higher resolution
image of 64 × 128.
Discriminator. For the discriminator, the input layer is an
image of dimension 64×128×3, followed by four convolutional layers where the feature map’s dimension is halved,
and the number of channels is doubled from the previous
layer. Specifically, we add four convolutional layers with
kernel size 5 and stride 2 to produce a 4 × 8 × 512 output.
We then add a fully connected layer to reshape the output
to one dimension, followed by a two class softmax layer for
computing loss.

4. Datasets and Implementation
In this section we describe the datasets used in training
and testing and implementation issues.
We tested our network using 3 categories of images:
face, car, and bird. Since the available sketch datasets are
very limited, we applied several tools to produce sketches
from images for training. We obtained the raw face and
bird images from the Large-scale CelebFaces Attributes
(CelebA) dataset [15], the UT-Zap50K [34] dataset, and
the Caltech-UCSD Birds-200-2011 [24] dataset. Figures 6
show some examples of our training data.

4.1. Data Preprocessing
For the face category, the CelebA dataset contains around
200K images. We cropped and aligned the face region using OpenFace dlib [2]. We obtained 400K images with 2
different landmark maps for alignment. We generate the
above three styles of sketches respectively and finally, we
obtained 1200K face sketch-image pairs.
For the bird category, the CUB-200-2011 dataset contains only 11.7K raw images. We first produced three styles
of sketch using the above methods. To remove background
as much as possible, we cropped the image and correspond-

3.4. Network Generalization
Real free-hand sketches exhibit a large variety of styles
and it may be very different from synthesis sketches automatically generated from images. To improve the network
generality and to avoid overfitting to some particular style of
sketch-image pairs, we augment our training data by using
multiple styles of sketches as the training set. Specifically,
5

XDoG

Figure 6. Training data preprocessing for different datasets. The
leftmost is the original image, and the others are generated
sketches.

PC

ing sketches based on the bounding box information provided in the dataset. To augment this dataset for training,
we randomly cropped 4 images per image, and flipped them
horizontally. Finally, we obtained around 100K bird sketchimage pairs.

Sim

Figure 7. Results on CelebA dataset in three sketch styles: XDoG,
Photocopy (PC) and simplified (Sim). Best viewed in color.

XDoG

4.2. Implementation
We pretrain the network for each category using DCGAN [17]. We use the Adam optimizer [12] with a learning
rate of 0.0002 and a beta of 0.5 for both the generator and
discriminator network. The network is trained with a batch
size of 64. The total number of epochs is 200, which takes 6
to 48 hours for training depending on the size of the training
set. After obtaining a well-trained model of XDoG style, we
then finetune it using other styles of sketches sequentially
at a lower learning rate (e.g., 1e−5 ) using the same network
structure to obtain other styles’ models.
The input z is updated during completion, using a contextual loss and a perceptual loss with a λ of 0.01 and a
momentum of 0.9. Stochastic clipping is applied during
back-propagation. We set a relatively small λ so contextual loss is more important in test-time optimization so that
the sketch portion in generated image best resembles the input sketch. The generator and the discriminator are fixed
during back-propagation. For the experimental results, this
update can be done in 1000 iterations (In our case the loss
converges very fast, typically becomes stable after 200 iterations). We use the same network architecture for all of the
three categories. All model hyperparameters are optimized
for best qualitative results on held out validation set.

PC

Sim

Figure 8. Results on CUB dataset in three sketch styles: XDoG,
Photocopy (PC) and simplified (Sim). Best viewed in color.

Figure 9. Results on Car dataset in FDoG style.

ally we can generate images using arbitrary styles of sketch
providing that the pretrained model learns semantically correct content as well as faithful correspondence. We will test
using hand-drawn sketches never seen by the network.
5.1.1

CelebA

Figure 7 shows some of our results on the CelebA dataset
with three different styles, which demonstrates that our
method can successfully predict or “complete” the missing image part given the uncorrupted content, and generate a high-quality image that corresponds well to the given
sketch. Note that the generated results not only capture the
important details from and thus remains faithful to the input sketch, but also exhibit some degree of freedom in the
appearance, in comparison to state-of-the-art image generation methods such as pixel-to-pixel approach [9] where the
generated results conform strictly to the input sketch’s edge
profile.

5. Results
In this section we will present our experimental results
and comparison on the benchmark datasets described above.
We also test our deep contextual completion using in our
opinion quite ugly hand-drawn sketches. These hand-drawn
sketches are never presented in the training examples. Then,
we present latent space interpolation, followed by result on
forward generation (i.e., sketch synthesis from photos) and
finally we will present failure cases. More results are in
supplemental material.

5.1.2

CUB

We further validate the robustness of the proposed framework using the CUB bird dataset. Compared to face and
car, the CUB bird images contain much more texture information (e.g., the feather) which makes learning sketches
as well as the correspondence more difficult. To get rid of
the negative effect, we adopt relative total variation smoothing [31] to preprocess the sketches which was designed to
separate structure from texture. Then, we combine them
with the original images to form our joint images. Results
are shown in Figure 8.

5.1. Benchmark Datasets
As stated above, we first train a base model using the
network we described above on one style, i.e., the XDOG
style, and then finetune it on other styles: the photocopy effect and simplification style. This strategy is more effective
compared to training with all different styles together. Ide6

input

CE

pix2pix

Ours

input

CE

pix2pix

Ours

input

CE

pix2pix

Ours

Figure 10. Comparison with CE [16] and pix2pix [9] using ugly/poorly drawn sketches on three different classes. For each group, from
left to right: input sketches, CE results, pix2pix results and our results. Our method is resilient to corrupted/abstract inputs given in bad
quality. Best viewed in color.

5.1.3 Car
To demonstrate that our framework is generic and can be
applied to other categories of images, we tested it on car
images. We obtained the 16K car images from Stanford’s
Cars Dataset [13]. We generated sketches using FDoG filter proposed in [10]. Figure 9 shows some results using our
approach. Note that the first two input sketches are identical. However, our network is able to generate two different
images, i.e., cars with different shapes and colors while the
input sketch still constrains the generated car features but
not requiring them to be in strict alignment with the sketch.

put falls far apart from the learned data distribution and has
a high adversarial loss (in our case, the perceptual loss). CE
is even worse when it comes to the car dataset. By contrast,
our results show that the proposed framework is resilient
to corrupted inputs given in bad quality, where we manage
to map the input to the closest z in the latent space and use
this closest vector to generate images that reflect the semantics of the input sketch while looking natural and realistic.
Note that pix2pix produces deterministic outputs with little
stochasticity, while our method is able to produce stochastic
output by updating the manifold z, which is likely to capture
a fuller spectrum of the data distribution. Figure 4 gives an
evidence by providing visualization of traversing the latent
space during back-propagation (detailed in section 3.3.2).

5.1.4 Comparison on Hand-Drawn Sketches
We also investigated our model’s capability of generating
images from (ugly) free-hand sketches. We collect 50 freehand sketches for each of the 3 categories. Each of the
sketches is drawn given a random image. We compare our
results with Context Encoder (CE) [16] and Image-to-Image
Translation (pix2pix) [9]. Figure 10 demonstrates sample
results. For fair of comparison, all methods are tested with
the same style model (simplification for the face and bird
datasets, FDoG for the car dataset) using the default parameter settings, without finetuning on the hand-drawn data.
From Figure 10 we learn that when directly applied
to free-hand sketches without finetuning, pix2pix is unable to generate photo-realistic natural images given the
ugly/abstract sketch inputs. Though it can learn accurate
semantic contents, e.g., eyes, nose, mouth etc., it tends to
strictly follow the shape of the input sketch even if the out-

5.2. Quantitative Evaluation
While we clearly outperform CE [16] and pix2pix [9]
on badly drawn sketches, we further conducted two quantitative experiments on good sketches for sake of fairness,
where the edges correspond quite well to the corresponding photographic objects: (a) SSIM with ground truth; (b)
face verification accuracy. Both (a) and (b) are evaluated on
CelebA with 1000 test images.
SSIM: The Structural similarity metric (SSIM) [28] is used
to measure the similarity between generated image and
ground truth. Results are shown in Table 1. We achieved
comparable results to pix2pix on typical sketches, much
better than CE. Note that outputs of pix2pix and CE strictly
follow the input sketches and SSIM may fail to incorpo7

Method
SSIM

pix2pix [9]
0.9012

CE [16]
0.5477

Ours
0.8856

Table 1. SSIM results on CelebA test sets.

Method
Verification Acc.

pix2pix [9]
99.69

CE [16]
97.19

Ours
99.80

Table 2. Verification accuracy results on CelebA test sets.

rate measures of human perception if input sketch is badly
drawn.
Verification Accuracy: The motivation of this study is that
if the generated faces are plausible it should have the same
identity label with ground truth. The identity preserving
features were extracted using the pretrained Light CNN [30]
and compared using L2 norm. Table 2 tabulates the results:
we outperformed pix2pix, which shows that our model not
only learns to capture the important details but is more resilient to different sketches.

Figure 11. Reverse generation. Odd columns are input images,
while even columns are generated sketches.

(a)

(b)

(c)

Figure 12. Failure cases. (a) missing eye glasses; (b) missing
beard; (c) irregular shape of input face.

tor learns the natural appearance of faces, cars, and birds
such that any noise vector in the latent space would be able
to generate a visual plausible image. On testing, although
we have augmented the sketch drawing by three different
sketch styles, we should not restrict the human free hand
drawing to strictly follow the three augmented styles. This
is especially important for a practical sketch-to-image system. Because of this practical consideration, it is almost
impossible to train an end-to-end system which can convert any free hand drawing style into a meaningful image
since the number of free hand drawing style is unlimited.
In contrast, our two-stage approach seeks for the most similar sketch in the latent space during testing which is general
enough to handle many free hand drawing styles. Because
of the pretraining in the generator, our generated images
from sketches are guanranteed to be a natural image.

5.3. Bi-directional Generation
We also provide comparative results on forward generation, i.e., synthesizing a sketch from an image by corrupting
the sketch portion of the joint image. As we learn a joint
distribution of the sketch and image, there is no difference
whether we generate image from sketch, or sketch from image under our deep contextual completion framework.
We adopt the same network architecture (see Sec. 3.3.3)
and implementation (see Sec. 4.2) as in sketch-to-image
scenario, except that we swap the role of sketch and image
as the held-out portion in training and testing. Figure 11
shows some convincing results on generating a sketch from
a photographic image, which also demonstrates that our
model can learn faithful correspondence between a sketch
and its corresponding image.

6. Conclusion and Future Work

5.4. Failure Case Study

We show that the problem of sketch-to-image generation can be formulated as the joint image completion problem with the sketch providing the context for completion.
Based on this novel idea we propose a new and simple deep
contextual completion framework. A generative adversarial
network is trained to learn the joint distribution and capture
the inherent correspondence between a sketch and its corresponding image, thus bypassing the cross-domain learning issues. This approach encodes the “corrupted” joint image into the closest “uncorrupted” joint image in the latent
space, which can be used to predict and hence generate the
output image part of the joint image. We conduct thorough
experiments to demonstrate the advantages of the proposed
framework. In the future we plan to investigate more powerful generative models and explore more application scenarios. While our output is faithful to the input sketch, new
quantitative measurement may be needed to measure the
“perceptual” correspondence between a (badly drawn) input sketch and our generated image (e.g. Figure 1), a subject
and difficult problem in its own right.

Though our deep contextual completion is resilient to
ugly/abstract sketches and can realistically generate images which exhibit more freedom in appearance, one potential limitation is that it may fail to identify some kinds
of attributes associated with the input sketch. Figure 12(a)
and Figure 12(b) visualize two failure cases of missing eye
glasses and beard for the outputs, while they perceptually
correspond to their inputs overall. We believe adding constraints like face attributes will better guide the generation
process. We mainly focus on our proposed framework and
leave it for future work. Figure 12(c) shows another failure
example with irregular shape of handdrawn face, which lies
outside the data subspace, making it hard to find the closest
mapping in the latent space.

5.5. Discussions
Compare with the end-to-end methods, our approach is a
two-stage method which requires longer inferring time during testing. However, the two-stage approach allows us to
separate the training and testing. In training, our genera8

References

[19] P. Sangkloy, N. Burnell, C. Ham, and J. Hays. The sketchy
database: Learning to retrieve badly drawn bunnies. ACM
TOG, 2016. 2
[20] S. Sclaroff. Deformable prototypes for encoding shape categories in image databases. PR, 30(4):627–641, 1997. 2
[21] A. Shrivastava, T. Malisiewicz, A. Gupta, and A. A. Efros.
Data-driven visual similarity for cross-domain image matching. ACM TOG, 30(6), 2011. 2
[22] E. Simo-Serra, S. Iizuka, K. Sasaki, and H. Ishikawa. Learning to Simplify: Fully Convolutional Networks for Rough
Sketch Cleanup. ACM TOG, 35(4), 2016. 5
[23] A. van den Oord, N. Kalchbrenner, O. Vinyals, L. Espeholt,
A. Graves, and K. Kavukcuoglu. Conditional image generation with pixelcnn decoders. CoRR, abs/1606.05328, 2016.
2
[24] C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie.
The Caltech-UCSD Birds-200-2011 Dataset. Technical Report CNS-TR-2011-001, California Institute of Technology,
2011. 2, 5
[25] F. Wang, L. Kang, and Y. Li. Sketch-based 3d shape
retrieval using convolutional neural networks.
CoRR,
abs/1504.03504, 2015. 2
[26] J. Wang, Y. Song, T. Leung, C. Rosenberg, J. Wang,
J. Philbin, B. Chen, and Y. Wu. Learning fine-grained image
similarity with deep ranking. In CVPR, pages 1386–1393,
2014. 2
[27] X. Wang and A. Gupta. Generative image modeling using
style and structure adversarial networks. In ECCV, 2016. 2
[28] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli. Image quality assessment: from error visibility to
structural similarity. IEEE transactions on image processing, 13(4):600–612, 2004. 7
[29] H. Winnemller, J. E. Kyprianidis, and S. C. Olsen. Xdog: An
extended difference-of-gaussians compendium including advanced image stylization. Computers & Graphics, 36(6):740
– 753, 2012. 5
[30] X. Wu, R. He, Z. Sun, and T. Tan. A light cnn for
deep face representation with noisy labels. arXiv preprint
arXiv:1511.02683, 2015. 8
[31] L. Xu, Q. Yan, Y. Xia, and J. Jia. Structure extraction from
texture via natural variation measure. ACM TOG, 2012. 6
[32] X. Yan, J. Yang, K. Sohn, and H. Lee. Attribute2image:
Conditional image generation from visual attributes. CoRR,
abs/1512.00570, 2015. 2
[33] R. A. Yeh, C. Chen, T. Y. Lim, A. G. Schwing, M. HasegawaJohnson, and M. N. Do. Semantic image inpainting with
deep generative models. In CVPR, 2017. 2, 3, 4
[34] A. Yu and K. Grauman. Fine-Grained Visual Comparisons
with Local Learning. In IEEE CVPR, June 2014. 5
[35] Q. Yu, F. Liu, Y.-Z. SonG, T. Xiang, T. Hospedales, and C. C.
Loy. Sketch me that shoe. In CVPR, 2016. 2, 3

[1] Create filter gallery photocopy effect with single step in
photoshop. https://www.youtube.com/watch?v=
QNmniB_5Nz0. 5
[2] B. Amos, B. Ludwiczuk, and M. Satyanarayanan. Openface:
A general-purpose face recognition library with mobile applications. Technical report, CMU-CS-16-118, 2016. 5
[3] A. D. Bimbo and P. Pala. Visual image retrieval by elastic
matching of user sketches. IEEE Trans. Pattern Anal. Mach.
Intell., 19(2):121–132, 1997. 2
[4] T. Bui, L. Ribeiro, M. Ponti, and J. P. Collomosse. Generalisation and sharing in triplet convnets for sketch based visual
search. CoRR, abs/1611.05301, 2016. 2, 3
[5] X. Cao, H. Zhang, S. Liu, X. Guo, and L. Lin. Sym-fish:
A symmetry-aware flip invariant sketch histogram shape descriptor. In IEEE ICCV, December 2013. 2
[6] Y. Cao, C. Wang, L. Zhang, and L. Zhang. Edgel index for
large-scale sketch-based image search. In IEEE CVPR, pages
761–768, 2011. 2
[7] S. Chopra, R. Hadsell, and Y. LeCun. Learning a similarity
metric discriminatively, with application to face verification.
In CVPR, volume 1, pages 539–546 vol. 1, June 2005. 2
[8] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu,
D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In NIPS, pages 2672–2680. 2014. 2,
3
[9] P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros. Imageto-image translation with conditional adversarial networks.
arXiv preprint arXiv:1611.07004, 2016. 1, 6, 7, 8
[10] H. Kang, S. Lee, and C. K. Chui. Coherent line drawing.
In ACM Symposium on Non-Photorealistic Animation and
Rendering (NPAR), pages 43–50, Aug. 2007. 7
[11] T. Kato, T. Kurita, N. Otsu, and K. Hirata. A sketch retrieval
method for full color image database-query by visual example. In ICPR, pages 530–533, Aug 1992. 2
[12] D. P. Kingma and J. Ba. Adam: A method for stochastic
optimization. CoRR, abs/1412.6980, 2014. 6
[13] J. Krause, M. Stark, J. Deng, and L. Fei-Fei. 3d object representations for fine-grained categorization. In 2013 IEEE
International Conference on Computer Vision Workshops,
pages 554–561, Dec 2013. 2, 7
[14] Y. Li, H. Su, C. R. Qi, N. Fish, D. Cohen-Or, and L. J.
Guibas. Joint embeddings of shapes and images via cnn image purification. ACM TOG, 34(6):234:1–234:12, 2015. 2
[15] Z. Liu, P. Luo, X. Wang, and X. Tang. Deep learning face
attributes in the wild. CoRR, abs/1411.7766, 2014. 2, 5
[16] D. Pathak, P. Krähenbühl, J. Donahue, T. Darrell, and
A. Efros. Context encoders: Feature learning by inpainting.
In CVPR, 2016. 1, 2, 7, 8
[17] A. Radford, L. Metz, and S. Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. CoRR, abs/1511.06434, 2015. 2, 3, 4, 5,
6
[18] J. M. Saavedra and J. M. Barrios. Sketch based image
retrieval using learned keyshapes (lks). In BMVC, pages
164.1–164.11. BMVA Press, September 2015. 2

9

