# embedding_metadata

key: chroma:document
string_value:
High-Performance Neural Networks
for Visual Object Classication
Dan C. Cire san, Ueli Meier, Jonathan Masci,
Luca M. Gambardella and J urgen Schmidhuber
Technical Report No. IDSIA-01-11
January 2011
IDSIA / USI-SUPSI
Dalle Molle Institute for Articial Intelligence
Galleria 2, 6928 Manno, Switzerland
IDSIA is a joint institute of both University of Lugano (USI) and University of Applied Sciences of Southern Switzerland (SUPSI),
and was founded in 1988 by the Dalle Molle Foundation which promoted quality of life.
This work was partially supported by the Swiss Commission for Technology and Innovation (CTI), Project n. 9688.1 IFF:
Intelligent Fill in Form.arXiv:1102.0183v1  [cs.AI]  1 Feb 2011
Technical Report No. IDSIA-01-11 1
High-Performance Neural Networks
for Visual Object Classication
Dan C. Cire san, Ueli Meier, Jonathan Masci,
Luca M. Gambardella and J urgen Schmidhuber
January 2011
Abstract
We present a fast, fully parameterizable GPU implementation of Convolutional Neural
Network variants. Our feature extractors are neither carefully designed nor pre-wired, but


key: chroma:document
string_value:
January 2011
Abstract
We present a fast, fully parameterizable GPU implementation of Convolutional Neural
Network variants. Our feature extractors are neither carefully designed nor pre-wired, but
rather learned in a supervised way. Our deep hierarchical architectures achieve the best
published results on benchmarks for object classication (NORB, CIFAR10) and handwritten
digit recognition (MNIST), with error rates of 2.53%, 19.51%, 0.35%, respectively. Deep
nets trained by simple back-propagation perform better than more shallow ones. Learning
is surprisingly rapid. NORB is completely trained within ve epochs. Test error rates on
MNIST drop to 2.42%, 0.97% and 0.48% after 1, 3 and 17 epochs, respectively.
1 Introduction
The human visual system eciently recognizes and localizes objects within cluttered scenes. For
articial systems, however, this is still dicult, due to viewpoint-dependent object variability,
and the high in-class variability of many object types. Deep hierarchical neural models roughly
mimick the nature of mammalian visual cortex, and by community consensus are among the most
promising architectures for such tasks. The most successful hierarchical object recognition systems
all extract localized features from input images, convolving image patches with lters. Filter


key: chroma:document
string_value:
promising architectures for such tasks. The most successful hierarchical object recognition systems
all extract localized features from input images, convolving image patches with lters. Filter
responses are then repeatedly sub-sampled and re-ltered, resulting in a deep feed-forward network
architecture whose output feature vectors are eventually classied. One of the rst hierarchical
neural systems was the Neocognitron (Fukushima, 1980) which inspired many of the more recent
variants.
Unsupervised learning methods applied to patches of natural images tend to produce localized
lters that resemble o-center-on-surround lters, orientation-sensitive bar detectors, Gabor lters
(Schmidhuber et al. , 1996; Olshausen and Field, 1997; Hoyer and Hyv arinen, 2000). These ndings
in conjunction with experimental studies of the visual cortex justify the use of such lters in the
so-called standard model for object recognition (Riesenhuber and Poggio, 1999; Serre et al. , 2007;
Mutch and Lowe, 2008), whose lters are xed, in contrast to those of Convolutional Neural


key: chroma:document
string_value:
Mutch and Lowe, 2008), whose lters are xed, in contrast to those of Convolutional Neural
Networks (CNNs) (LeCun et al. , 1998; Behnke, 2003; Simard et al. , 2003), whose weights (lters)
are randomly initialized and changed in a supervised way using back-propagation (BP).
Despite the hardware progress of the past decades, computational speed is still a limiting
factor for CNN architectures characterized by many building blocks typically set by trial and
error. To systematically test the impact of various architectures on classication performance,
we present a fast CNN implementation on Graphics Processing Units (GPUs). Previous GPU
implementations of CNNs (Chellapilla et al. , 2006; Uetz and Behnke, 2009) were hard-coded to
satisfy GPU hardware constraints, whereas our implementation is 
exible and fully online (i.e.,
Technical Report No. IDSIA-01-11 2
weight updates after each image). It allows for training large CNNs within days instead of months,
such that we can investigate the in
uence of various structural parameters by exploring large
parameter spaces (Pinto et al. , 2009) and performing error analysis on repeated experiments.
We evaluate various networks on the handwritten digit benchmark MNIST (LeCun et al. , 1998)


key: chroma:document
string_value:
We evaluate various networks on the handwritten digit benchmark MNIST (LeCun et al. , 1998)
and two image classication benchmarks: NORB (LeCun et al. , 2004) and CIFAR10 (Krizhevsky,
2009).
2 Convolutional neural networks
CNNs are hierarchical neural networks whose convolutional layers alternate with subsampling
layers, reminiscent of simple and complex cells in the primary visual cortex (Wiesel and Hubel,
1959). CNNs vary in how convolutional and subsampling layers are realized and how the nets
are trained. The CNN architecture considered in this study diers from the one of Simard et al.
(2003) in the sense that after each CNN-layer an optional max-pooling layer (Scherer et al. , 2010)
can be used. Here we give a complete description of this independent implementation (Fig. 1).
2.1 Image processing layer
The image processing layer is an optional pre-processing layer of predened lters that are kept
xed during training. Thus additional information besides the raw input image can be provided
to the network, such as edges and gradients. In particular, we nd that a contrast-extracting layer
(Fukushima, 2003) helps to improve the recognition rate for NORB.
2.2 Convolutional layer


key: chroma:document
string_value:
(Fukushima, 2003) helps to improve the recognition rate for NORB.
2.2 Convolutional layer
A convolutional layer is parametrized by the size and the number of the maps, kernel sizes, skipping
factors, and the connection table. Each layer has Mmaps of equal size ( Mx,My). A kernel (blue
rectangle in Fig 1) of size ( Kx,Ky) is shifted over the valid region of the input image (i.e. the
kernel has to be completely inside the image). The skipping factors SxandSydene how many
pixels the lter/kernel skips in x- and y-direction between subsequent convolutions. The size of
the output map is then dened as:
Mn
x=Mn1
xKn
x
Snx+ 1+ 1;Mn
y=Mn1
yKn
y
Sny+ 1+ 1 (1)
where index nindicates the layer. Each map in layer Lnis connected to at most Mn1maps in
layerLn1. Neurons of a given map share their weights but have dierent receptive elds.
2.3 Max-pooling layer
The biggest architectural dierence between our implementation and the CNN of LeCun et al.


key: chroma:document
string_value:
2.3 Max-pooling layer
The biggest architectural dierence between our implementation and the CNN of LeCun et al.
(1998) is the use of a max-pooling layer instead of a sub-sampling layer. No such layer is used
by Simard et al. (2003) who simply skips nearby pixels prior to convolution, instead of pooling
or averaging. Scherer et al. (2010) found that max-pooling can lead to faster convergence, select
superior invariant features, and improve generalization. The output of the max-pooling layer is
given by the maximum activation over non-overlapping rectangular regions of size ( Kx,Ky). Maxpooling enables position invariance over larger local regions and downsamples the input image by
a factor ofKxandKyalong each direction.
Technical Report No. IDSIA-01-11 3
2.4 Classication layer
Kernel sizes of convolutional lters and max-pooling rectangles as well as skipping factors are
chosen such that either the output maps of the last convolutional layer are downsampled to 1
pixel per map, or a fully connected layer combines the outputs of the topmost convolutional layer
into a 1D feature vector. The top layer is always fully connected, with one output unit per class
label.


key: chroma:document
string_value:
into a 1D feature vector. The top layer is always fully connected, with one output unit per class
label.
Figure 1: Architecture of a convolutional neural network. In this case, the convolutional layers
are fully connected. Both convolutional layers use a kernel of 5 x 5 and skipping factors of 1.
3 GPU implementation
The latest generation of NVIDIA GPUs, the 400 and 500 series (we use GTX 480 & GTX 580), has
many advantages over older GPUs, most notably the presence of a R/W L2 global cache for device
memory. This permits faster programs and simplies writing the code. In fact, the corresponding
transfer of complexity into hardware alleviates many software and optimization problems. Our
experiments show that the CNN program becomes 2-3 times faster just by switching from GTX
285 to GTX 480.
Technical Report No. IDSIA-01-11 4
Manual optimization of CUDA code is very time-consuming and error prone. We optimize for
the new architecture, relying on the L2 cache for many of the device memory accesses, instead of
manually writing code that uses textures and shared memory. Code obtained by this pragmatic
strategy is fast enough. We use the following types of optimization: pre-computed expressions,
unrolled loops within template kernels, strided matrices to obtain coalesced memory accesses and


key: chroma:document
string_value:
strategy is fast enough. We use the following types of optimization: pre-computed expressions,
unrolled loops within template kernels, strided matrices to obtain coalesced memory accesses and
registers wherever possible. Additional manual optimizations are possible in case future image
classication problems will require even more computing power.
3.1 Data structures
Both outputs yand deltasof layerLnare 2D strided. Their original size is MxMM y, but they
are horizontally strided with a pitch of 32 
oats (we use this stride for all 2D data), resulting in
coalesced memory accesses. The vertical stride avoids additional bounding tests in CUDA kernels.
All connections between maps of consecutive layers Ln1andLnare stored in matrix Cn.
Each row of Cncontains all connections that feed into a particular map in layer Ln. Because
we aim for a 
exible architecture with partially connected layers, in the rst column we store the
number of previous connections. This index is useful for Forward Propagation (FP) and Adjusting
Weights (AW) CUDA kernels. The second column stores the number of connections, followed by
corresponding indices of maps in Ln1connected to the current map.


key: chroma:document
string_value:
Weights (AW) CUDA kernels. The second column stores the number of connections, followed by
corresponding indices of maps in Ln1connected to the current map.
For BP and FP, analogous information about connections is needed. We therefore store backward connections in CBP. AW requires a list of all map connections (see Subsection 3.4), stored
as an array of map index pairs. Dealing with biases in BP kernel requires to know where the
weights of particular connections start; this information is stored in a 2D array WIDX BPof size
MnMn1.
3.2 Forward propagation
A straightforward way of parallelizing FP is to assign a thread block to each map that has to
be computed. For maps bigger than 1024 neurons, the job is further split into smaller blocks by
assigning a block to each line of the map, because the number of threads per block is limited (1024
for GTX 480). A one to one correspondence between threads and the map's neurons is assumed.
Because of weight sharing, threads inside a block can access data in parallel, in particular the
same weights and inputs from the previous layer. Each thread starts by initializing its sum with
the bias, then loops over all map connections, convolving the appropriate patch of the input map
with the corresponding kernel. The output is obtained by passing the sum through a scaled tanh


key: chroma:document
string_value:
the bias, then loops over all map connections, convolving the appropriate patch of the input map
with the corresponding kernel. The output is obtained by passing the sum through a scaled tanh
activation function, and then written to device memory.
3.3 Backward propagation
BP of deltas can be done in two ways: by pushing or by pulling. Pushing deltas means taking each
delta from the current layer and computing the corresponding deltas for the previous layer. For
an architecture with shared weights this has the disadvantage of being hard to code. Each delta
from the current layer contributes to many deltas in the previous layer, which translates into a lot
of programming. There are two ways of avoiding this: either writing partial deltas to a separated
block of memory and then putting everything together by calling another kernel (slow because
of a tremendous increase in the number of memory accesses, and the need of another kernel), or
using atomic writes (to avoid data hazards) to update deltas (very slow because many writings
are serialized). We implement pulling deltas, which has almost none of the above speed-limiting
drawbacks, but is a bit more complicated.
Technical Report No. IDSIA-01-11 5
The (uni- or bi-dimensional) thread grid assigns a (bi- or uni-dimensional) thread block to each


key: chroma:document
string_value:
Technical Report No. IDSIA-01-11 5
The (uni- or bi-dimensional) thread grid assigns a (bi- or uni-dimensional) thread block to each
map in the previous layer and a thread to each neuron in every map. Similar to FP, for maps
with more than 1024 neurons, the 2D grid is further split into smaller 1D blocks by assigning a
2D block to each row of the map. Each thread computes the delta of its corresponding neuron by
pulling deltas from the current layer. For every neuron in the previous layer we have to determine
the list of neurons in the current layer which are connected to it. Let us consider neuron ( i;j)
from a map in layer Ln1, and then assume that ( x;y) are the coordinates of neurons in maps
ofLnthat contribute to the delta of neuron ( i;j). The (x;y) neuron is connected to kernel size
number neurons ( KxKy) from each connected map in the previous layer. The indices in Ln1
of the neurons connected through a kernel to the ( x;y) neuron are:
x(Sx+ 1) ix(Sx+ 1) +Kx1;
y(Sy+ 1) jy(Sy+ 1) +Ky1:


key: chroma:document
string_value:
y(Sy+ 1) jy(Sy+ 1) +Ky1:
We can now compute the inequalities for ( x;y):
iKx+ 1
Sx+ 1xi
Sx+ 1;
jKy+ 1
Sy+ 1yj
Sy+ 1:
Because (x;y) has to be inside the map, the nal inequalities are:
maxiKx+ 1
Sx+ 1
;0
xmini
Sx+ 1
;Mx1
;
maxjKy+ 1
Sy+ 1
;0
yminj
Sy+ 1
;My1
:
The above inequalities state that the delta of neuron ( i;j) fromLn1is computed from deltas of
neurons in a rectangular area in maps of Ln(Fig. 2). After summing up the deltas, each thread
multiplies the result by the derivative of the activation function.
3.4 Adjusting weights
FP and BP have a grid on the list of maps, but the AW thread grid is on the list of kernels (lters)


key: chroma:document
string_value:
3.4 Adjusting weights
FP and BP have a grid on the list of maps, but the AW thread grid is on the list of kernels (lters)
between maps of two consecutive layers. The 1D grid has a block for each connection between two
maps. Thread blocks are 2D, with a corresponding thread for every kernel weight. The bias weight
is included as an entire row of threads, thus requiring thread blocks to have ( Kx+1)Kythreads.
Most of the time these additional Kythreads will do nothing, thread (0,0) being activated only
for blocks that have to process the bias.
4 Experiments
We use a system with a Core i7-920 (2.66GHz), 12 GB DDR3 and four graphics cards: 2 x
GTX 480 and 2 x GTX 580. The correctness of the CPU version is checked by comparing the
analytical gradient with its nite dierence approximation. On GPU this is not possible because
all computations are performed with single precision 
oating point numbers. Hence the GPU
implementation's correctness is checked by comparing its results to those of a randomly initialized
net after training it for several epochs on the more accurate CPU version. Obtaining identical
results after trillions of operations is a strong indication of correctness.
Technical Report No. IDSIA-01-11 6


key: chroma:document
string_value:
results after trillions of operations is a strong indication of correctness.
Technical Report No. IDSIA-01-11 6
Figure 2: Back propagating deltas. A connection between two maps from two consecutive layers
is displayed. The map in Ln1has 29 x 29 neurons; the map in Lnhas 13 x 13 neurons. They
are linked through a 5 x 5 kernel K. Skipping factors of Sx= 1 andSy= 1 are assumed. Arrows
and colors depict the correspondence between neurons in Ln1and their sources in Ln.
The implemented CNN's plain feed-forward architecture is trained using on-line gradient descent. All images from the training set are used for training and also for validation. If deformations
are enabled, only the images from the training set will be deformed. Weights are initialized according to a uniform random distribution in the range [ 0:05;0:05]. Each neuron's activation
function is a scaled hyperbolic tangent: y(a) = 1:7159 tanh(0 :6666a) (LeCun et al. , 1998).
We pick the trained CNN with the lowest validation error, and evaluate it on the test set
(Test for best Validation - TfbV). The best test error (bT) is also listed for all experiments. The


key: chroma:document
string_value:
(Test for best Validation - TfbV). The best test error (bT) is also listed for all experiments. The
reported computation times per epoch include training, validation and testing as well as all data
transfers.
4.1 Experiments on MNIST
For the MNIST dataset the networks are trained on deformed images, continually generated in
on-line fashion. Ane (translation, rotation, scaling, horizontal shearing) and elastic deformations
(Simard et al. , 2003) are combined. We use a variable learning rate that shrinks by a multiplicative
constant after each epoch, from 103down to 3 105after 500 epochs.
Fully connected convolutional layers lead to an exploding number of network connections and
weights, making training of big and deep CNNs for hundreds of epochs impractical even on GPUs.
Partial connectivity alleviates this problem and is also biologically more plausible. We reduce
the number of connections between convolutional layers in a random way. Table 1 lists results
of various networks with 2 to 7 hidden layers with random connections. Additional layers result
in better networks, the best one achieving a test error of 0.35% for best validation and a best
test error of 0.27%. The best previous CNN result on MNIST is 0.40% (Simard et al. , 2003).


key: chroma:document
string_value:
test error of 0.27%. The best previous CNN result on MNIST is 0.40% (Simard et al. , 2003).
A 0.35% error rate was recently also obtained by a big, deep MLP (Cire san et al. , 2010) with
many more free parameters. Deeper nets require more computation time to complete an epoch,
but we observe that they also need fewer epochs to achieve good test errors. The deepest CNN
Technical Report No. IDSIA-01-11 7
Table 1: Error rates on MNIST test set for randomly connected CNNs with 2 to 6 convolutional
layers with M Maps and an optional fully connected layer with N neurons. Various kernel sizes
and skipping factors were used.
#M, #N bT TfbV
in Hidden Layers [%] [%]
20M-60M 0.95 1.02
20M-60M-150N 0.50 0.55
20M-60M-100M-150N 0.33 0.38
20M-40M-60M-80M-100M-120M-150N 0.27 0.35
from Table 1 reaches 2.42%, 0.97% and 0.48% after one, three and seventeen epochs, respectively.


key: chroma:document
string_value:
from Table 1 reaches 2.42%, 0.97% and 0.48% after one, three and seventeen epochs, respectively.
On the other hand, the network with 4 instead of 7 hidden layers reaches 4.71%, 1.58%, 0.68%
after one, three and seventeen epochs, achieving a test error below 0.50% after only 34 epochs.
This shows once more that deep networks, contrary to common belief, can be trained successfully
by back-propagation. Despite the numerous free parameters, deep networks seem to learn faster
(better recognition rates after fewer epochs) than shallow ones.
We consider MNIST an almost solved problem. The remaining errors stem from digits that
are ambiguous or miss parts.
4.2 Experiments on NORB
NORB contains stereo images of 3D objects. Hence there are two maps on the input layer.
Rotation, scaling, shearing and elastic distortions seem to have a negative impact on generalization.
These deformations improve recognition rates for digits that are intrinsically 2D (Cire san et al. ,
2010), but seem inadequate for 3D objects.
Initial experiments on NORB show that unlike with MNIST where we use deformations, the
CNN needs only 3 to 6 epochs to reach zero validation error. This allows us to quickly run numerous


key: chroma:document
string_value:
Initial experiments on NORB show that unlike with MNIST where we use deformations, the
CNN needs only 3 to 6 epochs to reach zero validation error. This allows us to quickly run numerous
repetitive experiments with huge networks with hundreds of maps per layer. We decided to use a
CNN with ve hidden layers: layer1, a convolutional layer with 300 maps, kernel size 6 6 and
skipping factors 1 1; layer2, a max-pooling layer over a 2 2 region; layer3, a convolutional layer
with 500 maps, kernel size 4 4, skipping factors 0 0; layer4, a max-pooling layer over a 4 4
region; layer5, a fully connected layer with 500 neurons. The learning rate is initialized by 0.001
and multiplied by 0.95 after every epoch.
Table 2 summarizes the results of four dierent experiments by switching on/o translation as
well as the xed image processing layer. We report the average error rate as well as the standard
deviation of N independent runs with identical architectures but dierent weight initializations.
For the rst experiment without translation and no image processing (IP), an average test error
rate of 7.86% is obtained. With additional translations of at most 5%, the average error rate drops


key: chroma:document
string_value:
rate of 7.86% is obtained. With additional translations of at most 5%, the average error rate drops
to 4.71%, contradicting the common belief that CNNs are translation invariant. These results are
on par or better than others in the literature: 5.90% error rate for a combination of CNNs and
SVMs (LeCun et al. , 2004) and 5.20% error rate for restricted Boltzman machines (Nair and
Hinton, 2009).
The best previously published result on NORB (2.87%) was obtained by a hierarchical neural
network which to every convolutional layer provides a subsampled version plus edge information
of the original image (Uetz and Behnke, 2009). This motivated us to implement a pre-processing
layer with xed lters. We tried simple edge masks (Sobel, Scharr) but obtained best results with a
contrast-extraction layer (Fukushima, 2003) realized by Mexican hat-shaped lters of size 21 21,
one with a concentric on-center receptive eld and one with a concentric o-center receptive eld,
Technical Report No. IDSIA-01-11 8
Table 2: Average error rates and standard deviations of N runs for a ve hidden layer CNN on


key: chroma:document
string_value:
Technical Report No. IDSIA-01-11 8
Table 2: Average error rates and standard deviations of N runs for a ve hidden layer CNN on
the NORB test set (see text for details).
trans. [%] IP TfbV [%] runs time/epoch [s]
0 no 7.86 0.55 50 1141
5 no 4.71 0.57 50 1563
0 yes 3.94 0.48 50 1658
5 yes 2.530.40 100 2080
similar to the lters automatically created by unsupervised Predictability Minimization (Schmidhuber, 1992) applied to natural images (Schmidhuber et al. , 1996). The rst lter extracts positive
contrast in brightness, whereas the latter extracts negative contrast. Each image from the original
NORB is ltered, consequently the input of the CNN has six maps: the original image plus the
positive and negative contrast for each of the two stereo channels. Using such a pre-processing
layer results in lower average error rates, 3.94% without translation and 2.53% with translation.
This result improves the previous state of the art on NORB (Uetz and Behnke, 2009).
Experience with other image datasets tells us that NORB is unusual. The training set has only


key: chroma:document
string_value:
Experience with other image datasets tells us that NORB is unusual. The training set has only
ve instances per class. The resulting poor training set variability makes the nets learn quickly
but generalize badly. NORB is the only dataset that prots from a xed pre-processing layer in
a substantial way. For MNIST and CIFAR10 such pre-processing has little or no eect. It is also
worth noting that NORB's standard error rate deviation is bigger than CIFAR10's (see Tables 2
and 3). Identical nets with dierent initializations do not produce very consistent results. The
best net had an error rate of 1.72%, the worst 3.69%.
4.3 Experiments on CIFAR 10
CIFAR10 is a collection of natural color images of 32x32 pixels. It contains 10 classes, each of them
with 5000 samples in the training set and 1000 in the test set. The images greatly vary inside each
class. They are not necessarily centered, may contain only parts of the object, and have varying
backgrounds. All of this makes CIFAR10 the hardest problem addressed in this paper. The CNN
has three maps, one for each color channel (RGB). The CIFAR10 images are relatively small in


key: chroma:document
string_value:
has three maps, one for each color channel (RGB). The CIFAR10 images are relatively small in
comparison to NORB's, and force us to use small kernels. The tested CNNs dier only in the
number of maps per convolutional and max-pooling layer. All have eight hidden layers: layer1, a
convolutional layer with 3 3 kernels and skipping factor of 0; layer2, a max-pooling layer over a
33 region; layer3, a convolutional layer with 3 3 kernels and skipping factors of 0 0; layer4,
a max-pooling over a 2 2 region; layer5, a convolutional layer with 3 3 kernels and a skipping
factors of 0 0; layer6, a max pooling layer over a 2 2 region; layer7, a fully connected layer
with 300 neurons; layer8, a fully connected layer with 100 neurons.
Like for MNIST, the learning rate is initialized by 0.001 and multiplied by 0.993 after every
epoch. Results in Table 3 show that without translation the error rate does not drop below 28%;
adding edge information does not help at all. Translations have a very positive eect, decreasing


key: chroma:document
string_value:
adding edge information does not help at all. Translations have a very positive eect, decreasing
the error rate to almost 20%. Contrast extraction lters are better than the Sobel/Scharr lters but
still worse than no pre-processing layer at all. Despite some CNN-inherent translation invariance,
additional training image translations cause better generalization; additional image processing
proved useless though.
To see if bigger nets are better, we increase the number of maps per layer from 100 to 200, 300
and 400, respectively (last three rows in Tab. 3). Training time increases exponentially, but the
test error decreases, reaching a minimum for nets with 300 maps per layer. Our 19.51% error rate
is better than the previous state of the art for this dataset, 20.40% (Coates et al. , 2010) and 25.50%
Technical Report No. IDSIA-01-11 9
Table 3: Average error rates and standard deviations for N runs of an eight hidden layer CNN on
the CIFAR10 test set (see text for details). The rst ve nets have 100 maps per convolutional
and max-pooling layer, whereas the sixth, seventh and eighth have 200, 300 and 400 maps per


key: chroma:document
string_value:
and max-pooling layer, whereas the sixth, seventh and eighth have 200, 300 and 400 maps per
hidden layer, respectively. IP - image processing layer: edge - 3 3 Sobel and Scharr lters; hat 1313 positive and negative contrast extraction lters.
trans. [%] maps IP TfbV [%] runs time/epoch [s]
0 100 no 28.87 0.37 11 93
0 100 edge 29.11 0.36 15 104
5 100 no 20.26 0.21 11 111
5 100 edge 21.87 0.57 5 120
5 100 hat 21.44 0.44 4 136
5 200 no 19.90 0.16 5 248
5 300 no 19.51 0.18 5 532
5 400 no 19.54 0.16 5 875
(Yu and Zhang, 2010). Unlike Coates et al. (2010), however, we use the original images without
any particular input normalization. Note that the error rate standard deviations are smaller than
those obtained on NORB, that is, dierent initializations yield consistent results.
4.4 Speedup factor of GPU code
The GPU code scales well with network size. For small nets the speedup is small (but still over 10)


key: chroma:document
string_value:
4.4 Speedup factor of GPU code
The GPU code scales well with network size. For small nets the speedup is small (but still over 10)
since they t better inside the CPU cache, and GPU resources are underutilized. For huge nets
(ex: Table 2) the GPU implementation is more than 60 times faster than a compiler-optimized
CPU version. Given the 
exibility of our GPU version, this is a signicant speedup. One epoch
takes 35 GPU minutes but more than 35 CPU hours.
5 Conclusion
We presented high-performance GPU-based CNN variants trained by on-line gradient descent, with
sparse random connectivity, computationally more ecient and biologically more plausible than
fully connected CNNs. Principal advantages include state-of-the-art generalization capabilities,
great 
exibility and speed. All structural CNN parameters such as input image size, number of
hidden layers, number of maps per layer, kernel sizes, skipping factors and connection tables are
adaptable to any particular application. We applied our networks to benchmark datasets for digit
recognition (MNIST), 3D object recognition (NORB), and natural images (CIFAR10). On MNIST
the best network achieved a recognition test error rate of 0.35%, on NORB 2.53% and on CIFAR10


key: chroma:document
string_value:
the best network achieved a recognition test error rate of 0.35%, on NORB 2.53% and on CIFAR10
19.51%. Our results are raising the bars for all three benchmarks. Currently the particular CNN
types discussed in this paper seem to be the best adaptive image recognizers, provided there is a
labeled dataset of sucient size. No unsupervised pretraining is required. Good results require
big and deep but sparsely connected CNNs, computationally prohibitive on CPUs, but feasible on
current GPUs, where our implementation is 10 to 60 times faster than a compiler-optimized CPU
version.
Technical Report No. IDSIA-01-11 10
Acknowledgment
This work was partially funded by the Swiss Commission for Technology and Innovation (CTI),
Project n. 9688.1 IFF: Intelligent Fill in Form.
References
S. Behnke. Hierarchical Neural Networks for Image Interpretation , volume 2766 of Lecture Notes
in Computer Science . Springer, 2003.
K. Chellapilla, S. Puri, and P. Simard. High performance convolutional neural networks for
document processing. In International Workshop on Frontiers in Handwriting Recognition ,
2006.
D. C. Cire san, U. Meier, L. M. Gambardella, and J. Schmidhuber. Deep big simple neural nets


key: chroma:document
string_value:
2006.
D. C. Cire san, U. Meier, L. M. Gambardella, and J. Schmidhuber. Deep big simple neural nets
for handwritten digit recogntion. Neural Computation , 22(12):3207{3220, 2010.
A. Coates, H. Lee, and A. Ng. An analysis of single-layer networks in unsupervised feature learning.
InAdvances in Neural Information Processing Systems , 2010.
K. Fukushima. Neocognitron: A self-organizing neural network for a mechanism of pattern recognition unaected by shift in position. Biological Cybernetics , 36(4):193{202, 1980.
K. Fukushima. Neocognitron for handwritten digit recognition. Neurocomputing , 51:161{180,
2003.
P. O. Hoyer and A. Hyv arinen. Independent component analysis applied to feature extraction
from colour and stero images. Network: Computation in Neural Systems , 11(3):191{210, 2000.
A. Krizhevsky. Learning multiple layers of features from tiny images. Master's thesis, Computer
Science Department, University of Toronto, 2009.
Y. LeCun, L. Bottou, Y. Bengio, and P. Haner. Gradient-based learning applied to document


key: chroma:document
string_value:
Y. LeCun, L. Bottou, Y. Bengio, and P. Haner. Gradient-based learning applied to document
recognition. Proceedings of the IEEE , 86(11):2278{2324, November 1998.
Y. LeCun, F.-J. Huang, and L. Bottou. Learning methods for generic object recognition with invariance to pose and lighting. In Proc. of Computer Vision and Pattern Recognition Conference ,
2004.
J. Mutch and D. G. Lowe. Object class recognition and localization using sparse features with
limited receptive elds. Int. J. Comput. Vision , 56(6):503{511, 2008.
V. Nair and G. E. Hinton. 3d object recognition with deep belief nets. In Advances in Neural
Information Processing Systems , 2009.
B. A. Olshausen and D. J. Field. Sparse coding with an overcomplete basis set: A strategy
employed by v1? Vision Research , 37(23):3311{3325, December 1997.
N. Pinto, D. Doukhan, J. J. DiCarlo, and D. D. Cox. A high-throughput screening approach
to discovering good forms of biologically inspired visual representation. PLoS computational
biology , 5(11):e1000579, November 2009.


key: chroma:document
string_value:
to discovering good forms of biologically inspired visual representation. PLoS computational
biology , 5(11):e1000579, November 2009.
M. Riesenhuber and T. Poggio. Hierarchical models of object recognition in cortex. Nat. Neurosci. ,
2(11):1019{1025, 1999.
Technical Report No. IDSIA-01-11 11
D. Scherer, A. M uller, and S. Behnke. Evaluation of pooling operations in convolutional architectures for object recognition. In International Conference on Articial Neural Networks , 2010.
J. Schmidhuber, M. Eldracher, and B. Foltin. Semilinear predictability minimization produces
well-known feature detectors. Neural Computation , 8(4):773{786, 1996.
J. Schmidhuber. Learning factorial codes by predictability minimization. Neural Computation ,
4(6):863{879, 1992.
T. Serre, L. Wolf, and T. Poggio. Object recognition with features inspired by visual cortex. In
Proc. of Computer Vision and Pattern Recognition Conference , 2007.
P. Simard, D. Steinkraus, and J. Platt. Best practices for convolutional neural networks applied
to visual document analysis. In Seventh International Conference on Document Analysis and


key: chroma:document
string_value:
to visual document analysis. In Seventh International Conference on Document Analysis and
Recognition , pages 958{963, 2003.
R. Uetz and S. Behnke. Large-scale object recognition with cuda-accelerated hierarchical neural
networks. In IEEE International Converence on Intelligent Computing and Intelligent Systems
(ICIS) , 2009.
D. H. Wiesel and T. N. Hubel. Receptive elds of single neurones in the cat's striate cortex. J.
Physiol. , 148:574{591, 1959.
K. Yu and T. Zhang. Improved local coordinate coding using local tangents. In Proceedings of the
International Conference on Machine Learning , 2010.


key: chroma:document
string_value:
Knowledge Matters: Importance of Prior Information for
Optimization
C  a glar G ul cehre gulcehrc@iro.umontreal.ca
D epartement d'informatique et de recherche op erationnelle
Universit e de Montr eal, Montr eal, QC, Canada
Yoshua Bengio bengioy@iro.umontreal.ca
D epartement d'informatique et de recherche op erationnelle
Universit e de Montr eal, Montr eal, QC, Canada
Editor: Not Assigned
Abstract
We explore the eect of introducing prior information into the intermediate level of deep
supervised neural networks for a learning task on which all the black-box state-of-the-art machine learning algorithms tested have failed to learn. We motivate our work from the hypothesis
that there is an optimization obstacle involved in the nature of such tasks, and that humans
learn useful intermediate concepts from other individuals via a form of supervision or guidance
using a curriculum. The experiments we have conducted provide positive evidence in favor of
this hypothesis. In our experiments, a two-tiered MLP architecture is trained on a dataset for


key: chroma:document
string_value:
using a curriculum. The experiments we have conducted provide positive evidence in favor of
this hypothesis. In our experiments, a two-tiered MLP architecture is trained on a dataset for
which each image input contains three sprites, and the binary target class is 1 if all three have
the same shape. Black-box machine learning algorithms only got chance on this task. Standard
deep supervised neural networks also failed. However, using a particular structure and guiding
the learner by providing intermediate targets in the form of intermediate concepts (the presence of each object) allows to nail the task. Much better than chance but imperfect results are
also obtained by exploring architecture and optimization variants, pointing towards a dicult
optimization task. We hypothesize that the learning diculty is due to the composition of two
highly non-linear tasks. Our ndings are also consistent with hypotheses on cultural learning
inspired by the observations of eective local minima (possibly due to ill-conditioning and the
training procedure not being able to escape what appears like a local minimum).
Keywords: Deep Learning, Neural Networks, Optimization, Evolution of Culture, Curriculum Learning, Training with Hints
1. Introduction
There is a recent emerging interest in dierent elds of science for cultural learning (Henrich
and McElreath, 2003) and how groups of individuals exchanging information can learn in ways


key: chroma:document
string_value:
and McElreath, 2003) and how groups of individuals exchanging information can learn in ways
superior to individual learning. This is also witnessed by the emergence of new research elds
such as "Social Neuroscience". Learning from other agents in an environment by the means
of cultural transmission of knowledge with a peer-to-peer communication is an ecient and
natural way of acquiring or propagating common knowledge. The most popular belief on how
the information is transmitted between individuals is that bits of information are transmitted
by small units, called memes, which share some characteristics of genes, such as self-replication,
mutation and response to selective pressures (Dawkins, 1976).
1arXiv:1301.4083v6  [cs.LG]  13 Jul 2013
This paper is based on the hypothesis (which is further elaborated in Bengio (2013a)) that
human culture and the evolution of ideas have been crucial to counter an optimization issue:
this diculty would otherwise make it dicult for human brains to capture high level knowledge of the world without the help of other educated humans. In this paper machine learning
experiments are used to investigate some elements of this hypothesis by seeking answers for
the following questions: are there machine learning tasks which are intrinsically hard for a
lone learning agent but that may become very easy when intermediate concepts are provided


key: chroma:document
string_value:
the following questions: are there machine learning tasks which are intrinsically hard for a
lone learning agent but that may become very easy when intermediate concepts are provided
by another agent as additional intermediate learning cues, in the spirit of Curriculum Learning (Bengio et al., 2009b)? What makes such learning tasks more dicult? Can specic initial
values of the neural network parameters yield success when random initialization yield complete failure? Is it possible to verify that the problem being faced is an optimization problem or
with a regularization problem? These are the questions discussed (if not completely addressed)
here, which relate to the following broader question: how can humans (and potentially one day,
machines) learn complex concepts?
In this paper, results of dierent machine learning algorithms on an articial learning task
involving binary 64 64 images are presented. In that task, each image in the dataset contains
3 Pentomino tetris sprites (simple shapes). The task is to gure out if all the sprites in the
image are the same or if there are dierent sprite shapes in the image. Several state-of-the-art
machine learning algorithms have been tested and none of them could perform better than
a random predictor on the test set. Nevertheless by providing hints about the intermediate
concepts (the presence and location of particular sprite classes), the problem can easily be solved


key: chroma:document
string_value:
a random predictor on the test set. Nevertheless by providing hints about the intermediate
concepts (the presence and location of particular sprite classes), the problem can easily be solved
where the same-architecture neural network without the intermediate concepts guidance fails.
Surprisingly, our attempts at solving this problem with unsupervised pre-training algorithms
failed solve this problem. However, with specic variations in the network architecture or
training procedure, it is found that one can make a big dent in the problem. For showing the
impact of intermediate level guidance, we experimented with a two-tiered neural network, with
supervised pre-training of the rst part to recognize the category of sprites independently of
their orientation and scale, at dierent locations, while the second part learns from the output
of the rst part and predicts the binary task of interest.
The objective of this paper is not to propose a novel learning algorithm or architecture,
but rather to rene our understanding of the learning diculties involved with composed tasks
(here a logical formula composed with the detection of object classes), in particular the training
diculties involved for deep neural networks. The results also bring empirical evidence in favor
of some of the hypotheses from Bengio (2013a), discussed below, as well as introducing a
particular form of curriculum learning (Bengio et al., 2009b).


key: chroma:document
string_value:
of some of the hypotheses from Bengio (2013a), discussed below, as well as introducing a
particular form of curriculum learning (Bengio et al., 2009b).
Building dicult AI problems has a long history in computer science. Specically hard
AI problems have been studied to create CAPTCHA's that are easy to solve for humans, but
hard to solve for machines (Von Ahn et al., 2003). In this paper we are investigating a dicult
problem for the o-the-shelf black-box machine learning algorithms.1
1.1 Curriculum Learning and Cultural Evolution Against Eective Local Minima
What Bengio (2013a) calls an eective local minimum is a point where iterative training
stalls, either because of an actual local minimum or because the optimization algorithm is
1. You can access the source code of some experiments presented in that paper and their hyperparameters from
here: https://github.com/caglar/kmatters
2
unable (in reasonable time) to nd a descent path (e.g., because of serious ill-conditioning). In
this paper, it is hypothesized that some more abstract learning tasks such as those obtained
by composing simpler tasks are more likely to yield eective local minima for neural networks,
and are generally hard for general-purpose machine learning algorithms.


key: chroma:document
string_value:
by composing simpler tasks are more likely to yield eective local minima for neural networks,
and are generally hard for general-purpose machine learning algorithms.
The idea that learning can be enhanced by guiding the learner through intermediate easier
tasks is old, starting with animal training by shaping (Skinner, 1958; Peterson, 2004; Krueger
and Dayan, 2009). Bengio et al. (2009b) introduce a computational hypothesis related to a
presumed issue with eective local minima when directly learning the target task: the good
solutions correspond to hard-to-nd-by-chance eective local minima, and intermediate tasks
prepare the learner's internal conguration (parameters) in a way similar to continuation methods in global optimization (which go through a sequence of intermediate optimization problems,
starting with a convex one where local minima are no issue, and gradually morphing into the
target task of interest).
In a related vein, Bengio (2013a) makes the following inferences based on experimental
observations of deep learning and neural network learning:
Point 1: Training deep architectures is easier when some hints are given about the function
that the intermediate levels should compute (Hinton et al., 2006; Weston et al., 2008;
Salakhutdinov and Hinton, 2009; Bengio, 2009). The experiments performed here expand


key: chroma:document
string_value:
Salakhutdinov and Hinton, 2009; Bengio, 2009). The experiments performed here expand
in particular on this point.
Point 2: It is much easier to train a neural network with supervision (where examples ar
provided to it of when a concept is present and when it is not present in a variety of
examples) than to expect unsupervised learning to discover the concept (which may also
happen but usually leads to poorer renditions of the concept). The poor results obtained
with unsupervised pre-training reinforce that hypothesis .
Point 3: Directly training all the layers of a deep network together not only makes it dicult to
exploit all the extra modeling power of a deeper architecture but in many cases it actually
yields worse results as the number of required layers is increased (Larochelle et al., 2009;
Erhan et al., 2010). The experiments performed here also reinforce that hypothesis.
Point 4: Erhan et al. (2010) observed that no two training trajectories ended up in the
same eective local minimum, out of hundreds of runs, even when comparing solutions as
functions from input to output, rather than in parameter space (thus eliminating from the
picture the presence of symmetries and multiple local minima due to relabeling and other


key: chroma:document
string_value:
functions from input to output, rather than in parameter space (thus eliminating from the
picture the presence of symmetries and multiple local minima due to relabeling and other
reparametrizations). This suggests that the number of dierent eective local minima
(even when considering them only in function space) must be huge.
Point 5: Unsupervised pre-training, which changes the initial conditions of the descent procedure, sometimes allows to reach substantially better eective local minima (in terms
of generalization error!), and these better local minima do not appear to be reachable
by chance alone (Erhan et al., 2010). The experiments performed here provide another
piece of evidence in favor of the hypothesis that where random initialization can yield
rather poor results, specically targeted initialization can have a drastic impact, i.e., that
3
eective local minima are not just numerous but that some small subset of them are much
better and hard to reach by chance.2
Based on the above points, Bengio (2013a) then proposed the following hypotheses regarding
learning of high-level abstractions.
Optimization Hypothesis: When it learns, a biological agent performs an approximate
optimization with respect to some implicit objective function.


key: chroma:document
string_value:
learning of high-level abstractions.
Optimization Hypothesis: When it learns, a biological agent performs an approximate
optimization with respect to some implicit objective function.
Deep Abstractions Hypothesis: Higher level abstractions represented in brains require deeper computations (involving the composition of more non-linearities).
Local Descent Hypothesis: The brain of a biological agent relies on approximate local
descent and gradually improves itself while learning.
Eective Local Minima Hypothesis: The learning process of a single human learner
(not helped by others) is limited by eective local minima.
Deeper Harder Hypothesis: Eective local minima are more likely to hamper learning
as the required depth of the architecture increases.
Abstractions Harder Hypothesis: High-level abstractions are unlikely to be discovered by a single human learner by chance, because these abstractions are represented by
a deep subnetwork of the brain, which learns by local descent.
Guided Learning Hypothesis: A human brain can learn high level abstractions if
guided by the signals produced by other agents that act as hints or indirect supervision
for these high-level abstractions.
Memes Divide-and-Conquer Hypothesis: Linguistic exchange, individual learning


key: chroma:document
string_value:
for these high-level abstractions.
Memes Divide-and-Conquer Hypothesis: Linguistic exchange, individual learning
and the recombination of memes constitute an ecient evolutionary recombination operator in the meme-space. This helps human learners to collectively build better internal
representations of their environment, including fairly high-level abstractions.
This paper is focused on \ Point 1 " and testing the \ Guided Learning Hypothesis ", using
machine learning algorithms to provide experimental evidence. The experiments performed
also provide evidence in favor of the \ Deeper Harder Hypothesis " and associated \ Abstractions
Harder Hypothesis ". Machine Learning is still far beyond the current capabilities of humans,
and it is important to tackle the remaining obstacles to approach AI. For this purpose, the
question to be answered is why tasks that humans learn eortlessly from very few examples,
while machine learning algorithms fail miserably?
2. Recent work showed that rather deep feedforward networks can be very successfully trained when large
quantities of labeled data are available (Ciresan et al., 2010; Glorot et al., 2011a; Krizhevsky et al., 2012).
Nonetheless, the experiments reported here suggest that it all depends on the task being considered, since
even with very large quantities of labeled examples, the deep networks trained here were unsuccessful.
4


key: chroma:document
string_value:
Nonetheless, the experiments reported here suggest that it all depends on the task being considered, since
even with very large quantities of labeled examples, the deep networks trained here were unsuccessful.
4
2. Culture and Optimization Diculty
As hypothesized in the \ Local Descent Hypothesis ", human brains would rely on a local approximate descent, just like a Multi-Layer Perceptron trained by a gradient-based iterative optimization. The main argument in favor of this hypothesis relies on the biologically-grounded
assumption that although ring patterns in the brain change rapidly, synaptic strengths underlying these neural activities change only gradually, making sure that behaviors are generally
consistent across time. If a learning algorithm is based on a form of local (e.g. gradient-based)
descent, it can be sensitive to eective local minima (Bengio, 2013a).
When one trains a neural network, at some point in the training phase the evaluation of
error seems to saturate, even if new examples are introduced. In particular Erhan et al. (2010)
nd that early examples have a much larger weight in the nal solution. It looks like the learner
is stuck in or near a local minimum. But since it is dicult to verify if this is near a true local


key: chroma:document
string_value:
is stuck in or near a local minimum. But since it is dicult to verify if this is near a true local
minimum or simply an eect of strong ill-conditioning, we call such a \stuck" conguration an
eective local minimum , whose denition depends not just on the optimization objective but
also on the limitations of the optimization algorithm.
Erhan et al. (2010) highlighted both the issue of eective local minima and a regularization eect when initializing a deep network with unsupervised pre-training. Interestingly, as
the network gets deeper the diculty due to eective local minima seems to be get more pronounced. That might be because of the number of eective local minima increases (more like
an actual local minima issue), or maybe because the good ones are harder to reach (more like
an ill-conditioning issue) and more work will be needed to clarify this question.
As a result of Point 4 we hypothesize that it is very dicult for an individual's brain to
discover some higher level abstractions by chance only. As mentioned in the \ Guided Learning
Hypothesis " humans get hints from other humans and learn high-level concepts with the guidance of other humans3. Curriculum learning (Bengio et al., 2009a) and incremental learning


key: chroma:document
string_value:
(Solomono, 1989), are examples of this. This is done by properly choosing the sequence of
examples seen by the learner, where simpler examples are introduced rst and more complex
examples shown when the learner is ready for them. One of the hypothesis on why curriculum
works states that curriculum learning acts as a continuation method that allows one to discover
a good minimum, by rst nding a good minimum of a smoother error function. Recent experiments on human subjects also indicates that humans teach by using a curriculum strategy
(Khan et al., 2011).
Some parts of the human brain are known to have a hierarchical organization (i.e. visual
cortex) consistent with the deep architecture studied in machine learning papers. As we go from
the sensory level to higher levels of the visual cortex, we nd higher level areas corresponding
to more abstract concepts. This is consistent with the Deep Abstractions Hypothesis .
Training neural networks and machine learning algorithms by decomposing the learning
task into sub-tasks and exploiting prior information about the task is well-established and
in fact constitutes the main approach to solving industrial problems with machine learning.
The contribution of this paper is rather on rendering explicit the eective local minima issue
and providing evidence on the type of problems for which this diculty arises. This prior


key: chroma:document
string_value:
The contribution of this paper is rather on rendering explicit the eective local minima issue
and providing evidence on the type of problems for which this diculty arises. This prior
information and hints given to the learner can be viewed as inductive bias for a particular task,
an important ingredient to obtain a good generalization error (Mitchell, 1980). An interesting
3. But some high-level concepts may also be hardwired in the brain, as assumed in the universal grammar
hypothesis (Montague, 1970), or in nature vs nurture discussions in cognitive science.
5
earlier nding in that line of research was done with Explanation Based Neural Networks
(EBNN) in which a neural network transfers knowledge across multiple learning tasks. An
EBNN uses previously learned domain knowledge as an initialization or search bias (i.e. to
constrain the learner in the parameter space) (O'Sullivan, 1996; Mitchell and Thrun, 1993).
Another related work in machine learning is mainly focused on reinforcement learning algorithms, based on incorporating prior knowledge in terms of logical rules to the learning
algorithm as a prior knowledge to speed up and bias learning (Kunapuli et al., 2010; Towell
and Shavlik, 1994).
As discussed in \ Memes Divide and Conquer Hypothesis \ societies can be viewed as a


key: chroma:document
string_value:
and Shavlik, 1994).
As discussed in \ Memes Divide and Conquer Hypothesis \ societies can be viewed as a
distributed computational processing systems. In civilized societies knowledge is distributed
across dierent individuals, this yields a space eciency. Moreover computation, i.e. each
individual can specialize on a particular task/topic, is also divided across the individuals in the
society and hence this will yield a computational eciency. Considering the limitations of the
human brain, the whole processing can not be done just by a single agent in an ecient manner.
A recent study in paleoantropology states that there is a substantial decline in endocranial
volume of the brain in the last 30000 years Henneberg (1988). The volume of the brain shrunk
to 1241 ml from 1502 ml (Henneberg and Steyn, 1993). One of the hypothesis on the reduction
of the volume of skull claims that, decline in the volume of the brain might be related to the
functional changes in brain that arose as a result of cultural development and emergence of
societies given that this time period overlaps with the transition from hunter-gatherer lifestyle
to agricultural societies.
3. Experimental Setup
Some tasks, which seem reasonably easy for humans to learn4, are nonetheless appearing almost
impossible to learn for current generic state-of-art machine learning algorithms.


key: chroma:document
string_value:
3. Experimental Setup
Some tasks, which seem reasonably easy for humans to learn4, are nonetheless appearing almost
impossible to learn for current generic state-of-art machine learning algorithms.
Here we study more closely such a task, which becomes learnable if one provides hints to
the learner about appropriate intermediate concepts. Interestingly, the task we used in our
experiments is not only hard for deep neural networks but also for non-parametric machine
learning algorithms such as SVM's, boosting and decision trees.
The result of the experiments for varying size of dataset with several o-the-shelf black box
machine learning algorithms and some popular deep learning algorithms are provided in Table
1. The detailed explanations about the algorithms and the hyperparameters used for those
algorithms are given in the Appendix Section 5.2. We also provide some explanations about
the methodologies conducted for the experiments at Section 3.2.
3.1 Pentomino Dataset
In order to test our hypothesis, an articial dataset for object recognition using 64 64 binary
images is designed5. If the task is two tiered (i.e., with guidance provided), the task in the
rst part is to recognize and locate each Pentomino object class6in the image. The second
4. keeping in mind that humans can exploit prior knowledge, either from previous learning or innate knowledge.


key: chroma:document
string_value:
4. keeping in mind that humans can exploit prior knowledge, either from previous learning or innate knowledge.
5. The source code for the script that generates the articial Pentomino datasets (Arcade-Universe) is available
at:https://github.com/caglar/Arcade-Universe . This implementation is based on Olivier Breuleux's
bugland dataset generator.
6. A human learner does not seem to need to be taught the shape categories of each Pentomino sprite in order
to solve the task. On the other hand, humans have lots of previously learned knowledge about the notion of
shape and how central it is in dening categories.
6
(a) sprites, not all same type
 (b) sprites, all of same type
Figure 1: Left (a): An example image from the dataset which has a dierent sprite type in it.
Right (b): An example image from the dataset that has only one type of Pentomino
object in it, but with dierent orientations and scales.
part/nal binary classication task is to gure out if all the Pentominos in the image are of
the same shape class or not. If a neural network learned to detect the categories of each object
at each location in an image, the remaining task becomes an XOR-like operation between the


key: chroma:document
string_value:
the same shape class or not. If a neural network learned to detect the categories of each object
at each location in an image, the remaining task becomes an XOR-like operation between the
detected object categories. The types of Pentomino objects that is used for generating the
dataset are as follows:
Pentomino sprites N, P, F, Y, J, and Q, along with the Pentomino N2 sprite (mirror of
\Pentomino N" sprite), the Pentomino F2 sprite (mirror of \Pentomino F" sprite), and the
Pentomino Y2 sprite (mirror of \Pentomino Y" sprite).
Figure 2: Dierent classes of Pentomino shapes used in our dataset.
As shown in Figures 1(a) and 1(b), the synthesized images are fairly simple and do not
have any texture. Foreground pixels are \1" and background pixels are \0". Images of the
training and test sets are generated iid. For notational convenience, assume that the domain
of raw input images is X, the set of sprites is S, the set of intermediate object categories is Y
for each possible location in the image and the set of nal binary task outcomes is Z=f0;1g.


key: chroma:document
string_value:
for each possible location in the image and the set of nal binary task outcomes is Z=f0;1g.
Two dierent types of rigid body transformation is performed: sprite rotation rot(X;
) where
 =f
: (
= 90)^[(2N);(03)]gand scaling scale (X;) where2f1;2gis the
scaling factor. The data generating procedure is summarized below.
Sprite transformations: Before placing the sprites in an empty image, for each image x2X,
a value for z2Zis randomly sampled which is to have (or not) the same three sprite
shapes in the image. Conditioned on the constraint given by z, three sprites are randomly
7
selectedsijfromSwithout replacement. Using a uniform probability distribution over
all possible scales, a scale is chosen and accordingly each sprite image is scaled. Then
rotate each sprite is randomly rotated by a multiple of 90 degrees.
Sprite placement: Upon completion of sprite transformations, a 64 64 uniform grid is generated which is divided into 8 8 blocks, each block being of size 8 8 pixels, and randomly
select three dierent blocks from the 64=8 8 on the grid and place the transformed


key: chroma:document
string_value:
select three dierent blocks from the 64=8 8 on the grid and place the transformed
objects into dierent blocks (so they cannot overlap, by construction).
Each sprite is centered in the block in which it is located. Thus there is no object translation
inside the blocks. The only translation invariance is due to the location of the block inside the
image.
A Pentomino sprite is guaranteed to not over
ow the block in which it is located, and there
are no collisions or overlaps between sprites, making the task simpler. The largest possible
Pentomino sprite can be t into an 8 4 mask.
3.2 Learning Algorithms Evaluated
Initially the models are cross-validated by using 5-fold cross-validation. With 40,000 examples,
this gives 32,000 examples for training and 8,000 examples for testing. For neural network
algorithms, stochastic gradient descent (SGD) is used for training. The following standard
learning algorithms were rst evaluated: decision trees, SVMs with Gaussian kernel, ordinary
fully-connected Multi-Layer Perceptrons, Random Forests, k-Nearest Neighbors, Convolutional
Neural Networks, and Stacked Denoising Auto-Encoders with supervised ne-tuning. More


key: chroma:document
string_value:
Neural Networks, and Stacked Denoising Auto-Encoders with supervised ne-tuning. More
details of the congurations and hyper-parameters for each of them are given in Appendix
Section 5.2. The only better than chance results were obtained with variations of the Structured
Multi-Layer Perceptron described below.
3.2.1 Structured Multi-Layer Perceptron (SMLP)
The neural network architecture that is used to solve this task is called the SMLP (Structured
Multi-Layer Perceptron), a deep neural network with two parts as illustrated in Figure 5 and
7:
The lower part, P1NN ( Part 1 Neural Network , as it is called in the rest of the paper),
has shared weights and local connectivity, with one identical MLP instance of the P1NN for
each patch of the image, and typically an 11-element output vector per patch (unless otherwise
noted). The idea is that these 11 outputs per patch could represent the detection of the sprite
shape category (or the absence of sprite in the patch). The upper part, P2NN ( Part 2 Neural
Network ) is a fully connected one hidden layer MLP that takes the concatenation of the outputs


key: chroma:document
string_value:
Network ) is a fully connected one hidden layer MLP that takes the concatenation of the outputs
of all patch-wise P1NNs as input. Note that the rst layer of P1NN is similar to a convolutional
layer but where the stride equals the kernel size, so that windows do not overlap, i.e., P1NN can
be decomposed into separate networks sharing the same parameters but applied on dierent
patches of the input image, so that each network can actually be trained patch-wise in the case
where a target is provided for the P1NN outputs. The P1NN output for patch piwhich is
extracted from the image xis computed as follows:
f(pi) =g2(Vg1(Upi+b) +c) (1)
8
where pi2Rdis the input patch/receptive eld extracted from location iof a single image.
U2Rdhdis the weight matrix for the rst layer of P1NN and b2Rd
his the vector of biases
for the rst layer of P1NN. g1() is the activation function of the rst layer and g2() is the
activation function of the second layer. In many of the experiments, best results were obtained


key: chroma:document
string_value:
activation function of the second layer. In many of the experiments, best results were obtained
withg1() a rectifying non-linearity (a.k.a. as RELU), which is max(0; X) (Jarrett et al.,
2009b; Nair and Hinton, 2010; Glorot et al., 2011a; Krizhevsky et al., 2012). V2Rdhdois the
second layer's weights matrix, such that and c2Rdoare the biases of the second layer of the
P1NN, with doexpected to be smaller than dh.
In this way, g1(Upi+b) is an overcomplete representation of the input patch that can
potentially represent all the possible Pentomino shapes for all factors of variations in the patch
(rotation, scaling and Pentomino shape type). On the other hand, when trained with hints,
f(pi) is expected to be the lower dimensional representation of a Pentomino shape category
invariant to scaling and rotation in the given patch.
In the experiments with SMLP trained with hints (targets at the output of P1NN), the
P1NN is expected to perform classication of each 8 8 non-overlapping patches of the original


key: chroma:document
string_value:
P1NN is expected to perform classication of each 8 8 non-overlapping patches of the original
6464 input image without having any prior knowledge of whether that specic patch contains
a Pentomino shape or not. P1NN in SMLP without hints just outputs the local activations for
each patch, and gradients on f(pi) are backpropagated from the upper layers. In both cases
P1NN produces the input representation for the Part 2 Neural Net (P2NN). Thus the input
representation of P2NN is the concatenated output of P1NN across all the 64 patch locations:
ho= [f(p0);:::;f(pi);:::;f(pN))] whereNis the number of patches and the ho2Rdi;di=
doN.hois the concatenated output of the P1NN at each patch.
There is a standardization layer on top of the output of P1NN that centers the activations
and performs divisive normalization by dividing by the standard deviation over a minibatch
of the activations of that layer. We denote the standardization function z(). Standardization
makes use of the mean and standard deviation computed for each hidden unit such that each


key: chroma:document
string_value:
of the activations of that layer. We denote the standardization function z(). Standardization
makes use of the mean and standard deviation computed for each hidden unit such that each
hidden unit of howill have 0 activation and unit standard deviation on average over the
minibatch. Xis the set of pentomino images in the minibatch, where X2RdinNis a matrix
withNimages.h(i)
o(xj) is the vector of activations of the i-th hidden unit of hidden layer
ho(xj) for thej-th example, with xj2X.
h(i)
o=1
NX
xj2Xh(i)
o(xj) (2)
h(i)
o=sPN
j(h(i)
o(xj)h(i)
o)2
N+ (3)
z(h(i)
o(xj)) =h(i)
o(xj)h(i)
o
max(h(i)
o;)(4)
whereis a very small constant, that is used to prevent numerical under
ows in the standard
deviation. P1NN is trained on each 8 8 patches extracted from the image.


key: chroma:document
string_value:
deviation. P1NN is trained on each 8 8 patches extracted from the image.
hois standardized for each training and test sample separately. Dierent values of were
used for SMLP-hints and SMLP-nohints.
The concatenated output of P1NN is fed as an input to the P2NN. P2NN is a feedforward
MLP with a sigmoid output layer using a single RELU hidden layer. The task of P2NN is to
perform a nonlinear logical operation on the representation provided at the output of P1NN.
9
3.2.2 Structured Multi Layer Perceptron Trained with Hints (SMLP-hints)
The SMLP-hints architecture exploits a hint about the presence and category of Pentomino
objects, specifying a semantics for the P1NN outputs. P1NN is trained with the intermediate
targetY, specifying the type of Pentomino sprite shape present (if any) at each of the 64 patches
(88 non-overlapping blocks) of the image. Because a possible answer at a given location can
be \none of the object types" i.e., an empty patch, yp(for patchp) can take one of the 11


key: chroma:document
string_value:
be \none of the object types" i.e., an empty patch, yp(for patchp) can take one of the 11
possible values, 1 for rejection and the rest is for the Pentomino shape classes, illustrated in
Figure 2:
yp=(
0 if patch pis empty
s2Sif the patch pcontains a Pentomino sprite :
A similar task has been studied by Fleuret et al. (2011) (at SI appendix Problem 17), who
compared the performance of humans vs computers.
The SMLP-hints architecture takes advantage of dividing the task into two subtasks during
training with prior information about intermediate-level relevant factors. Because the sum of
the training losses decomposes into the loss on each patch, the P1NN can be pre-trained patchwise. Each patch-specic component of the P1NN is a fully connected MLP with 8 8 inputs
and 11 outputs with a softmax output layer. SMLP-hints uses the the standardization given
in Equation 3 but with = 0.
The standardization is a crucial step for training the SMLP on the Pentomino dataset, and
yields much sparser outputs, as seen on Figures 3 and 4. If the standardization is not used,


key: chroma:document
string_value:
yields much sparser outputs, as seen on Figures 3 and 4. If the standardization is not used,
even SMLP-hints could not solve the Pentomino task. In general, the standardization step
dampens the small activations and augments larger ones(reducing the noise). Centering the
activations of each feature detector in a neural network has been studied in (Raiko et al., 2012)
and (Vatanen et al., 2013). They proposed that transforming the outputs of each hidden neuron
in a multi-layer perceptron network to have zero output and zero slope on average makes rst
order optimization methods closer to the second order techniques.
By default, the SMLP uses rectier hidden units as activation function, we found a signicant boost by using rectication compared to hyperbolic tangent and sigmoid activation
functions. The P1NN has a highly overcomplete architecture with 1024 hidden units per patch,
and L1 and L2 weight decay regularization coecients on the weights (not the biases) are respectively 1e-6 and 1e-5. The learning rate for the P1NN is 0.75. 1 training epoch was enough


key: chroma:document
string_value:
for the P1NN to learn the features of Pentomino shapes perfectly on the 40000 training examples. The P2NN has 2048 hidden units. L1 and L2 penalty coecients for the P2NN are
1e-6, and the learning rate is 0.1. These were selected by trial and error based on validation
set error. Both P1NN (for each patch) and P2NN are fully-connected neural networks, even
though P1NN globally is a special kind of convolutional neural network.
Filters of the rst layer of SMLP are shown in Figure 6. These are the examples of the
lters obtained with the SLMP-hints trained with 40k examples, whose results are given in
Table 1. Those lters look very noisy but they work perfectly on the Pentomino task.
3.2.3 Deep and Structured Supervised MLP without Hints (SMLP-nohints)
SMLP-nohints uses the same connectivity pattern (and deep architecture) that is also used
in the SMLP-hints architecture, but without using the intermediate targets ( Y). It directly
predicts the nal outcome of the task ( Z), using the same number of hidden units, the same
10


key: chroma:document
string_value:
predicts the nal outcome of the task ( Z), using the same number of hidden units, the same
10
Figure 3: Bar chart of concatenated softmax output activations hoof P1NN (1164=704 outputs) in SMLP-hints before standardization, for a selected example. There are very
large spikes at each location for one of the possible 11 outcome (1 of K representation).
11
Figure 4: Softmax output activations hoof P1NN at SMLP-hints before standardization. There
are positive spiked outputs at the locations where there is a Pentomino shape. Positive and negative spikes arise because most of the outputs are near an average value.
Activations are higher at the locations where there is a pentomino shape.
12
Structured MLP 
Architecture with 
Hints
Final Binary task 
labels
Intermediate level 
targets .
Second 
Level 
Neural 
Network
First 
Level 
Neural 
Network
Figure 5: Structured MLP architecture, used with hints (trained in two phases, rst P1NN,
bottom two layers, then P2NN, top two layers). In SMLP-hints, P1NN is trained


key: chroma:document
string_value:
bottom two layers, then P2NN, top two layers). In SMLP-hints, P1NN is trained
on each 8x8 patch extracted from the image and the softmax output probabilities of
all 64 patches are concatenated into a 64 11 vector that forms the input of P2NN.
OnlyUandVare learned in the P1NN and its output on each patch is fed into
P2NN. The rst level and the second level neural networks are trained separately,
not jointly.
Figure 6: Filters of Structured MLP architecture, trained with hints on 40k examples.
13
connectivity and the same activation function for the hidden units as SMLP-hints. 120 hyperparameter values have been evaluated by randomly selecting the number of hidden units
from [64;128;256;512;1024;1200;2048] and randomly sampling 20 learning rates uniformly in
the log-domain within the interval of [0 :008;0:8]. Two fully connected hidden layers with 1024
hidden units (same as P1NN) per patch is used and 2048 (same as P2NN) for the last hidden layer, with twenty training epochs. For this network the best results are obtained with a
learning rate of 0.05.7
Structured MLP 
Architecture without 
Hints
Final Binary task


key: chroma:document
string_value:
learning rate of 0.05.7
Structured MLP 
Architecture without 
Hints
Final Binary task 
labels
Second 
Level 
Neural 
Network
First 
Level 
Neural 
Network
Figure 7: Structured MLP architecture, used without hints (SMLP-nohints). It is the same
architecture as SMLP-hints (Figure 5) but with both parts (P1NN and P2NN) trained
jointly with respect to the nal binary classication task.
We chose to experiment with various SMLP-nohint architectures and optimization procedures, trying unsuccessfully to achieve as good results with SMLP-nohint as with SMLP-hints.
Rectier Non-Linearity A rectier nonlinearity is used for the activations of MLP hidden
layers. We observed that using piecewise linear nonlinearity activation function such as the
rectier can make the optimization more tractable.
7. The source code of the structured MLP is available at the github repository: https://github.com/caglar/
structured_mlp
14


key: chroma:document
string_value:
7. The source code of the structured MLP is available at the github repository: https://github.com/caglar/
structured_mlp
14
Figure 8: First layer lters learned by the Structured MLP architecture, trained without using hints on 447600 examples with online SGD and a sigmoid intermediate layer
activation.
Intermediate Layer The output of the P1NN is considered as an intermediate layer of the
SMLP. For the SMLP-hints, only softmax output activations have been tried at the intermediate
layer, and that suced to learn the task. Since things did not work nearly as well with
the SMLP-nohints, several dierent activation functions have been tried: softmax( ), tanh(),
sigmoid() and linear activation functions.
Standardization Layer Normalization at the last layer of the convolutional neural networks
has been used occasionaly to encourage the competition between the hidden units. (Jarrett
et al., 2009a) used a local contrast normalization layer in their architecture which performs
subtractive and divisive normalization. A local contrast normalization layer enforces a local
competition between adjacent features in the feature map and between features at the same


key: chroma:document
string_value:
subtractive and divisive normalization. A local contrast normalization layer enforces a local
competition between adjacent features in the feature map and between features at the same
spatial location in dierent feature maps. Similarly (Krizhevsky et al., 2012) observed that
using a local response layer that enjoys the benet of using local normalization scheme aids
generalization.
Standardization has been observed to be crucial for both SMLP trained with or without hints. In both SMLP-hints and SMLP-nohints experiments, the neural network was not
able to generalize or even learn the training set without using standardization in the SMLP
intermediate layer, doing just chance performance. More specically, in the SMLP-nohints architecture, standardization is part of the computational graph, hence the gradients are being
backpropagated through it. The mean and the standard deviation is computed for each hidden
unit separately at the intermediate layer as in Equation 4. But in order to prevent numerical
under
ows or over
ows during the backpropagation we have used = 1e8 (Equation 3).
The benet of having sparse activations may be specically important for the ill-conditioned


key: chroma:document
string_value:
The benet of having sparse activations may be specically important for the ill-conditioned
problems, for the following reasons. When a hidden unit is \o", its gradient (the derivative of
the loss with respect to its output) is usually close to 0 as well, as seen here. That means that all
o-diagonal second derivatives involving that hidden unit (e.g. its input weights) are also near 0.
This is basically like removing some columns and rows from the Hessian matrix associated with
a particular example. It has been observed that the condition number of the Hessian matrix
(specically, its largest eigenvalue) increases as the size of the network increases (Dauphin and
Bengio, 2013), making training considerably slower and inecient (Dauphin and Bengio, 2013).
Hence one would expect that as sparsity of the gradients (obtained because of sparsity of the
activations) increases, training would become more ecient, as if we were training a smaller
sub-network for each example, with shared weights across examples, as in dropouts (Hinton
et al., 2012).
In Figure 9, the activation of each hidden unit in a bar chart is shown: the eect of
standardization is signicant, making the activations sparser.


key: chroma:document
string_value:
In Figure 9, the activation of each hidden unit in a bar chart is shown: the eect of
standardization is signicant, making the activations sparser.
15
(a) Before standardization.
 (b) After standardization.
Figure 9: Activations of the intermediate-level hidden units of an SLMP-nohints for a particular
examples (x-axis: hidden unit number, y-axis: activation value). Left (a): before
standardization. Right (b): after standardization.
In Figure 10, one can see the activation histogram of the SMLP-nohints intermediate layer,
showing the distribution of activation values, before and after standardization. Again the
sparsifying eect of standardization is very apparent.
(a) Before standardization.
 (b) After standardization.
Figure 10: Distribution histogram of activation values of SMLP-nohints intermediate layer.
Left (a): before standardization. Right (b): after standardization.
In Figures 10 and 9, the intermediate level activations of SMLP-nohints are shown before
and after standardization. These are for the same SMLP-nohints architecture whose results
are presented on Table 1. For that same SMLP, the Adadelta (Zeiler, 2012) adaptive learning
16


key: chroma:document
string_value:
are presented on Table 1. For that same SMLP, the Adadelta (Zeiler, 2012) adaptive learning
16
rate scheme has been used, with 512 hidden units for the hidden layer of P1NN and rectier
activation function. For the output of the P1NN, 11 sigmoidal units have been used while
P2NN had 1200 hidden units with rectier activation function. The output nonlinearity of the
P2NN is a sigmoid and the training objective is the binary crossentropy.
Adaptive Learning Rates We have experimented with several dierent adaptive learning
rate algorithms. We tried rmsprop8, Adadelta (Zeiler, 2012), Adagrad (Duchi et al., 2010) and
a linearly (1/t) decaying learning rate (Bengio, 2013b). For the SMLP-nohints with sigmoid
activation function we have found Adadelta(Zeiler, 2012) converging faster to an eective local
minima and usually yielding better generalization error compared to the others.
3.2.4 Deep and Structured MLP with Unsupervised Pre-Training
Several experiments have been conducted using an architecture similar to the SMLP-nohints,


key: chroma:document
string_value:
3.2.4 Deep and Structured MLP with Unsupervised Pre-Training
Several experiments have been conducted using an architecture similar to the SMLP-nohints,
but by using unsupervised pre-training of P1NN, with Denoising Auto-Encoder (DAE) and/or
Contractive Auto-Encoders (CAE). Supervised ne-tuning proceeds as in the deep and structured MLP without hints. Because an unsupervised learner may not focus the representation
just on the shapes, a larger number of intermediate-level units at the output of P1NN has been
explored: previous work on unsupervised pre-training generally found that larger hidden layers
were optimal when using unsupervised pre-training, because not all unsupervised features will
be relevant to the task at hand. Instead of limiting to 11 units per patch, we experimented
with networks with up to 20 hidden (i.e., code) units per patch in the second-layer patch-wise
auto-encoder.
In Appendix 5.1 we also provided the result of some experiments with binary-binary RBMs
trained on 88 patches from the 40k training dataset.
In unsupervised pretraining experiments in this paper, both contractive auto-encoder (CAE)


key: chroma:document
string_value:
trained on 88 patches from the 40k training dataset.
In unsupervised pretraining experiments in this paper, both contractive auto-encoder (CAE)
with sigmoid nonlinearity and binary cross entropy cost function and denoising auto-encoder (DAE)
have been used. In the second layer, experiments were performed with a DAE with rectier
hidden units utilizing L1 sparsity and weight decay on the weights of the auto-encoder. Greedy
layerwise unsupervised training procedure is used to train the deep auto-encoder architecture
(Bengio et al., 2007). In unsupervised pretraining experiments, tied weights have been used.
Dierent combinations of CAE and DAE for unsupervised pretraining have been tested, but
none of the congurations tested managed to learn the Pentomino task, as shown in Table 1.
3.3 Experiments with 1 of K representation
To explore the eect of changing the complexity of the input representation on the diculty
of the task, a set of experiments have been designed with symbolic representations of the
information in each patch. In all cases an empty patch is represented with a 0 vector. These
representation can be seen as an alternative input for a P2NN-like network, i.e., they were fed


key: chroma:document
string_value:
representation can be seen as an alternative input for a P2NN-like network, i.e., they were fed
as input to an MLP or another black-box classier.
The following four experiments have been conducted, each one using one using a dierent
input representation for each patch:
8. This is learning rate scaling method that is discussed by G. Hinton in his Video Lecture 6.5 - rmsprop: Divide
the gradient by a running average of its recent magnitude. COURSERA: Neural Networks for Machine
Learning, 2012.
17
Algorithm 20k dataset 40k dataset 80k dataset
Training Test Training Test Training Test
Error Error Error Error Error Error
SVM RBF 26.2 50.2 28.2 50.2 30.2 49.6
K Nearest Neighbors 24.7 50.0 25.3 49.5 25.6 49.0
Decision Tree 5.8 48.6 6.3 49.4 6.9 49.9
Randomized Trees 3.2 49.8 3.4 50.5 3.5 49.1
MLP 26.5 49.3 33.2 49.9 27.2 50.1
Convnet/Lenet5 50.6 49.8 49.4 49.8 50.2 49.8


key: chroma:document
string_value:
Convnet/Lenet5 50.6 49.8 49.4 49.8 50.2 49.8
Maxout Convnet 14.5 49.5 0.0 50.1 0.0 44.6
2 layer sDA 49.4 50.3 50.2 50.3 49.7 50.3
Struct. Supervised MLP w/o hints 0.0 48.6 0.0 36.0 0.0 12.4
Struct. MLP+CAE Supervised Finetuning 50.5 49.7 49.8 49.7 50.3 49.7
Struct. MLP+CAE+DAE, Supervised Finetuning 49.1 49.7 49.4 49.7 50.1 49.7
Struct. MLP+DAE+DAE, Supervised Finetuning 49.5 50.3 49.7 49.8 50.3 49.7
Struct. MLP with Hints 0.21 30.7 0 3.1 0 0.01
Table 1: The error percentages with dierent learning algorithms on Pentomino dataset with
dierent number of training examples.


key: chroma:document
string_value:
Table 1: The error percentages with dierent learning algorithms on Pentomino dataset with
dierent number of training examples.
Experiment 1-Onehot representation without transformations: In this experiment several trials have been done with a 10-input one-hot vector per patch. Each input corresponds to an object category given in clear, i.e., the ideal input for P2NN if a supervised
P1NN perfectly did its job.
Experiment 2-Disentangled representations: In this experiment, we did trials with 16
binary inputs per patch, 10 one-hot bits for representing each object category, 4 for
rotations and 2 for scaling, i.e., the whole information about the input is given, but it is
perfectly disentangled. This would be the ideal input for P2NN if an unsupervised P1NN
perfectly did its job.
Experiment 3-Onehot representation with transformations: For each of the ten object
types there are 8 = 4 2 possible transformations. Two objects in two dierent patches
are the considered \the same" (for the nal task) if their category is the same regardless
of the transformations. The one-hot representation of a patch corresponds to the crossproduct between the 10 object shape classes and the 4 2 transformations, i.e., one out of


key: chroma:document
string_value:
of the transformations. The one-hot representation of a patch corresponds to the crossproduct between the 10 object shape classes and the 4 2 transformations, i.e., one out of
80=1042 possibilities represented in an 80-bit one-hot vector. This also contains all
the information about the input image patch, but spread out in a kind of non-parametric
and non-informative (not disentangled) way, like a perfect memory-based unsupervised
learner (like clustering) could produce. Nevertheless, the shape class would be easier to
read out from this representation than from the image representation (it would be an OR
over 8 of the bits).
Experiment 4-Onehot representation with 80 choices: This representation has the same
1 of 80 one-hot representation per patch but the target task is dened dierently. Two objects in two dierent patches are considered the same i they have exactly the same 80-bit
onehot representation (i.e., are of the same object category with the same transformation
applied).
The rst experiment is a sanity check. It was conducted with single hidden-layered MLP's
with rectier and tanh nonlinearity, and the task was learned perfectly (0 error on both training
and test dataset) with very few training epochs.
18


key: chroma:document
string_value:
with rectier and tanh nonlinearity, and the task was learned perfectly (0 error on both training
and test dataset) with very few training epochs.
18
0 100 200 300 400 500 600 700 800 9000.00.10.20.30.40.50.6
Training Error Rate
Test Error Rate(a) Training and Test Errors for Experiment 4
0 100 200 300 400 500 600 700 8000.00.10.20.30.40.50.6
Training Error Rate
Test Error Rate (b) Training and Test Errors for Experiment 3
Figure 11: Tanh MLP training curves. Left (a): The training and test errors of Experiment
3 over 800 training epochs with 100k training examples using Tanh MLP. Right
(b):The training and test errors of Experiment 4 over 700 training epochs with 100k
training examples using Tanh MLP.
The results of Experiment 2 are given in Table 2. To improve results, we experimented
with the Maxout non-linearity in a feedforward MLP (Goodfellow et al., 2013) with two hidden
layers. Unlike the typical Maxout network mentioned in the original paper, regularizers have
been deliberately avoided in order to focus on the optimization issue, i.e: no weight decay, norm
constraint on the weights, or dropout. Although learning from a disentangled representation is


key: chroma:document
string_value:
constraint on the weights, or dropout. Although learning from a disentangled representation is
more dicult than learning from perfect object detectors, it is feasible with some architectures
such as the Maxout network. Note that this representation is the kind of representation that
one could hope an unsupervised learning algorithm could discover, at best, as argued in Bengio
et al. (2012).
The only results obtained on the validation set for Experiment 3 and Experiment 4 are
shown respectively in Table 3 and Table 4. In these experiments a tanh MLP with two hidden
layers have been tested with the same hyperparameters. In experiment 3 the complexity of the
problem comes from the transformations (8=4 2) and the number of object types.
But in experiment 4, the only source of complexity of the task comes from the number of
dierent object types. These results are in between the complete failure and complete success
observed with other experiments, suggesting that the task could become solvable with better
training or more training examples. Figure 11 illustrates the progress of training a tanh MLP,
on both the training and test error, for Experiments 3 and 4. Clearly, something has been
learned, but the task is not nailed yet. On experiment 3 for both maxout and tanh the maxout
there was a long plateau where the training error and objective stays almost same. Maxout


key: chroma:document
string_value:
there was a long plateau where the training error and objective stays almost same. Maxout
did just chance on the experiment for about 120 iterations on the training and the test set.
But after 120th iteration the training and test error started decline and eventually it was able
to solve the task. Moreover as seen from the curves in Figure 11(a) and 11(b), the training
and test error curves are almost the same for both tasks. This implies that for onehot inputs,
whether you increase the number of possible transformations for each object or the number of
19
Learning Algorithm Training Error Test Error
SVM 0.0 35.6
Random Forests 1.29 40.475
Tanh MLP 0.0 0.0
Maxout MLP 0.0 0.0
Table 2: Performance of dierent learning algorithms on disentangled representation in Experiment 2.
Learning Algorithm Training Error Test Error
SVM 11.212 32.37
Random Forests 24.839 48.915
Tanh MLP 0.0 22.475
Maxout MLP 0.0 0.0
Table 3: Performance of dierent learning algorithms using a dataset with onehot vector and
80 inputs as discussed for Experiment 3.
object categories, as soon as the number of possible congurations is same, the complexity of
the problem is almost the same for the MLP.


key: chroma:document
string_value:
object categories, as soon as the number of possible congurations is same, the complexity of
the problem is almost the same for the MLP.
3.4 Does the Eect Persist with Larger Training Set Sizes?
The results shown in this section indicate that the problem in the Pentomino task clearly
is not just a regularization problem, but rather basically hinges on an optimization problem.
Otherwise, we would expect test error to decrease as the number of training examples increases.
This is shown rst by studying the online case and then by studying the ordinary training
case with a xed size training set but considering increasing training set sizes. In the online
minibatch setting, parameter updates are performed as follows:
t+1=tt (5)
t=PN
irtL(xt;t)
N(6)
whereL(xt;t) is the loss incurred on example xtwith parameters t, wheret2Z+and
is the learning rate.
Ordinary batch algorithms converge linearly to the optimum , however the noisy gradient
estimates in the online SGD will cause parameter to 
uctuate near the local optima. However,
online SGD directly optimizes the expected risk, because the examples are drawn iid from the


key: chroma:document
string_value:
online SGD directly optimizes the expected risk, because the examples are drawn iid from the
ground-truth distribution (Bottou, 2010). Thus:
L1=E[L(x;)] =Z
xL(x;)p(x)dx (7)
20
Learning Algorithm Training Error Test Error
SVM 4.346 40.545
Random Forests 23.456 47.345
Tanh MLP 0 25.8
Table 4: Performance of dierent algorithms using a dataset with onehot vector and 80 binary
inputs as discussed in Experiment 4.
whereL1is the generalization error. Therefore online SGD is trying to minimize the
expected risk with noisy updates. Those noisy updates have the eect of regularizer:
t=PN
irtL(xt;t)
N=rtL(x;t) +t (8)
wherertL(x;t) is the true gradient and tis the zero-mean stochastic gradient \noise"
due to computing the gradient over a nite-size minibatch sample.
We would like to know if the problem with the Pentomino dataset is more a regularization


key: chroma:document
string_value:
due to computing the gradient over a nite-size minibatch sample.
We would like to know if the problem with the Pentomino dataset is more a regularization
or an optimization problem. An SMLP-nohints model was trained by online SGD with the randomly generated online Pentomino stream. The learning rate was adaptive, with the Adadelta
procedure (Zeiler, 2012) on minibatches of 100 examples. In the online SGD experiments, two
SMLP-nohints that is trained with and without standardization at the intermediate layer with
exactly the same hyperparameters are tested. The SMLP-nohints P1NN patch-wise submodel
has 2048 hidden units and the SMLP intermediate layer has 1152 = 64 18 hidden units. The
nonlinearity that is used for the intermediate layer is the sigmoid. P2NN has 2048 hidden units.
SMLP-nohints has been trained either with or without standardization on top of the output
units of the P1NN. The experiments illustrated in Figures 12 and 13 are with the same SMLP
without hints architecture for which results are given in Table 1. In those graphs only the results
for the training on the randomly generated 545400 Pentomino samples have been presented. As


key: chroma:document
string_value:
without hints architecture for which results are given in Table 1. In those graphs only the results
for the training on the randomly generated 545400 Pentomino samples have been presented. As
shown in the plots SMLP-nohints was not able to generalize without standardization. Although
without standardization the training loss seems to decrease initially, it eventually gets stuck in
a plateau where training loss doesn't change much.
Training of SMLP-nohints online minibatch SGD is performed using standardization in the
intermediate layer and Adadelta learning rate adaptation, on 1046000 training examples from
the randomly generated Pentomino stream. At the end of the training, test error is down to
27.5%, which is much better than chance but from from the score obtained with SMLP-hints
of near 0 error.
In another SMLP-nohints experiment without standardization the model is trained with the
1580000 Pentomino examples using online minibatch SGD. P1NN has 2048 hidden units and 16
sigmoidal outputs per patch. for the P1NN hidden layer. P2NN has 1024 hidden units for the
hidden layer. Adadelta is used to adapt the learning rate. At the end of training this SMLP,
the test error remained stuck, at 50.1%.
21
0 100 200 300 400 500 600 700 800


key: chroma:document
string_value:
the test error remained stuck, at 50.1%.
21
0 100 200 300 400 500 600 700 800
Batch no0.200.250.300.350.400.450.500.55Test Error
SMLP with standardization
SMLP without standardizationFigure 12: Test errors of SMLP-nohints with and without standardization in the intermediate
layer. Sigmoid as an intermediate layer activation has been used. Each tick (batch
no) in the x-axis represents 400 examples.
22
0 100 200 300 400 500 600 700
Batch no01234567Training LossSMLP with standardization
SMLP without standardizationFigure 13: Training errors of SMLP-nohints with and without standardization in the intermediate layer. Sigmoid nonlinearity has been used as an intermediate layer activation
function. The x-axis is in units of blocks of 400 examples in the training set.
23
3.4.1 Experiments with Increased Training Set Size
Here we consider the eect of training dierent learners with dierent numbers of training
examples. For the experimental results shown in Table 1, 3 training set sizes (20k, 40k and 80k
examples) had been used. Each dataset was generated with dierent random seeds (so they


key: chroma:document
string_value:
examples) had been used. Each dataset was generated with dierent random seeds (so they
do not overlap). Figure 14 also shows the error bars for an ordinary MLP with three hidden
layers, for a larger range of training set sizes, between 40k and 320k examples. The number of
training epochs is 8 (more did not help), and there are three hidden layers with 2048 feature
detectors. The learning rate we used in our experiments is 0.01. The activation function of the
MLP is a tanh nonlinearity, while the L1, L2 penalty coecients are both 1e-6.
Table 1 shows that, without guiding hints, none of the state-of-art learning algorithms could
perform noticeably better than a random predictor on the test set. This shows the importance
of intermediate hints introduced in the SMLP. The decision trees and SVMs can overt the
training set but they could not generalize on the test set. Note that the numbers reported in the
table are for hyper-parameters selected based on validation set error, hence lower training errors
are possible if avoiding all regularization and taking large enough models. On the training set,
the MLP with two large hidden layers (several thousands) could reach nearly 0% training error,
but still did not manage to achieve good test error.


key: chroma:document
string_value:
the MLP with two large hidden layers (several thousands) could reach nearly 0% training error,
but still did not manage to achieve good test error.
In the experiment results shown in Figure 14, we evaluate the impact of adding more training
data for the fully-connected MLP. As mentioned before for these experiments we have used a
MLP with three hidden layers where each layer has 2048 hidden units. The tanh( ) activation
function is used with 0.05 learning rate and minibatches of size 200.
As can be seen from the gure, adding more training examples did not help either training
or test error (both are near 50%, with training error slightly lower and test error slightly
higher), reinforcing the hypothesis that the dicult encountered is one of optimization, not of
regularization.
Figure 14: Training and test error bar charts for a regular MLP with 3 hidden layers. There
is no signicant improvement on the generalization error of the MLP as the new
training examples are introduced.
24
3.5 Experiments on Eect of Initializing with Hints
Initialization of the parameters in a neural network can have a big impact on the learning
and generalization (Glorot and Bengio, 2010). Previously Erhan et al. (2010) showed that


key: chroma:document
string_value:
and generalization (Glorot and Bengio, 2010). Previously Erhan et al. (2010) showed that
initializing the parameters of a neural network with unsupervised pretraining guides the learning towards basins of attraction of local minima that provides better generalization from the
training dataset. In this section we analyze the eect of initializing the SMLP with hints and
then continuing without hints at the rest of the training. For experimental analysis of hints
based initialization, SMLP is trained for 1 training epoch using the hints and for 60 epochs it
is trained without hints on the 40k examples training set. We also compared the same architecture with the same hyperparameters, against to SMLP-nohints trained for 61 iterations on
the same dataset. After one iteration of hint-based training SMLP obtained 9% training error
and 39% test error. Following the hint based training, SMLP is trained without hints for 60
epochs, but at epoch 18, it already got 0% training and 0% test error. The hyperparameters
for this experiment and the experiment that the results shown for the SMLP-hints in Table 1
are the same. The test results for initialization with and without hints are shown on Figure 15.
This gure suggests that initializing with hints can give the same generalization performance
but training takes longer.


key: chroma:document
string_value:
This gure suggests that initializing with hints can give the same generalization performance
but training takes longer.
Figure 15: Plots showing the test error of SMLP with random initialization vs initializing with
hint based training.
3.5.1 Further Experiments on Optimization for Pentomino Dataset
With extensive hyperparameter optimization and using standardization in the intermediate
level of the SMLP with softmax nonlinearity, SMLP-nohints was able to get 5.3% training and
25
6.7% test error on the 80k Pentomino training dataset. We used the 2050 hidden units for the
hidden layer of P1NN and 11 softmax output per patch. For the P2NN, we used 1024 hidden
units with sigmoid and learning rate 0.1 without using any adaptive learning rate method. This
SMLP uses a rectier nonlinearity for hidden layers of both P1NN and P2NN. Considering that
architecture uses softmax as the intermediate activation function of SMLP-nohints. It is very
likely that P1NN is trying to learn the presence of specic Pentomino shape in a given patch.
This architecture has a very large capacity in the P1NN, that probably provides it enough


key: chroma:document
string_value:
This architecture has a very large capacity in the P1NN, that probably provides it enough
capacity to learn the presence of Pentomino shapes at each patch eortlessly.
An MLP with 2 hidden layers, each 1024 rectier units, was trained using LBFGS (the
implementation from the scipy.optimize library) on 40k training examples, with gradients computed on batches of 10000 examples at each iteration. However, after convergence of training,
the MLP was still doing chance on the test dataset.
We also observed that using linear units for the intermediate layer yields better generalization error without standardization compared to using activation functions such as sigmoid,
tanh and RELU for the intermediate layer. SMLP-nohints was able to get 25% generalization error with linear units without standardization whereas all the other activation functions
that has been tested failed to generalize with the same number of training iterations without
standardization and hints. This suggests that using non-linear intermediate-level activation
functions without standardization introduces an optimization diculty for the SMLP-nohints,
maybe because the intermediate level acts like a bottleneck in this architecture.
4. Conclusion and Discussion
In this paper we have shown an example of task which seems almost impossible to solve by
standard black-box machine learning algorithms, but can be almost perfectly solved when


key: chroma:document
string_value:
4. Conclusion and Discussion
In this paper we have shown an example of task which seems almost impossible to solve by
standard black-box machine learning algorithms, but can be almost perfectly solved when
one encourages a semantics for the intermediate-level representation that is guided by prior
knowledge. The task has the particularity that it is dened by the composition of two nonlinear sub-tasks (object detection on one hand, and a non-linear logical operation similar to
XOR on the other hand).
What is interesting is that in the case of the neural network, we can compare two networks
with exactly the same architecture but a dierent pre-training, one of which uses the known
intermediate concepts to teach an intermediate representation to the network. With enough
capacity and training time they can overt but did not not capture the essence of the task, as
seen by test set performance.
We know that a structured deep network can learn the task, if it is initialized in the right
place, and do it from very few training examples. Furthermore we have shown that if one
pre-trains SMLP with hints for only one epoch, it can nail the task. But the exactly same
architecture which started training from random initialization, failed to generalize.
Consider the fact that even SMLP-nohints with standardization after being trained using


key: chroma:document
string_value:
architecture which started training from random initialization, failed to generalize.
Consider the fact that even SMLP-nohints with standardization after being trained using
online SGD on 1046000 generated examples and still gets 27.5% test error. This is an indication
that the problem is not a regularization problem but possibly an inability to nd a good eective
local minima of generalization error .
What we hypothesize is that for most initializations and architectures (in particular the
fully-connected ones), although it is possible to nd a good eective local minimum of training
error when enough capacity is provided, it is dicult (without the proper initialization) to nd a
good local minimum of generalization error. On the other hand, when the network architecture
26
is constrained enough but still allows it to represent a good solution (such as the structured
MLP of our experiments), it seems that the optimization problem can still be dicult and even
training error remains stuck high if the standardization isn't used. Standardization obviously
makes the training objective of the SMLP easier to optimize and helps it to nd at least a
better eective local minimum of training error . This nding suggests that by using specic
architectural constraints and sometimes domain specic knowledge about the problem, one can


key: chroma:document
string_value:
architectural constraints and sometimes domain specic knowledge about the problem, one can
alleviate the optimization diculty that generic neural network architectures face.
It could be that the combination of the network architecture and training procedure produces a training dynamics that tends to yield into these minima that are poor from the point of
view of generalization error, even when they manage to nail training error by providing enough
capacity. Of course, as the number of examples increases, we would expect this discrepancy to
decrease, but then the optimization problem could still make the task unfeasible in practice.
Note however that our preliminary experiments with increasing the training set size (8-fold) for
MLPs did not reveal signs of potential improvements in test error yet, as shown in Figure 14.
Even using online training on 545400 Pentomino examples, the SMLP-nohints architecture was
still doing far from perfect in terms of generalization error (Figure 12).
These ndings bring supporting evidence to the \Guided Learning Hypothesis" and \Deeper
Harder Hypothesis" from Bengio (2013a): higher level abstractions, which are expressed by
composing simpler concepts, are more dicult to learn (with the learner often getting in an
eective local minimum ), but that diculty can be overcome if another agent provides hints


key: chroma:document
string_value:
eective local minimum ), but that diculty can be overcome if another agent provides hints
of the importance of learning other, intermediate-level abstractions which are relevant to the
task.
Many interesting questions remain open. Would a network without any guiding hint eventually nd the solution with a enough training time and/or with alternate parametrizations?
To what extent is ill-conditioning a core issue? The results with LBFGS were disappointing but
changes in the architectures (such as standardization of the intermediate level) seem to make
training much easier. Clearly, one can reach good solutions from an appropriate initialization,
pointing in the direction of an issue with local minima, but it may be that good solutions are
also reachable from other initializations, albeit going through a tortuous ill-conditioned path
in parameter space. Why did our attempts at learning the intermediate concepts in an unsupervised way fail? Are these results specic to the task we are testing or a limitation of the
unsupervised feature learning algorithm tested? Trying with many more unsupervised variants and exploring explanatory hypotheses for the observed failures could help us answer that.
Finally, and most ambitious, can we solve these kinds of problems if we allow a community
of learners to collaborate and collectively discover and combine partial solutions in order to
obtain solutions to more abstract tasks like the one presented here? Indeed, we would like to


key: chroma:document
string_value:
of learners to collaborate and collectively discover and combine partial solutions in order to
obtain solutions to more abstract tasks like the one presented here? Indeed, we would like to
discover learning algorithms that can solve such tasks without the use of prior knowledge as
specic and strong as the one used in the SMLP here. These experiments could be inspired by
and inform us about potential mechanisms for collective learning through cultural evolutions
in human societies.
Acknowledgments
We would like to thank to the ICLR 2013 reviewers for their insightful comments, and NSERC,
CIFAR, Compute Canada and Canada Research Chairs for funding.
27
References
A. Ben-Hur and J. Weston. A user's guide to support vector machines. Methods in Molecular
Biology , 609:223{239, 2010.
Y. Bengio, P. Lamblin, D. Popovici, and H. Larochelle. Greedy layer-wise training of deep
networks. In NIPS'2006 , 2007.
Yoshua Bengio. Learning deep architectures for AI. Foundations and Trends in Machine
Learning , 2(1):1{127, 2009. Also published as a book. Now Publishers, 2009.
Yoshua Bengio. Evolving culture vs local minima. In Growing Adaptive Machines: Integrating


key: chroma:document
string_value:
Yoshua Bengio. Evolving culture vs local minima. In Growing Adaptive Machines: Integrating
Development and Learning in Articial Neural Networks , number also as ArXiv 1203.2990v1,
pages T. Kowaliw, N. Bredeche & R. Doursat, eds. Springer-Verlag, March 2013a. URL
http://arxiv.org/abs/1203.2990 .
Yoshua Bengio. Practical recommendations for gradient-based training of deep architectures.
In K.-R. M uller, G. Montavon, and G. B. Orr, editors, Neural Networks: Tricks of the Trade .
Springer, 2013b.
Yoshua Bengio, Jerome Louradour, Ronan Collobert, and Jason Weston. Curriculum learning.
In L eon Bottou and Michael Littman, editors, Proceedings of the Twenty-sixth International
Conference on Machine Learning (ICML'09) . ACM, 2009a.
Yoshua Bengio, Jerome Louradour, Ronan Collobert, and Jason Weston. Curriculum learning.
InICML'09 , 2009b.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Unsupervised feature learning and deep


key: chroma:document
string_value:
InICML'09 , 2009b.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Unsupervised feature learning and deep
learning: A review and new perspectives. Technical Report arXiv:1206.5538, U. Montreal,
2012. URL http://arxiv.org/abs/1206.5538 .
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Unsupervised feature learning and deep
learning: A review and new perspectives. IEEE Trans. Pattern Analysis and Machine Intelligence (PAMI) , 2013.
James Bergstra, Olivier Breuleux, Fr ed eric Bastien, Pascal Lamblin, Razvan Pascanu, Guillaume Desjardins, Joseph Turian, David Warde-Farley, and Yoshua Bengio. Theano: a CPU
and GPU math expression compiler. In Proceedings of the Python for Scientic Computing
Conference (SciPy) , 2010.
L eon Bottou. Large-scale machine learning with stochastic gradient descent. In Proceedings of
COMPSTAT'2010 , pages 177{186. Springer, 2010.
Leo Breiman. Random forests. Machine Learning , 45(1):5{32, 2001.


key: chroma:document
string_value:
COMPSTAT'2010 , pages 177{186. Springer, 2010.
Leo Breiman. Random forests. Machine Learning , 45(1):5{32, 2001.
D. C. Ciresan, U. Meier, L. M. Gambardella, and J. Schmidhuber. Deep big simple neural nets
for handwritten digit recognition. Neural Computation , 22:1{14, 2010.
Yann Dauphin and Yoshua Bengio. Big neural networks waste capacity. Technical Report
arXiv:1301.3583, Universite de Montreal, 2013.
Richard Dawkins. The Selsh Gene . Oxford University Press, 1976.
28
J. Duchi, E. Hazan, and Y. Singer. Adaptive subgradient methods for online learning and
stochastic optimization. Journal of Machine Learning Research , 12:2121{2159, 2010.
Dumitru Erhan, Yoshua Bengio, Aaron Courville, Pierre-Antoine Manzagol, Pascal Vincent,
and Samy Bengio. Why does unsupervised pre-training help deep learning? Journal of
Machine Learning Research , 11:625{660, February 2010.
Fran cois Fleuret, Ting Li, Charles Dubout, Emma K Wampler, Steven Yantis, and Donald


key: chroma:document
string_value:
Fran cois Fleuret, Ting Li, Charles Dubout, Emma K Wampler, Steven Yantis, and Donald
Geman. Comparing machines and humans on a visual categorization test. Proceedings of the
National Academy of Sciences , 108(43):17621{17625, 2011.
X. Glorot, A. Bordes, and Y. Bengio. Deep sparse rectier neural networks. In AISTATS ,
2011a.
Xavier Glorot and Yoshua Bengio. Understanding the diculty of training deep feedforward
neural networks. In JMLR W&CP: Proceedings of the Thirteenth International Conference
on Articial Intelligence and Statistics (AISTATS 2010) , volume 9, pages 249{256, May
2010.
Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectier neural networks. In
JMLR W&CP: Proceedings of the Fourteenth International Conference on Articial Intelligence and Statistics (AISTATS 2011) , April 2011b.
Ian J. Goodfellow, David Warde-Farley, Mehdi Mirza, Aaron Courville, and Yoshua Bengio.
Maxout networks. In ICML , 2013.


key: chroma:document
string_value:
Ian J. Goodfellow, David Warde-Farley, Mehdi Mirza, Aaron Courville, and Yoshua Bengio.
Maxout networks. In ICML , 2013.
Maciej Henneberg. Decrease of human skull size in the holocene. Human biology , pages 395{405,
1988.
Maciej Henneberg and Maryna Steyn. Trends in cranial capacity and cranial index in subsaharan africa during the holocene. American journal of human biology , 5(4):473{479, 1993.
J. Henrich and R. McElreath. The evolution of cultural evolution. Evolutionary Anthropology:
Issues, News, and Reviews , 12(3):123{135, 2003.
Georey E. Hinton, Simon Osindero, and Yee Whye Teh. A fast learning algorithm for deep
belief nets. Neural Computation , 18:1527{1554, 2006.
Georey E. Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Improving neural networks by preventing co-adaptation of feature detectors. Technical
report, arXiv:1207.0580, 2012.


key: chroma:document
string_value:
report, arXiv:1207.0580, 2012.
C.W. Hsu, C.C. Chang, C.J. Lin, et al. A practical guide to support vector classication, 2003.
Kevin Jarrett, Koray Kavukcuoglu, Marc'Aurelio Ranzato, and Yann LeCun. What is the
best multi-stage architecture for object recognition? In Proc. International Conference on
Computer Vision (ICCV'09) , pages 2146{2153. IEEE, 2009a.
Kevin Jarrett, Koray Kavukcuoglu, Marc'Aurelio Ranzato, and Yann LeCun. What is the best
multi-stage architecture for object recognition? In ICCV'09 , 2009b.
29
Faisal Khan, Xiaojin Zhu, and Bilge Mutlu. How do humans teach: On curriculum learning and
teaching dimension. In Advances in Neural Information Processing Systems 24 (NIPS'11) ,
pages 1449{1457, 2011.
Alex Krizhevsky, Ilya Sutskever, and Georey Hinton. ImageNet classication with deep
convolutional neural networks. In Advances in Neural Information Processing Systems 25
(NIPS'2012) . 2012.


key: chroma:document
string_value:
convolutional neural networks. In Advances in Neural Information Processing Systems 25
(NIPS'2012) . 2012.
Kai A. Krueger and Peter Dayan. Flexible shaping: how learning in small steps helps. Cognition ,
110:380{394, 2009.
G. Kunapuli, K.P. Bennett, R. Maclin, and J.W. Shavlik. The adviceptron: Giving advice to
the perceptron. Proceedings of the Conference on Articial Neural Networks In Engineering
(ANNIE 2010) , 2010.
Hugo Larochelle, Yoshua Bengio, Jerome Louradour, and Pascal Lamblin. Exploring strategies
for training deep neural networks. Journal of Machine Learning Research , 10:1{40, 2009.
Y. LeCun, L. Bottou, Y. Bengio, and P. Haner. Gradient-based learning applied to document
recognition. Proceedings of the IEEE , 86(11):2278{2324, 1998.
T.M. Mitchell. The need for biases in learning generalizations . Department of Computer
Science, Laboratory for Computer Science Research, Rutgers Univ., 1980.
T.M. Mitchell and S.B. Thrun. Explanation-based neural network learning for robot control.
Advances in Neural information processing systems , pages 287{287, 1993.


key: chroma:document
string_value:
T.M. Mitchell and S.B. Thrun. Explanation-based neural network learning for robot control.
Advances in Neural information processing systems , pages 287{287, 1993.
R. Montague. Universal grammar. Theoria , 36(3):373{398, 1970.
V. Nair and G. E Hinton. Rectied linear units improve restricted Boltzmann machines. In
ICML'10 , 2010.
L.B.J.H.F.R.A. Olshen and C.J. Stone. Classication and regression trees. Belmont, Calif.:
Wadsworth , 1984.
Joseph O'Sullivan. Integrating initialization bias and search bias in neural network learning,
1996.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel,
P. Prettenhofer, R. Weiss, V. Dubourg, et al. Scikit-learn: Machine learning in python. The
Journal of Machine Learning Research , 12:2825{2830, 2011.
Gail B. Peterson. A day of great illumination: B. F. Skinner's discovery of shaping. Journal of
the Experimental Analysis of Behavior , 82(3):317{328, 2004.


key: chroma:document
string_value:
the Experimental Analysis of Behavior , 82(3):317{328, 2004.
Tapani Raiko, Harri Valpola, and Yann LeCun. Deep learning made easier by linear transformations in perceptrons. In International Conference on Articial Intelligence and Statistics ,
pages 924{932, 2012.
Salah Rifai, Pascal Vincent, Xavier Muller, Xavier Glorot, and Yoshua Bengio. Contractive
auto-encoders: Explicit invariance during feature extraction. In ICML'2011 , 2011.
30
Salah Rifai, Yoshua Bengio, Yann Dauphin, and Pascal Vincent. A generative process for sampling contractive auto-encoders. In Proceedings of the Twenty-nine International Conference
on Machine Learning (ICML'12) . ACM, 2012. URL http://icml.cc/discuss/2012/590.
html .
R. Salakhutdinov and G.E. Hinton. Deep Boltzmann machines. In Proceedings of the Twelfth
International Conference on Articial Intelligence and Statistics (AISTATS 2009) , volume 8,
2009.
Burrhus F. Skinner. Reinforcement today. American Psychologist , 13:94{99, 1958.


key: chroma:document
string_value:
2009.
Burrhus F. Skinner. Reinforcement today. American Psychologist , 13:94{99, 1958.
R.J. Solomono. A system for incremental learning based on algorithmic probability. In Proceedings of the Sixth Israeli Conference on Articial Intelligence, Computer Vision and Pattern
Recognition , pages 515{527. Citeseer, 1989.
G.G. Towell and J.W. Shavlik. Knowledge-based articial neural networks. Articial intelligence , 70(1):119{165, 1994.
Tommi Vatanen, Tapani Raiko, Harri Valpola, and Yann LeCun. Pushing stochastic gradient
towards second-order methods{backpropagation learning with transformations in nonlinearities. arXiv preprint arXiv:1301.3476 , 2013.
Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, and Pierre-Antoine Manzagol.
Stacked denoising autoencoders: Learning useful representations in a deep network with a
local denoising criterion. Journal of Machine Learning Research , 11:3371{3408, December
2010.
Luis Von Ahn, Manuel Blum, Nicholas J Hopper, and John Langford. Captcha: Using hard


key: chroma:document
string_value:
2010.
Luis Von Ahn, Manuel Blum, Nicholas J Hopper, and John Langford. Captcha: Using hard
ai problems for security. In Advances in CryptologyEUROCRYPT 2003 , pages 294{311.
Springer, 2003.
Jason Weston, Fr ed eric Ratle, and Ronan Collobert. Deep learning via semi-supervised embedding. In William W. Cohen, Andrew McCallum, and Sam T. Roweis, editors, Proceedings of the Twenty-fth International Conference on Machine Learning (ICML'08) ,
pages 1168{1175, New York, NY, USA, 2008. ACM. ISBN 978-1-60558-205-4. doi:
10.1145/1390156.1390303.
Matthew D Zeiler. Adadelta: An adaptive learning rate method. arXiv preprint
arXiv:1212.5701 , 2012.
5. Appendix
5.1 Binary-Binary RBMs on Pentomino Dataset
We trained binary-binary RBMs (both visible and hidden are binary) on 8 8 patches extracted
from the Pentomino Dataset using PCD (stochastic maximum likelihood), a weight decay of
.0001 and a sparsity penalty9. We used 256 hidden units and trained by SGD with a batch


key: chroma:document
string_value:
.0001 and a sparsity penalty9. We used 256 hidden units and trained by SGD with a batch
size of 32 and a annealing learning rate (Bengio, 2013b) starting from 1e-3 with annealing rate
9. implemented as TorontoSparsity in pylearn2, see the yaml le in the repository for more details
31
1.000015. The RBM is trained with momentum starting from 0.5. The biases are initialized to
-2 in order to get a sparse representation. The RBM is trained for 120 epochs (approximately
50 million updates).
After pretraining the RBM, its parameters are used to initialize the rst layer of an SMLPnohints network. As in the usual architecture of the SMLP-nohints on top of P1NN, there is
an intermediate layer. Both P1NN and the intermediate layer have a sigmoid nonlinearity, and
the intermediate layer has 11 units per location. This SMLP-nohints is trained with Adadelta
and standardization at the intermediate layer10.
0 5 10 15 20 25 30 35 40 45
Epoch0.00.10.20.30.40.5Error percentageRBM Test and Training Errors
Training Error
Test Error


key: chroma:document
string_value:
0 5 10 15 20 25 30 35 40 45
Epoch0.00.10.20.30.40.5Error percentageRBM Test and Training Errors
Training Error
Test Error
Figure 16: Training and test errors of an SMLP-nohints network whose rst layer is pre-trained
as an RBM. Training error reduces to 0% at epoch 42, but test error is still chance.
5.2 Experimental Setup and Hyper-parameters
5.2.1 Decision Trees
We used the decision tree implementation in the scikit-learn (Pedregosa et al., 2011) python
package which is an implementation of the CART (Regression Trees) algorithm. The CART
algorithm constructs the decision tree recursively and partitions the input space such that the
samples belonging to the same category are grouped together (Olshen and Stone, 1984). We
used The Gini index as the impurity criteria. We evaluated the hyper-parameter congurations with a grid-search. We cross-validated the maximum depth ( maxdepth ) of the tree (for
preventing the algorithm to severely overt the training set) and minimum number of samples
10. In our auto-encoder experiments we directly fed features to P2NN without standardization and Adadelta.
32


key: chroma:document
string_value:
10. In our auto-encoder experiments we directly fed features to P2NN without standardization and Adadelta.
32
Figure 17: Filters learned by the binary-binary RBM after training on the 40k examples. The
RBM did learn the edge structure of Pentomino shapes.
Figure 18: 100 samples generated from trained RBM. All the generated samples are valid Pentomino shapes.
33
required to create a split ( minsplit). 20 dierent congurations of hyper-parameter values were
evaluated. We obtained the best validation error with maxdepth = 300 andminsplit = 8.
5.2.2 Support Vector Machines
We used the \Support Vector Classier (SVC)" implementation from the scikit-learn package
which in turn uses the libsvm's Support Vector Machine (SVM) implementation. Kernelbased SVMs are non-parametric models that map the data into a high dimensional space and
separate dierent classes with hyperplane(s) such that the support vectors for each category
will be separated by a large margin. We cross-validated three hyper-parameters of the model
using grid-search: C,
and the type of kernel( kerneltype).Cis the penalty term (weight


key: chroma:document
string_value:
using grid-search: C,
and the type of kernel( kerneltype).Cis the penalty term (weight
decay) for the SVM and 
is a hyper-parameter that controls the width of the Gaussian for
the RBF kernel. For the polynomial kernel, 
controls the 
exibility of the classier (degree
of the polynomial) as the number of parameters increases (Hsu et al., 2003; Ben-Hur and
Weston, 2010). We evaluated forty-two hyper-parameter congurations. That includes, two
kernel types:fRBF; Polynomial g; three gammas:f1e2;1e3;1e4gfor the RBF kernel,
f1;2;5gfor the polynomial kernel, and seven Cvalues among:f0:1;1;2;4;8;10;16g. As a
result of the grid search and cross-validation, we have obtained the best test error by using the
RBF kernel, with C= 2 and
= 1.
5.2.3 Multi Layer Perceptron
We have our own implementation of Multi Layer Perceptron based on the Theano (Bergstra
et al., 2010) machine learning libraries. We have selected 2 hidden layers, the rectier activation


key: chroma:document
string_value:
et al., 2010) machine learning libraries. We have selected 2 hidden layers, the rectier activation
function, and 2048 hidden units per layer. We cross-validated three hyper-parameters of the
model using random-search, sampling the learning rates in log-domain, and selecting L1
andL2 regularization penalty coecients in sets of xed values, evaluating 64 hyperparameter
values. The range of the hyperparameter values are 2[0:0001;1],L12f0:;1e6;1e5;1e4g
andL22f0;1e6;1e5g. As a result, the following were selected: L1 = 1e6,L2 = 1e5
and= 0:05.
5.2.4 Random Forests
We used scikit-learn's implementation of \Random Forests" decision tree learning. The Random Forests algorithm creates an ensemble of decision trees by randomly selecting for each tree
a subset of features and applying bagging to combine the individual decision trees (Breiman,
2001). We have used grid-search and cross-validated the maxdepth ,minsplit, and number


key: chroma:document
string_value:
2001). We have used grid-search and cross-validated the maxdepth ,minsplit, and number
of trees (nestimators ). We have done the grid-search on the following hyperparameter values,nestimators2 f5;10;15;25;50g,maxdepth2 f100;300;600;900g, andminsplits2
f1;4;16g. We obtained the best validation error with maxdepth = 300,minsplit = 4 and
nestimators = 10.
5.2.5 k-Nearest Neighbors
We used scikit-learn's implementation of k-Nearest Neighbors (k-NN). k-NN is an instancebased, lazy learning algorithm that selects the training examples closest in Euclidean distance
to the input query. It assigns a class label to the test example based on the categories of the
kclosest neighbors. The hyper-parameters we have evaluated in the cross-validation are the
number of neighbors ( k) andweights . Theweights hyper-parameter can be either \uniform" or
34
\distance". With \uniform", the value assigned to the query point is computed by the majority
vote of the nearest neighbors. With \distance", each value assigned to the query point is
computed by weighted majority votes where the weights are computed with the inverse distance


key: chroma:document
string_value:
vote of the nearest neighbors. With \distance", each value assigned to the query point is
computed by weighted majority votes where the weights are computed with the inverse distance
between the query point and the neighbors. We have used nneighbours2f1;2;4;6;8;12gand
weights2f"uniform ";"distance "gfor hyper-parameter search. As a result of cross-validation
and grid search, we obtained the best validation error with k= 2 andweights =\uniform".
5.2.6 Convolutional Neural Nets
We used a Theano (Bergstra et al., 2010) implementation of Convolutional Neural Networks
(CNN) from the deep learning tutorial at deeplearning.net , which is based on a vanilla
version of a CNN LeCun et al. (1998). Our CNN has two convolutional layers. Following
each convolutional layer, we have a max-pooling layer. On top of the convolution-poolingconvolution-pooling layers there is an MLP with one hidden layer. In the cross-validation we
have sampled 36 learning rates in log-domain in the range [0 :0001;1] and the number of lters


key: chroma:document
string_value:
have sampled 36 learning rates in log-domain in the range [0 :0001;1] and the number of lters
from the range [10 ;20;30;40;50;60] uniformly. For the rst convolutional layer we used 9 9
receptive elds in order to guarantee that each object ts inside the receptive eld. As a result
of random hyperparameter search and doing manual hyperparameter search on the validation
dataset, the following values were selected:
The number of features used for the rst layer is 30 and the second layer is 60.
For the second convolutional layer, 7 7 receptive elds. The stride for both convolutional
layers is 1.
Convolved images are downsampled by a factor of 2 2 at each pooling operation.
The learning rate for CNN is 0.01 and it was trained for 8 epochs.
5.2.7 Maxout Convolutional Neural Nets
We used the pylearn2 ( https://github.com/lisa-lab/pylearn2 ) implementation of maxout
convolutional networks (Goodfellow et al., 2013). There are two convolutional layers in the


key: chroma:document
string_value:
convolutional networks (Goodfellow et al., 2013). There are two convolutional layers in the
selected architecture, without any pooling. In the last convolutional layer, there is a maxout
non-linearity. The following were selected by cross-validation: learning rate, number of channels
for the both convolution layers, number of kernels for the second layer and number of units and
pieces per maxout unit in the last layer, a linearly decaying learning rate, momentum starting
from 0.5 and saturating to 0.8 at the 200'th epoch. Random search for the hyperparameters
was used to evaluate 48 dierent hyperparameter congurations on the validation dataset. For
the rst convolutional layer, 8 8 kernels were selected to make sure that each Pentomino shape
ts into the kernel. Early stopping was used and test error on the model that has the best
validation error is reported. Using norm constraint on the fan-in of the nal softmax units
yields slightly better result on the validation dataset.
As a result of cross-validation and manually tuning the hyperparameters we used the following hyperparameters:
16 channels per convolutional layer. 600 hidden units for the maxout layer.
6x6 kernels for the second convolutional layer.
35


key: chroma:document
string_value:
16 channels per convolutional layer. 600 hidden units for the maxout layer.
6x6 kernels for the second convolutional layer.
35
5 pieces for the convolution layers and 4 pieces for the maxout layer per maxout units.
We decayed the learning rate by the factor of 0.001 and the initial learning rate is 0.026367.
But we scaled the learning rate of the second convolutional layer by a constant factor of
0.6.
The norm constraint (on the incoming weights of each unit) is 1.9365.
Figure 19 shows the rst layer lters of the maxout convolutional net, after being trained
on the 80k training set for 85 epochs.
Figure 19: Maxout convolutional net rst layer lters. Most of the lters were able to learn
the basic edge structure of the Pentomino shapes.
5.2.8 Stacked Denoising Auto-Encoders
Denoising Auto-Encoders (DAE) are a form of regularized auto-encoder (Bengio et al., 2013).
The DAE forces the hidden layer to discover more robust features and prevents it from simply
learning the identity by reconstructing the input from a corrupted version of it (Vincent et al.,


key: chroma:document
string_value:
The DAE forces the hidden layer to discover more robust features and prevents it from simply
learning the identity by reconstructing the input from a corrupted version of it (Vincent et al.,
2010). Two DAEs were stacked, resulting in an unsupervised transformation with two hidden
layers of 1024 units each. Parameters of all layers are then ne-tuned with supervised netuning using logistic regression as the classier and SGD as the gradient-based optimization
algorithm. The stochastic corruption process is binomial (0 or 1 replacing each input value,
with probability 0.2). The selected learning rate is 0= 0:01 for the DAe and 1= 0:1 for
supervised ne-tuning. Both L1 and L2 penalty for the DAEs and for the logistic regression
layer are set to 1e-6.
CAE+MLP with Supervised Finetuning: A regularized auto-encoder which sometimes
outperforms the DAE is the Contractive Auto-Encoder (CAE), (Rifai et al., 2012), which
penalizes the Frobenius norm of the Jacobian matrix of derivatives of the hidden units with
respect to the CAE's inputs. The CAE serves as pre-training for an MLP, and in the supervised


key: chroma:document
string_value:
respect to the CAE's inputs. The CAE serves as pre-training for an MLP, and in the supervised
ne-tuning state, the Adagrad method was used to automatically tune the learning rate (Duchi
et al., 2010).
After training a CAE with 100 sigmoidal units patch-wise, the features extracted on each
patch are concatenated and fed as input to an MLP. The selected Jacobian penalty coecient
is 2, the learning rate for pre-training is 0.082 with batch size of 200 and 200 epochs of unsupervised learning are performed on the training set. For supervised netuning, the learning
rate is 0.12 over 100 epochs, L1 and L2 regularization penalty terms respectively are 1e-4 and
1e-6, and the top-level MLP has 6400 hidden units.
36
Greedy Layerwise CAE+DAE Supervised Finetuning: For this experiment we stack a
CAE with sigmoid non-linearities and then a DAE with rectier non-linearities during the pretraining phase. As recommended by Glorot et al. (2011b) we have used a softplus nonlinearity
for reconstruction, softplus (x) =log(1 +ex). We used an L1 penalty on the rectier outputs


key: chroma:document
string_value:
for reconstruction, softplus (x) =log(1 +ex). We used an L1 penalty on the rectier outputs
to obtain a sparser representation with rectier non-linearity and L2 regularization to keep the
non-zero weights small.
The main dierence between the DAE and CAE is that the DAE yields more robust reconstruction whereas the CAE obtains more robust features (Rifai et al., 2011).
As seen on Figure 7 the weights U and V are shared on each patch and we concatenate the
outputs of the last auto-encoder on each patch to feed it as an input to an MLP with a large
hidden layer.
We used 400 hidden units for the CAE and 100 hidden units for DAE. The learning rate
used for the CAE is 0.82 and for DAE it is 9*1e-3. The corruption level for the DAE (binomial
noise) is 0.25 and the contraction level for the CAE is 2.0. The L1 regularization penalty for
the DAE is 2.25*1e-4 and the L2 penalty is 9.5*1e-5. For the supervised netuning phase the


key: chroma:document
string_value:
the DAE is 2.25*1e-4 and the L2 penalty is 9.5*1e-5. For the supervised netuning phase the
learning rate used is 4*1e-4 with L1 and L2 penalties respectively 1e-5 and 1e-6. The top-level
MLP has 6400 hidden units. The auto-encoders are each trained for 150 epochs while the whole
MLP is ne-tuned for 50 epochs.
Greedy Layerwise DAE+DAE Supervised Finetuning: For this architecture, we have
trained two layers of denoising auto-encoders greedily and performed supervised netuning
after unsupervised pre-training. The motivation for using two denoising auto-encoders is the
fact that rectier nonlinearities work well with the deep networks but it is dicult to train
CAEs with the rectier non-linearity. We have used the same type of denoising auto-encoder
that is used for the greedy layerwise CAE+DAE supervised netuning experiment.
In this experiment we have used 400 hidden units for the rst layer DAE and 100 hidden
units for the second layer DAE. The other hyperparameters for DAE and supervised netuning


key: chroma:document
string_value:
units for the second layer DAE. The other hyperparameters for DAE and supervised netuning
are the same as with the CAE+DAE MLP Supervised Finetuning experiment.
37


key: chroma:document
string_value:
DeepPose: Human Pose Estimation via Deep Neural Networks
Alexander Toshev Christian Szegedy
Google
1600 Amphitheatre Pkwy
Mountain View, CA 94043
toshev,szegedy@google.com
Figure 1. Besides extreme variability in articulations, many of the
joints are barely visible. We can guess the location of the right
arm in the left image only because we see the rest of the pose and
anticipate the motion or activity of the person. Similarly, the left
body half of the person on the right is not visible at all. These
are examples of the need for holistic reasoning . We believe that
DNNs can naturally provide such type of reasoning.
Abstract
We propose a method for human pose estimation based
on Deep Neural Networks (DNNs). The pose estimation
is formulated as a DNN-based regression problem towards
body joints. We present a cascade of such DNN regressors which results in high precision pose estimates. The
approach has the advantage of reasoning about pose in a
holistic fashion and has a simple but yet powerful formulation which capitalizes on recent advances in Deep Learning. We present a detailed empirical analysis with state-ofart or better performance on four academic benchmarks of
diverse real-world images.
1. Introduction
The problem of human pose estimation, deﬁned as the


key: chroma:document
string_value:
diverse real-world images.
1. Introduction
The problem of human pose estimation, deﬁned as the
problem of localization of human joints, has enjoyed substantial attention in the computer vision community. In
Fig. 1, one can see some of the challenges of this problem – strong articulations, small and barely visible joints,
occlusions and the need to capture the context.
The main stream of work in this ﬁeld has been motivatedmainly by the ﬁrst challenge, the need to search in the large
space of all possible articulated poses. Part-based models
lend themselves naturally to model articulations ([16, 8])
and in the recent years a variety of models with efﬁcient
inference have been proposed ([6, 19]).
The above efﬁciency, however, is achieved at the cost of
limited expressiveness – the use of local detectors, which
reason in many cases about a single part, and most importantly by modeling only a small subset of all interactions
between body parts. These limitations, as exempliﬁed in
Fig. 1, have been recognized and methods reasoning about
pose in a holistic manner have been proposed [15, 21] but
with limited success in real-world problems.
In this work we ascribe to this holistic view of human
pose estimation. We capitalize on recent developments of


key: chroma:document
string_value:
with limited success in real-world problems.
In this work we ascribe to this holistic view of human
pose estimation. We capitalize on recent developments of
deep learning and propose a novel algorithm based on a
Deep Neural Network (DNN). DNNs have shown outstanding performance on visual classiﬁcation tasks [14] and more
recently on object localization [23, 9]. However, the question of applying DNNs for precise localization of articulated
objects has largely remained unanswered. In this paper we
attempt to cast a light on this question and present a simple
and yet powerful formulation of holistic human pose estimation as a DNN.
We formulate the pose estimation as a joint regression
problem and show how to successfully cast it in DNN settings. The location of each body joint is regressed to using
as an input the full image and a 7-layered generic convolutional DNN. There are two advantages of this formulation.
First, the DNN is capable of capturing the full context of
each body joint – each joint regressor uses the full image
as a signal. Second, the approach is substantially simpler
to formulate than methods based on graphical models – no
need to explicitly design feature representations and detectors for parts; no need to explicitly design a model topology
and interactions between joints. Instead, we show that a
generic convolutional DNN can be learned for this problem.


key: chroma:document
string_value:
and interactions between joints. Instead, we show that a
generic convolutional DNN can be learned for this problem.
Further, we propose a cascade of DNN-based pose predictors. Such a cascade allows for increased precision of
1arXiv:1312.4659v3  [cs.CV]  20 Aug 2014
joint localization. Starting with an initial pose estimation,
based on the full image, we learn DNN-based regressors
which reﬁnes the joint predictions by using higher resolution sub-images.
We show state-of-art results or better than state-of-art on
four widely used benchmarks against all reported results.
We show that our approach performs well on images of people which exhibit strong variation in appearance as well as
articulations. Finally, we show generalization performance
by cross-dataset evaluation.
2. Related Work
The idea of representing articulated objects in general,
and human pose in particular, as a graph of parts has been
advocated from the early days of computer vision [16]. The
so called Pictorial Strictures (PSs), introduced by Fishler
and Elschlager [8], were made tractable and practical by
Felzenszwalb and Huttenlocher [6] using the distance transform trick. As a result, a wide variety of PS-based models


key: chroma:document
string_value:
Felzenszwalb and Huttenlocher [6] using the distance transform trick. As a result, a wide variety of PS-based models
with practical signiﬁcance were subsequently developed.
The above tractability, however, comes with the limitation of having a tree-based pose models with simple binary
potential not depending on image data. As a result, research
has focused on enriching the representational power of the
models while maintaining tractability. Earlier attempts to
achieve this were based on richer part detectors [19, 1, 4].
More recently, a wide variety of models expressing complex
joint relationships were proposed. Yang and Ramanan [27]
use a mixture model of parts. Mixture models on the full
model scale, by having mixture of PSs, have been studied
by Johnson and Everingham [13]. Richer higher-order spatial relationships were captured in a hierarchical model by
Tian et al. [25]. A different approach to capture higherorder relationship is through image-dependent PS models,
which can be estimated via a global classiﬁer [26, 20, 18].
Approaches which ascribe to our philosophy of reasoning about pose in a holistic manner have shown limited
practicality. Mori and Malik [15] try to ﬁnd for each test
image the closest exemplar from a set of labeled images


key: chroma:document
string_value:
practicality. Mori and Malik [15] try to ﬁnd for each test
image the closest exemplar from a set of labeled images
and transfer the joint locations. A similar nearest neighbor
setup is employed by Shakhnarovich et al. [21], who however use locality sensitive hashing. More recently, Gkioxari
et al. [10] propose a semi-global classiﬁer for part conﬁguration. This formulation has shown very good results on
real-world data, however, it is based on linear classiﬁers
with less expressive representation than ours and is tested
on arms only. Finally, the idea of pose regression has been
employed by Ionescu et al. [11], however they reason about
3D pose.
The closest work to ours uses convolution NNs together
with Neighborhood Component Analysis to regress toward
a point in an embedding representing pose [24]. However,
this work does not employ a cascade of networks. Cascadesof DNN regressors have been used for localization, however
of facial points [22]. On the related problem of face pose
estimation, Osadchy et al. [17] employ a NN-based pose
embedding trained with a contrastive loss.
3. Deep Learning Model for Pose Estimation


key: chroma:document
string_value:
estimation, Osadchy et al. [17] employ a NN-based pose
embedding trained with a contrastive loss.
3. Deep Learning Model for Pose Estimation
We use the following notation. To express a pose, we encode the locations of all kbody joints in pose vector deﬁned
asy= (:::;yT
i;:::)T;i2f1;:::;kg, where yicontains
thexandycoordinates of the ithjoint. A labeled image is
denoted by (x;y)wherexstands for the image data and y
is the ground truth pose vector.
Further, since the joint coordinates are in absolute image
coordinates, it proves beneﬁcial to normalize them w. r. t. a
boxbbounding the human body or parts of it. In a trivial
case, the box can denote the full image. Such a box is deﬁned by its center bc2R2as well as width bwand height
bh:b= (bc;bw;bh). Then the joint yican be translated by
the box center and scaled by the box size which we refer to
as normalization by b:
N(yi;b) =1=bw 0
0 1=bh
(yibc) (1)


key: chroma:document
string_value:
as normalization by b:
N(yi;b) =1=bw 0
0 1=bh
(yibc) (1)
Further, we can apply the same normalization to the elements of pose vector N(y;b) = (:::;N (yi;b)T;:::)Tresulting in a normalized pose vector . Finally, with a slight
abuse of notation, we use N(x;b)to denote a crop of the
imagexby the bounding box b, which de facto normalizes
the image by the box. For brevity we denote by N()normalization with bbeing the full image box.
3.1. Pose Estimation as DNN-based Regression
In this work, we treat the problem of pose estimation as
regression, where the we train and use a function  (x;)2
R2kwhich for an image xregresses to a normalized pose
vector, where denotes the parameters of the model. Thus,
using the normalization transformation from Eq. (1) the
pose prediction yin absolute image coordinates reads
y=N1( (N(x);)) (2)
Despite its simple formulation, the power and complexity of the method is in  , which is based on a convolutional


key: chroma:document
string_value:
Despite its simple formulation, the power and complexity of the method is in  , which is based on a convolutional
Deep Neural Network (DNN). Such a convolutional network consists of several layers – each being a linear transformation followed by a non-linear one. The ﬁrst layer takes
as input an image of predeﬁned size and has a size equal to
the number of pixels times three color channels. The last
layer outputs the target values of the regression, in our case
2kjoint coordinates.
We base the architecture of the  on the work by
Krizhevsky et al. [14] for image classiﬁcation since it has
shown outstanding results on object localization as well
(xi, yi)
(x(s-1)i, y (s-1) i)xsi - x(s-1)iysi - y(s-1)iInitial stageStage s
send reﬁned values to next stage220 x 220
DNN-based regressor
27 x 27 x 128
13 x 13 x 192
13 x 13 x192
13 x 13 x192
4096 
4096 
55 x 55 x 48xiyi...DNN-based reﬁner
27 x 27 x 128
13 x 13 x 192
13 x 13 x192


key: chroma:document
string_value:
4096 
55 x 55 x 48xiyi...DNN-based reﬁner
27 x 27 x 128
13 x 13 x 192
13 x 13 x192
13 x 13 x192
4096 
4096 
55 x 55 x 48Figure 2. Left: schematic view of the DNN-based pose regression. We visualize the network layers with their corresponding dimensions,
where convolutional layers are in blue, while fully connected ones are in green. We do not show the parameter free layers. Right: at stage
s, a reﬁning regressor is applied on a sub image to reﬁne a prediction from the previous stage.
[23]. In a nutshell, the network consists of 7layers (see
Fig. 2 left). Denote by Ca convolutional layer, by LRN
a local response normalization layer, Pa pooling layer
and byFa fully connected layer. Only CandFlayers
contain learnable parameters, while the rest are parameter free. Both CandFlayers consist of a linear transformation followed by a nonlinear one, which in our case
is a rectiﬁed linear unit. For Clayers, the size is deﬁned as widthheightdepth, where the ﬁrst two dimensions have a spatial meaning while the depth deﬁnes


key: chroma:document
string_value:
the number of ﬁlters. If we write the size of each layer in
parentheses, then the network can be described concisely
asC(555596)LRNPC(2727256)
LRNPC(1313384)C(1313384)
C(1313256)PF(4096)F(4096) . The ﬁlter
size for the ﬁrst two Clayers is 1111and55and for
the remaining three is 33. Pooling is applied after three
layers and contributes to increased performance despite the
reduction of resolution. The input to the net is an image
of220220which via stride of 4is fed into the network.
The total number of parameters in the above model is about
40M. For further details, we refer the reader to [14].
The use of a generic DNN architecture is motivated by
its outstanding results on both classiﬁcation and localization
problems. In the experimental section we show that such a
generic architecture can be used to learn a model resulting
in state-of-art or better performance on pose estimation as
well. Further, such a model is a truly holistic one — the


key: chroma:document
string_value:
generic architecture can be used to learn a model resulting
in state-of-art or better performance on pose estimation as
well. Further, such a model is a truly holistic one — the
ﬁnal joint location estimate is based on a complex nonlinear
transformation of the full image.
Additionally, the use of a DNN obviates the need to design a domain speciﬁc pose model. Instead such a model
and the features are learned from the data. Although the regression loss does not model explicit interactions between
joints, such are implicitly captured by all of the 7hidden
layers – all the internal features are shared by all joint regressors.
Training The difference to [14] is the loss. Instead of a
classiﬁcation loss, we train a linear regression on top of thelast network layer to predict a pose vector by minimizing
L2distance between the prediction and the true pose vector. Since the ground truth pose vector is deﬁned in absolute image coordinates and poses vary in size from image to
image, we normalize our training set Dusing the normalization from Eq. (1):
DN=f(N(x);N(y))j(x;y)2Dg (3)
Then theL2loss for obtaining optimal network parameters
reads:
arg min
X


key: chroma:document
string_value:
Then theL2loss for obtaining optimal network parameters
reads:
arg min
X
(x;y)2DNkX
i=1jjyi i(x;)jj2
2 (4)
For clarity we write out the optimization over individual
joints. It should be noted, that the above objective can
be used even if for some images not all joints are labeled.
In this case, the corresponding terms in the sum would be
omitted.
The above parameters are optimized for using Backpropagation in a distributed online implementation. For
each mini-batch of size 128, adaptive gradient updates are
computed [3]. The learning rate, as the most important parameter, is set to 0:0005 . Since the model has large number
of parameters and the used datasets are of relatively small
size, we augment the data using large number of randomly
translated image crops (see Sec. 3.2), left/right ﬂips as well
as DropOut regularization for the Flayers set to 0:6.
3.2. Cascade of Pose Regressors
The pose formulation from the previous section has the
advantage that the joint estimation is based on the full image and thus relies on context. However, due to its ﬁxed
input size of 220220, the network has limited capacity


key: chroma:document
string_value:
input size of 220220, the network has limited capacity
to look at detail – it learns ﬁlters capturing pose properties
at coarse scale. These are necessary to estimate rough pose
but insufﬁcient to always precisely localize the body joints.
Note that we cannot easily increase the input size since this
will increase the already large number of parameters. In order to achieve better precision, we propose to train a cascade
of pose regressors. At the ﬁrst stage, the cascade starts off
by estimating an initial pose as outlined in the previous section. At subsequent stages, additional DNN regressors are
trained to predict a displacement of the joint locations from
previous stage to the true location. Thus, each subsequent
stage can be thought of as a reﬁnement of the currently predicted pose, as shown in Fig. 2.
Further, each subsequent stage uses the predicted joint
locations to focus on the relevant parts of the image – subimages are cropped around the predicted joint location from
previous stage and the pose displacement regressor for this
joint is applied on this sub-image. In this way, subsequent
pose regressors see higher resolution images and thus learn
features for ﬁner scales which ultimately leads to higher
precision.
We use the same network architecture for all stages of
the cascade but learn different network parameters. For


key: chroma:document
string_value:
features for ﬁner scales which ultimately leads to higher
precision.
We use the same network architecture for all stages of
the cascade but learn different network parameters. For
stages2 f1;:::;Sgof totalScascade stages, we denote bysthe learned network parameters. Thus, the
pose displacement regressor reads  (x;s). To reﬁne a
given joint location yiwe will consider a joint bounding
boxbicapturing the sub-image around yi:bi(y;) =
(yi;diam(y);diam(y))having as center the i-th joint
and as dimension the pose diameter scaled by . The diameter diam (y)of the pose is deﬁned as the distance between
opposing joints on the human torso, such as left shoulder
and right hip, and depends on the concrete pose deﬁnition
and dataset.
Using the above notation, at the stage s= 1we start with
a bounding box b0which either encloses the full image or
is obtained by a person detector. We obtain an initial pose:
Stage 1 : y1 N1( (N(x;b0);1);b0)(5)


key: chroma:document
string_value:
is obtained by a person detector. We obtain an initial pose:
Stage 1 : y1 N1( (N(x;b0);1);b0)(5)
At each subsequent stage s2, for all joints i2f1;:::;kg
we regress ﬁrst towards a reﬁnement displacement ys
i
y(s1)
i by applying a regressor on the sub image deﬁned
byb(s1)
i from previous stage (s1). Then, we estimate
new joint boxes bs
i:
Stages:ys
i y(s1)
i +N1( i(N(x;b);s);b)(6)
forb=b(s1)
i
bs
i (ys
i;diam(ys);diam(ys)) (7)
We apply the cascade for a ﬁxed number of stages S,
which is determined as explained in Sec. 4.1.
Training The network parameters 1are trained as
outlined in Sec. 3.1, Eq. (4). At subsequent stages


key: chroma:document
string_value:
Training The network parameters 1are trained as
outlined in Sec. 3.1, Eq. (4). At subsequent stages
s2, the training is done identically with one important difference. Each joint ifrom a training example(x;y)is normalized using a different bounding box(y(s1)
i;diam(y(s1));diam(y(s1)))– the one centered at the prediction for the same joint obtained from previous stage – so that we condition the training of the stage
based on the model from previous stage.
Since deep learning methods have large capacity, we
augment the training data by using multiple normalizations
for each image and joint. Instead of using the prediction
from previous stage only, we generate simulated predictions. This is done by randomly displacing the ground truth
location for joint iby a vector sampled at random from a
2-dimensional Normal distribution N(s1)
i with mean and
variance equal to the mean and variance of the observed displacements (y(s1)
iyi)across all examples in the training data. The full augmented training data can be deﬁned
by ﬁrst sampling an example and a joint from the original
data at uniform and then generating a simulated prediction


key: chroma:document
string_value:
by ﬁrst sampling an example and a joint from the original
data at uniform and then generating a simulated prediction
based on a sampled displacement fromN(s1)
i :
Ds
A=f(N(x;b);N(yi;b))j
(x;yi)D;N(s1)
i;
b= (yi+;diam(y))g
The training objective for cascade stage sis done as in
Eq. (4) by taking extra care to use the correct normalization
for each joint:
s= arg min
X
(x;yi)2Ds
Ajjyi i(x;)jj2
2 (8)
4. Empirical Evaluation
4.1. Setup
Datasets There is a wide variety of benchmarks for human pose estimation. In this work we use datasets, which
have large number of training examples sufﬁcient to train a
large model such as the proposed DNN, as well as are realistic and challenging.
The ﬁrst dataset we use is Frames Labeled In Cinema
(FLIC), introduced by [20], which consists of 4000 training and 1000 test images obtained from popular Hollywood
movies. The images contain people in diverse poses and especially diverse clothing. For each labeled human, 10upper


key: chroma:document
string_value:
movies. The images contain people in diverse poses and especially diverse clothing. For each labeled human, 10upper
body joints are labeled.
The second dataset we use is Leeds Sports Dataset [12]
and its extension [13], which we will jointly denote by LSP.
Combined they contain 11000 training and 1000 testing images. These are images from sports activities and as such
are quite challenging in terms of appearance and especially
articulations. In addition, the majority of people have 150
pixel height which makes the pose estimation even more
challenging. In this dataset, for each person the full body is
labeled with total 14joints.
For all of the above datasets, we deﬁne the diameter of a
poseyto be the distance between a shoulder and hip from
opposing sides and denote it by diam (y). It should be noted,
that the joints in all datasets are arranged in a tree kinematically mimicking the human body. This allows for a deﬁnition of a limb being a pair of neighboring joints in the pose
tree.
Metrics In order to be able to compare with published results we will use two widely accepted evaluation metrics.
Percentage of Correct Parts (PCP) measures detection rate
of limbs, where a limb is considered detected if the distance
between the two predicted joint locations and the true limb


key: chroma:document
string_value:
Percentage of Correct Parts (PCP) measures detection rate
of limbs, where a limb is considered detected if the distance
between the two predicted joint locations and the true limb
joint locations is at most half of the limb length [5]. PCP
was the initially preferred metric for evaluation, however it
has the drawback of penalizing shorter limbs, such as lower
arms, which are usually harder to detect.
To address this drawback, recently detection rates of
joints are being reported using a different detection criterion – a joint is considered detected if the distance between
the predicted and the true joint is within a certain fraction of
the torso diameter. By varying this fraction, detection rates
are obtained for varying degrees of localization precision.
This metric alleviates the drawback of PCP since the detection criteria for all joints are based on the same distance
threshold. We refer to this metric as Percent of Detected
Joints (PDJ).
Experimental Details For all the experiments we use the
same network architecture. Inspired by [7], we use a body
detector on FLIC to obtain initially a rough estimate of the
human body bounding box. It is based on a face detector –
the detected face rectangle is enlarged by a ﬁxed scaler. This
scaler is determined on the training data such that it contains
all labeled joints. This face-based body detector results in


key: chroma:document
string_value:
scaler is determined on the training data such that it contains
all labeled joints. This face-based body detector results in
a rough estimate, which however presents a good starting
point for our approach. For LSP we use the full image as
initial bounding box since the humans are relatively tightly
cropped by design.
Using a small held-out set of 50 images for both datasets
to determine the algorithm hyperparameters. To measure
optimality of the parameters we used average over PDJ at
0:2across all joints. The scaler , which deﬁnes the size of
the reﬁnement joint bounding box as a fraction of the pose
size, is determined as follows: for FLIC we chose = 1:0
after exploring values f0:8;1:0;1:2g, for LSP we use =
2:0after tryingf1:5;1:7;2:0;2:3g. The number of cascade
stagesSis determined by training stages until the algorithm
stopped improving on the held-out set. For both FLIC and
LSP we arrived at S= 3.
To improve generalization, for each cascade stage starting ats= 2 we augment the training data by sampling 40
randomly translated crop boxes for each joint as explained


key: chroma:document
string_value:
To improve generalization, for each cascade stage starting ats= 2 we augment the training data by sampling 40
randomly translated crop boxes for each joint as explained
in Sec. 3.2. Thus, for LSP with 14joints and after mirroring the images and sampling the number training examples
is1100040214 = 12M, which is essential for
training a large network as ours.The presented algorithm allows for an efﬁcient implementation. The running time is approx. 0:1sper image,
as measured on a 12core CPU. This compares favorably
to other approaches, as some of the current state-of-art approaches have higher complexity: [20] runs in approx. 4s,
while [27] runs in 1:5s. The training complexity, however,
is higher. The initial stage was trained within 3days on
approx. 100 workers, most of the ﬁnal performance was
achieved after 12hours though. Each reﬁnement stage was
trained for 7days since the amount of data was 40larger
than the one for the initial stage due to the data augmentation in Sec. 3.2. Note that using more data led to increased
performance.
4.2. Results and Discussion


key: chroma:document
string_value:
than the one for the initial stage due to the data augmentation in Sec. 3.2. Note that using more data led to increased
performance.
4.2. Results and Discussion
Comparisons We present comparative results to other approaches. We compare on LSP using PCP metric in Fig. 1.
We show results for the four most challenging limbs – lower
and upper arms and legs – as well as the average value
across these limbs for all compared algorithms. We clearly
outperform all other approaches, especially achieving better estimation for legs. For example, for upper legs we obtain0:78up from 0:74for the next best performing method.
It is worth noting that while the other approaches exhibit
strengths for particular limbs, none of the other dataset consistently dominates across all limbs. In contrary, DeepPose
shows strong results for all challenging limbs.
Using the PDJ metric allows us to vary the threshold for
the distance between prediction and ground truth, which deﬁnes a detection. This threshold can be thought of as a
localization precision at which detection rates are plotted.
Thus one could compare approaches across different desired precisions. We present results on FLIC in Fig. 3 comparing against additional four methods as well is on LSP in
Fig. 4. For each dataset we train and test according the protocol for each dataset. Similarly to previous experiment we


key: chroma:document
string_value:
Fig. 4. For each dataset we train and test according the protocol for each dataset. Similarly to previous experiment we
outperform all ﬁve algorithms. Our gains are bigger in the
low precision domain, in the cases where we detect rough
pose without precisely localizing the joints. On FLIC, at
normalized distance 0:2we obtain a an increase of detection
rates by 0:15and0:2for elbow and wrists against the next
best performing method. On LSP, at normalized distance
0:5we get an absolute increase of 0:1. At low precision
regime of normalized distance of 0:2for LSP we show comparable performance for legs and slightly worse arms. This
can be attributed to the fact that the DNN-based approach
computes joint coordinates using 7 layers of transformation,
some of which contain max pooling.
Another observation is that our approach works well for
both appearance heavy movie data as well as string articulation such as the sports images in LSP.
Effects of cascade-based reﬁnement A single DNNbased joint regressor gives rough joint location. However,
0 0.05 0.1 0.15 0.20.10.30.50.70.9Elbows
Normalized distance to true jointDetection rate


key: chroma:document
string_value:
0 0.05 0.1 0.15 0.20.10.30.50.70.9Elbows
Normalized distance to true jointDetection rate
  
DeepPoseMODECEichner et al.Yang et al.Sapp et al.
0 0.05 0.1 0.15 0.20.10.30.50.70.9Wrists
Normalized distance to true jointDetection rateFigure 3. Percentage of detected joints (PDJ) on FLIC for two joints: elbow and wrist. We compare DeepPose, after two cascade stages,
with four other approaches.
MethodArm LegAve.Upper Lower Upper Lower
DeepPose-st1 0.5 0.27 0.74 0.65 0.54
DeepPose-st2 0.56 0.36 0.78 0.70 0.60
DeepPose-st3 0.56 0.38 0.77 0.71 0.61
Dantone et al. [2] 0.45 0.25 0.65 0.61 0.49
Tian et al. [25]0.52 0.33 0.70 0.60 0.56
Johnson et al. [13] 0.54 0.38 0.75 0.66 0.58


key: chroma:document
string_value:
Johnson et al. [13] 0.54 0.38 0.75 0.66 0.58
Wang et al. [26]0.565 0.37 0.76 0.68 0.59
Pishchulin [18]0.49 0.32 0.74 0.70 0.56
Table 1. Percentage of Correct Parts (PCP) at 0.5 on LSP for DeepPose as well as ﬁve state-of-art approaches.The authors use a
slightly looser version of PCP in which for each limb the average
distance from predicted limb joints belonging to the corresponding true joints is used to determine whether the limb is correctly
detected.The authors use person-centric joint annotations.
0 0.1 0.2 0.3 0.4 0.50.10.30.50.70.9Arms
Normalized distance to true jointDetection rate
  
DeepPose − wristsDeepPose − elbowsDantone et al. − wristsDantone et al. − elbows
0 0.1 0.2 0.3 0.4 0.50.10.30.50.70.9Legs
Normalized distance to true jointDetection rate
  
DeepPose − ankleDeepPose − kneeDantone et al. − ankleDantone et al. − knee


key: chroma:document
string_value:
Normalized distance to true jointDetection rate
  
DeepPose − ankleDeepPose − kneeDantone et al. − ankleDantone et al. − knee
Figure 4. Percentage of detected joints (PDJ) on LSP for four
limbs for DeepPose and Dantone et al. [2] over an extended range
of distances to true joint: [0;0:5]of the torso diameter. Results of
DeepPose are plotted with solid lines while all the results by [2]
are plotted in dashed lines. Results for the same joint from both
algorithms are colored with same color.
to obtain higher precision the subsequent stages of the cascade, which serve as a reﬁnement of the initial prediction,
are of paramount importance. To see this, in Fig. 5 we
present the joint detections at different precisions for the initial prediction as well as two subsequent cascade stages. As
expected, we can see that the major gains of the reﬁnement
00.05 0.10.15 0.20.25 0.30.35 0.40.10.30.50.70.9Wrists
Normalized distance to true jointDetection rate
  
DeepPose − initial stage 1DeepPose − stage 2DeepPose − stage 3


key: chroma:document
string_value:
Normalized distance to true jointDetection rate
  
DeepPose − initial stage 1DeepPose − stage 2DeepPose − stage 3
0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.40.10.30.50.70.9Elbows
Normalized distance to true jointDetection rateFigure 5. Percent of detected joints (PDJ) on FLIC or the ﬁrst three
stages of the DNN cascade. We present results over larger spectrum of normalized distances between prediction and ground truth.
procedure are at high-precision regime of at normalized distances of [0:15;0:2]. Further, the major gains are achieved
after one stage of reﬁnement. The reason being that subsequent stages end up using smaller sub-images around each
joint. And although the subsequent stages look at higher
resolution inputs, they have more limited context.
Examples of cases, where reﬁnement helps, are visualized in Fig. 6. The initial stage is usually successful at estimating a roughly correct pose, however, this pose is not
”snapped” to the correct one. For example, in row three the
pose has the right shape but incorrect scale. In the second
row, the predicted pose is translated north from the ideal


key: chroma:document
string_value:
pose has the right shape but incorrect scale. In the second
row, the predicted pose is translated north from the ideal
one. In most cases, the second stage of the cascade resolves
this snapping problem and better aligns the joints. In more
rare cases, such as in ﬁrst row, further facade stages improve
on individual joints.
Cross-dataset Generalization To evaluate the generalization properties of our algorithm, we used the trained
models on LSP and FLIC on two related datasets. The fullbody model trained on LSP is tested on the test portion of
the Image Parse dataset [19] with results presented in Table 2. The ImageParse dataset is similar to LSP as it contains people doing sports, however it contains a lot of people from personal photo collections involved in other activities. Further, the upper-body model trained on FLIC was
applied on the whole Buffy dataset [7]. We can see that our
Initial stage 1stage 2stage 3Figure 6. Predicted poses in red and ground truth poses in green
for the ﬁrst three stages of a cascade for three examples.
0 0.05 0.1 0.15 0.20.10.30.50.70.9Elbows
Normalized distance to true jointDetection rate
  
Eichner et al.Yang et al.Sapp et al.MODECDeepPose


key: chroma:document
string_value:
Normalized distance to true jointDetection rate
  
Eichner et al.Yang et al.Sapp et al.MODECDeepPose
0 0.05 0.1 0.15 0.20.10.30.50.70.9Wrists
Normalized distance to true jointDetection rate
Figure 7. Percentage of detected joints (PDJ) on Buffy dataset
for two joints: elbow and wrist. The models have been trained on
FLIC. We compare DeepPose, after two cascade stages, with four
other approaches.
MethodArm LegAve.Upper Lower Upper Lower
DeepPose 0.8 0.75 0.71 0.5 0.69
Pishchulin [18] 0.80 0.70 0.59 037 0.62
Johnson et al. [13] 0.75 0.67 0.67 0.46 0.64
Yang et al. [27] 0.69 0.64 0.55 0.35 0.56
Table 2. Percentage of Correct Parts (PCP) at 0.5 on Image Parse
dataset for DeepPose as well as two state-of-art approaches on Image Parse dataset. Results obtained from [18].
approach can retain state-of-art performance compared to
other approaches. This shows good generalization abilities.


key: chroma:document
string_value:
approach can retain state-of-art performance compared to
other approaches. This shows good generalization abilities.
Example poses To get a better idea of the performance of
our algorithm, we visualize a sample of estimated poses on
images from LSP in Fig. 8. We can see that our algorithm is
able to get correct pose for most of the joints under variety
of conditions: upside-down people (row 1, column 1), severe foreshortening (row1, column 3), unusual poses (row
3, column 5), occluded limbs as the occluded arms in row
3, columns 2 and 6, unusual illumination conditions (row 3,
column 3). In most of the cases, when the estimated pose isnot precise, it still has a correct shape. For example, in the
last row some of the predicted limbs are not aligned with
the true locations, however the overall shape of the pose is
correct. A common failure mode is confusing left with right
side when the person was photographed from the back (row
6, column 6). Results on FLIC (see Fig. 9) are usually better
with occasional visible mistakes on lower arms.
5. Conclusion
We present, to our knowledge, the ﬁrst application of
Deep Neural Networks (DNNs) to human pose estimation.
Our formulation of the problem as DNN-based regression to


key: chroma:document
string_value:
Deep Neural Networks (DNNs) to human pose estimation.
Our formulation of the problem as DNN-based regression to
joint coordinates and the presented cascade of such regressors has the advantage of capturing context and reasoning
about pose in a holistic manner. As a result, we are able to
achieve state-of-art or better results on several challenging
academic datasets.
Further, we show that using a generic convolutional neural network, which was originally designed for classiﬁcation tasks, can be applied to the different task of localization. In future, we plan to investigate novel architectures
which could be potentially better tailored towards localization problems in general, and in pose estimation in particular.
Acknowledgements I would like to thank Luca Bertelli,
Ben Sapp and Tianli Yu for assistance with data and fruitful
discussions.
References
[1] M. Andriluka, S. Roth, and B. Schiele. Pictorial structures
revisited: People detection and articulated pose estimation.
InCVPR , 2009.
[2] M. Dantone, J. Gall, C. Leistner, and L. Van Gool. Human
pose estimation using body parts dependent joint regressors.
InCVPR , 2013.
[3] J. Duchi, E. Hazan, and Y . Singer. Adaptive subgradient


key: chroma:document
string_value:
pose estimation using body parts dependent joint regressors.
InCVPR , 2013.
[3] J. Duchi, E. Hazan, and Y . Singer. Adaptive subgradient
methods for online learning and stochastic optimization. In
COLT . ACL, 2010.
[4] M. Eichner and V . Ferrari. Better appearance models for
pictorial structures. 2009.
[5] M. Eichner, M. Marin-Jimenez, A. Zisserman, and V . Ferrari.
Articulated human pose estimation and search in (almost)
unconstrained still images. ETH Zurich, D-ITET, BIWI,
Technical Report No , 272, 2010.
[6] P. F. Felzenszwalb and D. P. Huttenlocher. Pictorial structures for object recognition. International Journal of Computer Vision , 61(1):55–79, 2005.
[7] V . Ferrari, M. Marin-Jimenez, and A. Zisserman. Progressive
search space reduction for human pose estimation. In CVPR ,
2008.
[8] M. A. Fischler and R. A. Elschlager. The representation and
matching of pictorial structures. Computers, IEEE Transactions on , 100(1):67–92, 1973.


key: chroma:document
string_value:
matching of pictorial structures. Computers, IEEE Transactions on , 100(1):67–92, 1973.
Figure 8. Visualization of pose results on images from LSP. Each pose is represented as a stick ﬁgure, inferred from predicted joints.
Different limbs in the same image are colored differently, same limb across different images has the same color.
Figure 9. Visualization of pose results on images from FLIC. Meaning of stick ﬁgures is the same as in Fig. 8 above.
[9] R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature hierarchies for accurate object detection and semantic
segmentation. In CVPR , 2014.
[10] G. Gkioxari, P. Arbel ´aez, L. Bourdev, and J. Malik. Articulated pose estimation using discriminative armlet classiﬁers.
InCVPR , 2013.
[11] C. Ionescu, F. Li, and C. Sminchisescu. Latent structured
models for human pose estimation. In ICCV , 2011.
[12] S. Johnson and M. Everingham. Clustered pose and nonlinear appearance models for human pose estimation. In BMVC ,
2010.
[13] S. Johnson and M. Everingham. Learning effective human


key: chroma:document
string_value:
2010.
[13] S. Johnson and M. Everingham. Learning effective human
pose estimation from inaccurate annotation. In CVPR , 2011.
[14] A. Krizhevsky, I. Sutskever, and G. Hinton. Imagenet classiﬁcation with deep convolutional neural networks. In NIPS ,
2012.
[15] G. Mori and J. Malik. Estimating human body conﬁgurations
using shape context matching. In ECCV , 2002.[16] R. Nevatia and T. O. Binford. Description and recognition of
curved objects. Artiﬁcial Intelligence , 8(1):77–98, 1977.
[17] M. Osadchy, Y . LeCun, and M. L. Miller. Synergistic face
detection and pose estimation with energy-based models.
The Journal of Machine Learning Research , 8:1197–1215,
2007.
[18] L. Pishchulin, M. Andriluka, P. Gehler, and B. Schiele. Poselet conditioned pictorial structures. In CVPR , 2013.
[19] D. Ramanan. Learning to parse images of articulated bodies.
InNIPS , 2006.


key: chroma:document
string_value:
[19] D. Ramanan. Learning to parse images of articulated bodies.
InNIPS , 2006.
[20] B. Sapp and B. Taskar. Modec: Multimodal decomposable
models for human pose estimation. In CVPR , 2013.
[21] G. Shakhnarovich, P. Viola, and T. Darrell. Fast pose estimation with parameter-sensitive hashing. In CVPR , 2003.
[22] Y . Sun, X. Wang, and X. Tang. Deep convolutional network cascade for facial point detection. In Computer Vision
and Pattern Recognition (CVPR), 2013 IEEE Conference on ,
pages 3476–3483. IEEE, 2013.
[23] C. Szegedy, A. Toshev, and D. Erhan. Object detection via
deep neural networks. In NIPS 26 , 2013.
[24] G. W. Taylor, R. Fergus, G. Williams, I. Spiro, and C. Bregler. Pose-sensitive embedding by nonlinear nca regression.
InNIPS , 2010.
[25] Y . Tian, C. L. Zitnick, and S. G. Narasimhan. Exploring the
spatial hierarchy of mixture models for human pose estimation. In ECCV , 2012.


key: chroma:document
string_value:
spatial hierarchy of mixture models for human pose estimation. In ECCV , 2012.
[26] F. Wang and Y . Li. Beyond physical connections: Tree models in human pose estimation. In CVPR , 2013.
[27] Y . Yang and D. Ramanan. Articulated pose estimation with
ﬂexible mixtures-of-parts. In CVPR , 2011.


key: chroma:document
string_value:
1
Return of the Devil in the Details:
Delving Deep into Convolutional Nets
Ken Chatﬁeld, Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman
Visual Geometry Group, Department of Engineering Science, University of Oxford
fken,karen,vedaldi,az g@robots.ox.ac.uk
Abstract —The latest generation of Convolutional Neural Networks (CNN) have achieved impressive results in challenging benchmarks on image recognition and object detection, signiﬁcantly raising the interest of the community in
these methods. Nevertheless, it is still unclear how different CNN methods compare with each other and with previous
state-of-the-art shallow representations such as the Bag-of-Visual-Words and the Improved Fisher Vector. This paper
conducts a rigorous evaluation of these new techniques, exploring different deep architectures and comparing them on
a common ground, identifying and disclosing important implementation details. We identify several useful properties
of CNN-based representations, including the fact that the dimensionality of the CNN output layer can be reduced
signiﬁcantly without having an adverse effect on performance. We also identify aspects of deep and shallow methods
that can be successfully shared. In particular, we show that the data augmentation techniques commonly applied to
CNN-based methods can also be applied to shallow methods, and result in an analogous performance boost. Source


key: chroma:document
string_value:
CNN-based methods can also be applied to shallow methods, and result in an analogous performance boost. Source
code and models to reproduce the experiments in the paper is made publicly available.
F
1 I NTRODUCTION
PERHAPS the single most important design
choice in current state-of-the-art image classiﬁcation and object recognition systems is the choice
of visual features, or image representation. In fact,
most of the quantitative improvements to image
understanding obtained in the past dozen years can
be ascribed to the introduction of improved representations, from the Bag-of-Visual-Words (BoVW) [1],
[2] to the (Improved) Fisher Vector (IFV) [3]. A
common characteristic of these methods is that
they are largely handcrafted . They are also relatively
simple, comprising dense sampling of local image
patches, describing them by means of visual descriptors such as SIFT, encoding them into a highdimensional representation, and then pooling over
the image. Recently, these handcrafted approaches
have been substantially outperformed by the introduction of the latest generation of Convolutional
Neural Networks (CNNs) [4] to the computer vision
ﬁeld. These networks have a substantially more
sophisticated structure than standard representations, comprising several layers of non-linear feature extractors, and are therefore said to be deep


key: chroma:document
string_value:
ﬁeld. These networks have a substantially more
sophisticated structure than standard representations, comprising several layers of non-linear feature extractors, and are therefore said to be deep
(in contrast, classical representation will be referred
to as shallow ). Furthermore, while their structureis handcrafted, they contain a very large number
of parameters learnt from data. When applied to
standard image classiﬁcation and object detection
benchmark datasets such as ImageNet ILSVRC [5]
and PASCAL VOC [6] such networks have demonstrated excellent performance [7], [8], [9], [10], [11],
signiﬁcantly better than standard image encodings [12].
Despite these impressive results, it remains unclear how different deep architectures compare to
each other and to shallow computer vision methods such as IFV . Most papers did not test these
representations extensively on a common ground,
so a systematic evaluation of the effect of different design and implementation choices remains
largely missing. As noted in our previous work [12],
which compared the performance of various shallow visual encodings, the performance of computer
vision systems depends signiﬁcantly on implementation
details . For example, state-of-the-art methods such
as [13] not only involve the use of a CNN, but
also include other improvements such as the use


key: chroma:document
string_value:
details . For example, state-of-the-art methods such
as [13] not only involve the use of a CNN, but
also include other improvements such as the use
of very large scale datasets, GPU computation, and
data augmentation (also known as data jittering or
virtual sampling). These improvements could also
transfer to shallow representations such as the IFV ,
potentially explaining a part of the performancearXiv:1405.3531v4  [cs.CV]  5 Nov 2014
2
gap [14].
In this study we analyse and empirically clarify
these issues, conducting a large set of rigorous
experiments (Sect. 4), in many ways picking up
the story where it last ended in [12] with the
comparison of shallow encoders. We focus on methods to construct image representations ,i.e. encoding functions mapping an image Ito a vector
(I)2Rdsuitable for analysis with a linear classiﬁer, such as an SVM. We consider three scenarios
(Sect. 2, Sect. 3): shallow image representations,
deep representations pre-trained on outside data,
and deep representation pre-trained and then ﬁnetuned on the target dataset. As part of our tests,
we explore generally-applicable best practices that


key: chroma:document
string_value:
and deep representation pre-trained and then ﬁnetuned on the target dataset. As part of our tests,
we explore generally-applicable best practices that
are nevertheless more often found in combination with CNNs [13] or, alternatively, with shallow
encoders [12], porting them with mutual beneﬁt.
These are (Sect. 2): the use of colour information ,
feature normalisation , and, most importantly, the
use of substantial data augmentation . We also determine scenario-speciﬁc best-practices , improving
the ones in [12], [15] and others, including dimensionality reduction for deep features. Finally, we
achieve performance competitive with the state
of the art [16], [17] on PASCAL VOC classiﬁcation
using less additional training data and signiﬁcantly
simpler techniques. As in [12], the source code and
models to reproduce all experiments in this paper
is available on the project website1.
2 S CENARIOS
This section introduces the three types of image
representation (I)considered in this paper, describing them within the context of three different
scenarios. Having outlined details speciﬁc to each,
general methodologies which apply to all three
scenarios are reviewed, such as data augmentation


key: chroma:document
string_value:
scenarios. Having outlined details speciﬁc to each,
general methodologies which apply to all three
scenarios are reviewed, such as data augmentation
and feature normalisation, together with the linear
classiﬁer (trained with a standard hinge loss). We
also specify here the benchmark datasets used in
the evaluation.
2.1 Scenario 1: Shallow representation (IFV)
Our reference shallow image representation is the
IFV [3]. Our choice is motivated by the fact that
IFV usually outperforms related encoding methods
such as BoVW, LLC [12], and VLAD [18]. Given an
imageI, the IFVFV(I)is obtained by extracting
1. http://www.robots.ox.ac.uk/vgg/research/deep eval/a dense collection of patches and corresponding
local descriptors xi2RD(e.g. SIFT) from the image
at multiple scales. Each descriptor xiis then softquantized using a Gaussian Mixture Model with
Kcomponents. First and second order differences
between each descriptor xiand its Gaussian cluster
meankare accumulated in corresponding blocks
uk,vkin the vector FV(I)2R2KD, appropriately
weighed by the Gaussian soft-assignments and


key: chroma:document
string_value:
uk,vkin the vector FV(I)2R2KD, appropriately
weighed by the Gaussian soft-assignments and
covariance, leading to a 2KD-dimensional image
representation FV(I) = [u>
1;v>
1;:::u>
K;v>
K]>:The
improved version of the Fisher vector involves postprocessing FVby computing the signed squareroot of its scalar components and normalising the
result to a unit `2norm. The details of this construction can be found in [3]; here we follow the
notation of [12].
2.2 Scenario 2: Deep representation (CNN) with
pre-training
Our deep representations are inspired by the success of the CNN of Krizhevsky et al. [13]. As shown
in [7], [19], the vector of activities CNN(I)of the
penultimate layer of a deep CNN, learnt on a large
dataset such as ImageNet [5], can be used as a powerful image descriptor applicable to other datasets.
Numerous CNN architectures that improve the
previous state of the art obtained using shallow
representations have been proposed, but choosing


key: chroma:document
string_value:
Numerous CNN architectures that improve the
previous state of the art obtained using shallow
representations have been proposed, but choosing
the best one remains an open question. Many are inspired by [13]: DeCAF [7], [11], Caffe [20], Oquab et
al. [8]. Others use larger networks with a smaller
stride of the ﬁrst convolutional layer: Zeiler and
Fergus [19] and OverFeat [10], [9]. Other differences
include the CNN pre-training protocols. Here we
adopt a single learning framework and experiment
with architectures of different complexity exploring
their performance-speed trade-off.
2.3 Scenario 3: Deep representation (CNN) with
pre-training and ﬁne-tuning
In Scenario 2 features are trained on one (large)
dataset and applied to another (usually smaller).
However, it was demonstrated [11] that ﬁne-tuning
a pre-trained CNN on the target data can signiﬁcantly improve the performance. We consider
this scenario separately from that of Scenario 2, as
the image features become dataset-speciﬁc after the
ﬁne-tuning.
3
2.4 Commonalities
We now turn to what is in common across the
scenarios.


key: chroma:document
string_value:
ﬁne-tuning.
3
2.4 Commonalities
We now turn to what is in common across the
scenarios.
2.4.1 Data augmentation
Data augmentation is a method applicable to shallow and deep representations, but that has been
so far mostly applied to the latter [13], [19]. By
augmentation we mean perturbing an image Iby
transformations that leave the underlying class unchanged ( e.g. cropping and ﬂipping) in order to
generate additional examples of the class. Augmentation can be applied at training time, at test time,
or both. The augmented samples can either be taken
as-is or combined to form a single feature, e.g. using
sum/max-pooling or stacking.
2.4.2 Linear predictors
All the representations (I)in the three scenarios
are used to construct linear predictorshw;(I)ifor
each class to be recognized. These predictors are
learnt using Support Vector Machines (SVM) by
ﬁtting wto the available training data by minimizing an objective function balancing a quadratic
regularizer and the hinge-loss. The parameter C
in the SVM, trading-off regularizer and loss, is
determined using an held-off validation subset of


key: chroma:document
string_value:
regularizer and the hinge-loss. The parameter C
in the SVM, trading-off regularizer and loss, is
determined using an held-off validation subset of
the data. Here we use the same learning framework
with all representations. It is common experience
that linear classiﬁers are particularly sensitive to
thenormalisation of the data and that, in particular,
SVMs tend to beneﬁt from `2normalisation [3] (an
interpretation is that after normalisation the inner
product corresponds to the cosine similarly).
2.5 Benchmark data
As reference benchmark we use the PASCAL
VOC [6] data as already done in [12]. The VOC2007 edition contains about 10,000 images split into
train, validation, and test sets, and labelled with
twenty object classes. A one-vs-rest SVM classiﬁer
for each class is learnt and evaluated independently
and the performance is measured as mean Average
Precision (mAP) across all classes. The VOC-2012
edition contains roughly twice as many images
and does not include test labels; instead, evaluation
uses the ofﬁcial PASCAL Evaluation Server. To train
deep representations we use the ILSVRC-2012 challenge dataset. This contains 1,000 object categories


key: chroma:document
string_value:
uses the ofﬁcial PASCAL Evaluation Server. To train
deep representations we use the ILSVRC-2012 challenge dataset. This contains 1,000 object categories
from ImageNet [5] with roughly 1.2M training images, 50,000 validation images, and 100,000 testimages. Performance is evaluated using the top-5
classiﬁcation error. Finally, we also evaluate over
theCaltech-101 and Caltech-256 image classiﬁcation benchmarks [21], [22]. For Caltech-101, we
followed the protocol of [12], and considered three
random splits into training and testing data, each
of which comprises 30 training and up to 30 testing
images per class. For Caltech-256, two random
splits were generated, each of which contains 60
training images per class, and the rest are used for
testing. On both Caltech datasets, performance is
measured using mean class accuracy.
3 D ETAILS
This section gives the implementation details of the
methods introduced in Sect. 2.
3.1 Improved Fisher Vector details
Our IFV representation uses a slightly improved
setting compared to the best result of [12].
Computation starts by upscaling the image Iby a
factor of 2 [23], followed by SIFT features extraction


key: chroma:document
string_value:
setting compared to the best result of [12].
Computation starts by upscaling the image Iby a
factor of 2 [23], followed by SIFT features extraction
with a stride of 3 pixels at 7 different scales withp
2
scale increments. These features are square-rooted
as suggested by [24], and decorrelated and reduced
in dimension from 128Dto80Dusing PCA. A
GMM with K= 256 components is learnt from features sampled from the training images. Hence the
Fisher Vector FV(I)has dimension 2KD= 40;960.
Before use in classiﬁcation, the vector is signedsquare-rooted and l2-normalised (square rooting
correspond to the Hellinger’s kernel map [25]). As
in [12], square-rooting is applied twice, once to the
raw encodings, and once again after sum pooling
and normalisation. In order to capture weak geometrical information, the IFV representation is used
in a spatial pyramid [26]. As in [12], the image is
divided into 11,31, and 22spatial subdivisions
and corresponding IFVs are computed and stacked
with an overall dimension of 82KD= 327;680
elements.


key: chroma:document
string_value:
and corresponding IFVs are computed and stacked
with an overall dimension of 82KD= 327;680
elements.
In addition to this standard formulation, we experiment with a few modiﬁcations. The ﬁrst one
is the use of intra-normalisation of the descriptor
blocks, an idea recently proposed for the VLAD
descriptor [27]. In this case, the `2normalisation
is applied to the individual sub-blocks (uk;vk)of
the vector FV(I), which helps to alleviate the
local feature burstiness [28]. In the case of the
improved intra-normalised features, it was found
4
that applying the square-rooting only once to the
ﬁnal encoding produced the best results.
The second modiﬁcation is the use of spatiallyextended local descriptors [23] instead of a spatial
pyramid. Here descriptors xiare appended with
their image location (xi;yi)before quantization
with the GMM. Formally, xiis extended, after PCA
projection, with its normalised spatial coordinates:
[x>
i;xi=W0:5;yi=H0:5]>, whereWHare the
dimensions of the image. Since the GMM quantizes


key: chroma:document
string_value:
i;xi=W0:5;yi=H0:5]>, whereWHare the
dimensions of the image. Since the GMM quantizes
both appearance and location, this allows for spatial
information to be captured directly by the softquantization process. This method is signiﬁcantly
more memory-efﬁcient than using a spatial pyramid. Speciﬁcally, the PCA-reduced SIFT features
are spatially augmented by appending (x;y)yieldingD= 82 dimensional descriptors pooled in a
2KD= 41;984dimensional IFV .
The third modiﬁcation is the use of colour features in addition to SIFT descriptors. While colour
information is used in CNNs [13] and by the original FV paper [3], it was not explored in our previous comparison [12]. We do so here by adopting
the same Local Colour Statistics (LCS) features as
used by [3]. LCS is computed by dividing an input
patch into a 44spatial grid (akin to SIFT), and
computing the mean and variance of each of the
Lab colour channels for each cell of the grid. The
LCS dimensionality is thus 4423 = 96 . This
is then encoded in a similar manner to SIFT.


key: chroma:document
string_value:
Lab colour channels for each cell of the grid. The
LCS dimensionality is thus 4423 = 96 . This
is then encoded in a similar manner to SIFT.
3.2 Convolutional neural networks details
The CNN-based features are based on three CNN
architectures representative of the state of the art
(shown in Table 1) each exploring a different accuracy/speed trade-off. To ensure a fair comparison
between them, these networks are trained using the
same training protocol and the same implementation, which we developed based on the opensource Caffe framework [20]. `2-normalising the
CNN features CNN(I)before use in the SVM was
found to be important for performance.
Our Fast (CNN-F) architecture is similar to the
one used by Krizhevsky et al . [13]. It comprises
8 learnable layers, 5 of which are convolutional,
and the last 3 are fully-connected. The input image
size is 224224. Fast processing is ensured by the
4 pixel stride in the ﬁrst convolutional layer. The
main differences between our architecture and that
of [13] are the reduced number of convolutional
layers and the dense connectivity between convolu-tional layers ([13] used sparse connections to enable
training on two GPUs).


key: chroma:document
string_value:
of [13] are the reduced number of convolutional
layers and the dense connectivity between convolu-tional layers ([13] used sparse connections to enable
training on two GPUs).
Our Medium (CNN-M) architecture is similar
to the one used by Zeiler and Fergus [19]. It is
characterised by the decreased stride and smaller
receptive ﬁeld of the ﬁrst convolutional layer, which
was shown to be beneﬁcial on the ILSVRC dataset.
At the same time, conv2 uses larger stride (2 instead
of 1) to keep the computation time reasonable. The
main difference between our net and that of [19] is
we use less ﬁlters in the conv4 layer (512 vs. 1024).
Our Slow (CNN-S) architecture is related to the
‘accurate’ network from the OverFeat package [10].
It also uses 77ﬁlters with stride 2in conv1.
Unlike CNN-M and [19], the stride in conv2 is
smaller (1 pixel), but the max-pooling window in
conv1 and conv5 is larger ( 33) to compensate for
the increased spatial resolution. Compared to [10],
we use 5 convolutional layers as in the previous


key: chroma:document
string_value:
conv1 and conv5 is larger ( 33) to compensate for
the increased spatial resolution. Compared to [10],
we use 5 convolutional layers as in the previous
architectures ([10] used 6), and less ﬁlters in conv5
(512 instead of 1024); we also incorporate an LRN
layer after conv1 ([10] did not use contrast normalisation).
3.2.1 CNN training
In general, our CNN training procedure follows that of [13], learning on ILSVRC-2012 using gradient descent with momentum. The hyperparameters are the same as used by [13]: momentum0:9; weight decay 5104; initial learning rate
102, which is decreased by a factor of 10, when
the validation error stop decreasing. The layers are
initialised from a Gaussian distribution with a zero
mean and variance equal to 102. We also employ
similar data augmentation in the form of random
crops, horizontal ﬂips, and RGB colour jittering.
Test time crop sampling is discussed in Sect. 3.3;
at training time, 224224 crops are sampled randomly, rather than deterministically. Thus, the only
notable difference to [13] is that the crops are taken
from the whole training image P256;P256,


key: chroma:document
string_value:
notable difference to [13] is that the crops are taken
from the whole training image P256;P256,
rather than its 256256 centre. Training was performed on a single NVIDIA GTX Titan GPU and
the training time varied from 5 days for CNN-F to
3 weeks for CNN-S.
3.2.2 CNN ﬁne-tuning on the target dataset
In our experiments, we ﬁne-tuned CNN-S using
VOC-2007, VOC-2012, or Caltech-101 as the target
data. Fine-tuning was carried out using the same
framework (and the same data augmentation), as
we used for CNN training on ILSVRC. The last
5
Arch. conv1 conv2 conv3 conv4 conv5 full6 full7 full8
CNN-F64x11x11 256x5x5 256x3x3 256x3x3 256x3x3 4096 4096 1000
st. 4, pad 0 st. 1, pad 2 st. 1, pad 1 st. 1, pad 1 st. 1, pad 1 drop- drop- softLRN, x2 pool LRN, x2 pool - - x2 pool out out max


key: chroma:document
string_value:
CNN-M96x7x7 256x5x5 512x3x3 512x3x3 512x3x3 4096 4096 1000
st. 2, pad 0 st. 2, pad 1 st. 1, pad 1 st. 1, pad 1 st. 1, pad 1 drop- drop- softLRN, x2 pool LRN, x2 pool - - x2 pool out out max
CNN-S96x7x7 256x5x5 512x3x3 512x3x3 512x3x3 4096 4096 1000
st. 2, pad 0 st. 1, pad 1 st. 1, pad 1 st. 1, pad 1 st. 1, pad 1 drop- drop- softLRN, x3 pool x2 pool - - x3 pool out out max
TABLE 1
CNN architectures. Each architecture contains 5 convolutional layers (conv 1–5) and three
fully-connected layers (full 1–3). The details of each of the convolutional layers are given in three sub-rows:
the ﬁrst speciﬁes the number of convolution ﬁlters and their receptive ﬁeld size as “num x size x size”; the
second indicates the convolution stride (“st.”) and spatial padding (“pad”); the third indicates if Local


key: chroma:document
string_value:
second indicates the convolution stride (“st.”) and spatial padding (“pad”); the third indicates if Local
Response Normalisation (LRN) [13] is applied, and the max-pooling downsampling factor. For full 1–3, we
specify their dimensionality, which is the same for all three architectures. Full6 and full7 are regularised
using dropout [13], while the last layer acts as a multi-way soft-max classiﬁer. The activation function for all
weight layers (except for full8) is the REctiﬁcation Linear Unit (RELU) [13].
fully-connected layer (conv8) has output dimensionality equal to the number of classes, which
differs between datasets, so we initialised it from
a Gaussian distribution (as used for CNN training
above). Now we turn to dataset-speciﬁc ﬁne-tuning
details.
VOC-2007 and VOC-2012. Considering that PASCAL VOC is a multi-label dataset ( i.e. a single
image might have multiple labels), we replaced the
softmax regression loss with a more appropriate
loss function, for which we considered two options:
one-vs-rest classiﬁcation hinge loss (the same loss


key: chroma:document
string_value:
softmax regression loss with a more appropriate
loss function, for which we considered two options:
one-vs-rest classiﬁcation hinge loss (the same loss
as used in the SVM experiments) and ranking hinge
loss. Both losses deﬁne constraints on the scores of
positive (Ipos) and negative ( Ineg) images for each
class:wc(Ipos)>1;wc(Ineg)<1 +for the
classiﬁcation loss, wc(Ipos)> w c(Ineg) + 1
for the ranking loss ( wcis thec-th row of the
last fully-connected layer, which can be seen as a
linear classiﬁer on deep features (I);is a slack
variable). Our ﬁne-tuned networks are denoted as
“CNN S TUNE-CLS” (for the classiﬁcation loss)
and “CNN S TUNE-RNK” (for the ranking loss).
In the case of both VOC datasets, the training
and validation subsets were combined to form a
single training set. Given the smaller size of the
training data when compared to ILSVRC-2012, we


key: chroma:document
string_value:
and validation subsets were combined to form a
single training set. Given the smaller size of the
training data when compared to ILSVRC-2012, we
controlled for over-ﬁtting by using lower initial
learning rates for the ﬁne-tuned hidden layers.The learning rate schedule for the last layer /
hidden layers was: 102=104!103=104!
104=104!105=105.
Caltech-101 dataset contains a single class label
per image, so ﬁne-tuning was performed using the
softmax regression loss. Other settings (including
the learning rate schedule) were the same as used
for the VOC ﬁne-tuning experiments.
3.2.3 Low-dimensional CNN feature training
Our baseline networks (Table 1) have the same
dimensionality of the last hidden layer (full7): 4096.
This design choice is in accordance with the stateof-the-art architectures [13], [19], [10], and leads to
a 4096-D dimensional image representation, which
is already rather compact compared to IFV . We
further trained three modiﬁcations of the CNN-M
network, with lower dimensional full7 layers of:
2048, 1024, and 128 dimensions respectively. The


key: chroma:document
string_value:
further trained three modiﬁcations of the CNN-M
network, with lower dimensional full7 layers of:
2048, 1024, and 128 dimensions respectively. The
networks were learnt on ILSVRC-2012. To speedup training, all layers aside from full7/full8 were
set to those of the CNN-M net and a lower initial
learning rate of 103was used. The initial learning
rate of full7/full8 was set to 102.
3.3 Data augmentation details
We explore three data augmentation strategies. The
ﬁrst strategy is to use no augmentation . In contrast to IFV , however, CNNs require images to
6
be transformed to a ﬁxed size ( 224224) even
when no augmentation is used. Hence the image is
downsized so that the smallest dimension is equal
to224 pixels and a 224224 crop is extracted
from the centre.2The second strategy is to use
ﬂip augmentation , mirroring images about the yaxis producing two samples from each image. The
third strategy, termed C+F augmentation , combines
cropping and ﬂipping. For CNN-based representations, the image is downsized so that the smallest
dimension is equal to 256 pixels. Then 224224


key: chroma:document
string_value:
cropping and ﬂipping. For CNN-based representations, the image is downsized so that the smallest
dimension is equal to 256 pixels. Then 224224
crops are extracted from the four corners and the
centre of the image. Note that the crops are sampled
from the whole image, rather than its 256256
centre, as done by [13]. These crops are then ﬂipped
about they-axis, producing 10perturbed samples
per input image. In the case of the IFV encoding, the
same crops are extracted, but at the original image
resolution.
4 A NALYSIS
This section describes the experimental results,
comparing different features and data augmentation schemes. The results are given in Table 2 for
VOC-2007 and analysed next, starting from generally applicable methods such as augmentation and
then discussing the speciﬁcs of each scenario. We
then move onto other datasets and the state of the
art in Sect. 4.7.
4.1 Data augmentation
We experiment with no data augmentation (denoted Image Aug=– in Tab. 2), ﬂip augmentation ( Image Aug=F ), and C+F augmentation ( Image
Aug=C ). Augmented images are used as standalone samples ( f), or by fusing the corresponding


key: chroma:document
string_value:
Aug=C ). Augmented images are used as standalone samples ( f), or by fusing the corresponding
descriptors using sum ( s) or max ( m) pooling or
stacking ( t). So for example Image Aug=(C) f s in
row [f] of Tab. 2 means that C+F augmentation is
used to generate additional samples in training ( f),
and is combined with sum-pooling in testing ( s).
Augmentation consistently improves performance by3%for both IFV ( e.g. [d] vs. [f]) and
CNN ( e.g. [o] vs. [p]). Using additional samples for
training and sum-pooling for testing works best
([p]) followed by sum-pooling [r], max pooling [q],
and stacking [s]. In terms of the choice of transformations, ﬂipping improves only marginally ([o] vs.
2. Extracting a 224224 centre crop from a 256256
image [13] resulted in worse performance.[u]), but using the more expensive C+F sampling
improves, as seen, by about 23%([o]vs. [p]). We
experimented with sampling more transformations,
taking a higher density of crops from the centre of
the image, but observed no beneﬁt.
4.2 Colour


key: chroma:document
string_value:
experimented with sampling more transformations,
taking a higher density of crops from the centre of
the image, but observed no beneﬁt.
4.2 Colour
Colour information can be added and subtracted in
CNN and IFV . In IFV replacing SIFT with the colour
descriptors of [3] (denoted COL inMethod ) yields
signiﬁcantly worse performance ([j] vs. [h]). However, when SIFT and colour descriptors are combined by stacking the corresponding IFVs ( COL+ )
there is a small but signiﬁcant improvement of
around1%in the non-augmented case ( e.g. [h]
vs. [k]) but little impact in the augmented case
(e.g. [i] vs. [l]). For CNNs, retraining the network
after converting all the input images to grayscale
(denoted GSinMethods ) has a more signiﬁcant
impact, resulting in a performance drop of 3%
([w] vs. [p], [v] vs. [o]).
4.3 Scenario 1: Shallow representation (IFV)
The baseline IFV encoding using a spatial pyramid
[a] performs slightly better than the results [I] taken


key: chroma:document
string_value:
4.3 Scenario 1: Shallow representation (IFV)
The baseline IFV encoding using a spatial pyramid
[a] performs slightly better than the results [I] taken
from Chatﬁeld et al. [12], primarily due to a larger
number of spatial scales being used during SIFT
feature extraction, and the resultant SIFT features
being square-rooted. Intra-normalisation , denoted as
INin the Method column of the table, improves the
performance by 1%(e.g. [c] vs. [d]). More interestingly, switching from spatial pooling (denoted
spm in the SPool column) to feature spatial augmentation ( SPool=(x,y) ) has either little effect on the
performance or results in a marginal increase ([a] vs.
[c], [b] vs. [d]), whilst resulting in a representation
which is over 10 smaller. We also experimented
with augmenting with scale in addition to position
as in [23] but observed no improvement. Finally,
we investigate pushing the parameters of the representation setting K= 512 (rows [h]-[l]). Increasing
the number of GMM centres in the model from
K= 256 to512 results in a further performance
increase (e.g. [h] vs. [d]), but at the expense of


key: chroma:document
string_value:
K= 256 to512 results in a further performance
increase (e.g. [h] vs. [d]), but at the expense of
higher-dimensional codes (125K dimensional).
4.4 Scenario 2: Deep representation (CNN) with
pre-training
CNN-based methods consistently outperform the
shallow encodings, even after the improvements
7
Method SPool Image Aug. Dim mAP
(I) FK BL spm – 327K 61.69 79.0 67.4 51.9 70.9 30.8 72.2
(II) DECAF – (C) t t 327K 73.41 87.4 79.3 84.1 78.4 42.3 73.7
(a) FK spm – 327K 63.66 83.4 68.8 59.6 74.1 35.7 71.2
(b) FK IN spm – 327K 64.18 82.1 69.7 59.7 75.2 35.7 71.3
(c) FK (x,y) – 42K 63.51 83.2 69.4 60.6 73.9 36.3 68.6
(d) FK IN (x,y) – 42K 64.36 83.1 70.4 62.4 75.2 37.1 69.1


key: chroma:document
string_value:
(d) FK IN (x,y) – 42K 64.36 83.1 70.4 62.4 75.2 37.1 69.1
(e) FK IN (x,y) (F) f - 42K 64.35 83.1 70.5 62.3 75.4 37.1 69.1
(f) FK IN (x,y) (C) f s 42K 67.17 85.5 71.6 64.6 77.2 39.0 70.8
(g) FK IN (x,y) (C) s s 42K 66.68 84.9 70.1 64.7 76.3 39.2 69.8
(h) FK IN 512 (x,y) – 84K 65.36 84.1 70.4 65.0 76.7 37.2 71.3
(i) FK IN 512 (x,y) (C) f s 84K 68.02 85.9 71.8 67.1 77.1 38.8 72.3
(j) FK IN COL 512 – – 82K 52.18 69.5 52.1 47.5 64.0 24.6 49.8


key: chroma:document
string_value:
(j) FK IN COL 512 – – 82K 52.18 69.5 52.1 47.5 64.0 24.6 49.8
(k) FK IN 512 COL+ (x,y) – 166K 66.37 82.9 70.1 67.0 77.0 36.1 70.0
(l) FK IN 512 COL+ (x,y) (C) f s 166K 67.93 85.1 70.5 67.5 77.4 35.7 71.2
(m) CNN F – (C) f s 4K 77.38 88.7 83.9 87.0 84.7 46.9 77.5
(n) CNN S – (C) f s 4K 79.74 90.7 85.7 88.9 86.6 50.5 80.1
(o) CNN M – – 4K 76.97 89.5 84.3 88.8 83.2 48.4 77.0
(p) CNN M – (C) f s 4K 79.89 91.7 85.4 89.5 86.6 51.6 79.3
(q) CNN M – (C) f m 4K 79.50 90.9 84.6 89.4 85.8 50.3 78.4


key: chroma:document
string_value:
(q) CNN M – (C) f m 4K 79.50 90.9 84.6 89.4 85.8 50.3 78.4
(r) CNN M – (C) s s 4K 79.44 91.4 85.2 89.1 86.1 52.1 78.0
(s) CNN M – (C) t t 41K 78.77 90.7 85.0 89.2 85.8 51.0 77.8
(t) CNN M – (C) f - 4K 77.78 90.5 84.3 88.8 84.5 47.9 78.0
(u) CNN M – (F) f - 4K 76.99 90.1 84.2 89.0 83.5 48.1 77.2
(v) CNN M GS – – 4K 73.59 87.4 80.8 82.4 82.1 44.5 73.5
(w) CNN M GS – (C) f s 4K 77.00 89.4 83.8 85.1 84.4 49.4 77.6
(x) CNN M 2048 – (C) f s 2K 80.10 91.3 85.8 89.9 86.7 52.4 79.7


key: chroma:document
string_value:
(x) CNN M 2048 – (C) f s 2K 80.10 91.3 85.8 89.9 86.7 52.4 79.7
(y) CNN M 1024 – (C) f s 1K 79.91 91.4 86.9 89.3 85.8 53.3 79.8
(z) CNN M 128 – (C) f s 128 78.60 91.3 83.9 89.2 86.9 52.1 81.0
(a) FK+CNN F (x,y) (C) f s 88K 77.95 89.6 83.1 87.1 84.5 48.0 79.4
(b) FK+CNN M 2048 (x,y) (C) f s 86K 80.14 90.9 85.9 88.8 85.5 52.3 81.4
(g) CNN S TUNE-RNK – (C) f s 4K 82.42 95.3 90.4 92.5 89.6 54.4 81.9
TABLE 2
VOC 2007 results (continued overleaf) . See Sect. 4 for details.
discussed above, by a large 10% mAP margin
([i]vs. [p]). Our small architecture CNN-F, which is


key: chroma:document
string_value:
discussed above, by a large 10% mAP margin
([i]vs. [p]). Our small architecture CNN-F, which is
similar to DeCAF [7], performs signiﬁcantly better
than the latter ([II] vs. [s]), validating our implementation. Both medium CNN-M [m] and slow
CNN-S [p] outperform the fast CNN-F [m] by a
signiﬁcant 23% margin. Since the accuracy
of CNN-S and CNN-M is nearly the same, we
focus on the latter as it is simpler and marginally
(25%) faster. Remarkably, these good networks
work very well even with no augmentation [o].
Another advantage of CNNs compared to IFV is
the small dimensionality of the output features,although IFV can be compressed to an extent. We
explored retraining the CNNs such that the ﬁnal
layer was of a lower dimensionality, and reducing
from 4096 to 2048 actually resulted in a marginal
performance boost ([x] vs. [p]). What is surprising
is that we can reduce the output dimensionality
further to 1024D [y] and even 128D [z] with only
a drop of2%for codes that are 32smaller


key: chroma:document
string_value:
further to 1024D [y] and even 128D [z] with only
a drop of2%for codes that are 32smaller
(650smaller than our best performing IFV [i]).
Note,`2-normalising the features accounted for up
to5%of their performance over VOC 2007; it
should be applied before input to the SVM and
after pooling the augmented descriptors (where
8
(I) 79.9 61.4 56.0 49.6 58.4 44.8 78.8 70.8 85.0 31.7 51.0 56.4 80.2 57.5
(II) 83.7 83.7 54.3 61.9 70.2 79.5 85.3 77.2 90.9 51.1 73.8 57.0 86.4 68.0
(a) 80.7 64.4 53.8 53.8 60.2 47.8 79.9 68.9 86.1 37.3 51.1 55.8 83.7 56.9
(b) 80.6 64.8 53.9 54.9 60.7 50.5 80.4 69.5 86.2 38.3 54.4 56.3 82.7 56.7


key: chroma:document
string_value:
(c) 81.1 64.2 51.1 53.4 61.9 50.0 80.0 67.5 85.3 35.7 51.9 53.8 83.5 58.9
(d) 80.5 66.9 50.9 53.9 62.1 51.5 80.5 68.5 85.9 37.2 55.2 54.3 83.3 59.2
(e) 80.5 66.8 51.0 54.1 62.2 51.5 80.4 68.2 86.0 37.3 55.1 54.2 83.3 59.2
(f) 82.4 71.6 52.8 62.4 63.4 57.1 81.6 70.9 86.9 41.2 61.2 56.9 85.2 61.5
(g) 81.9 71.0 52.8 61.6 62.2 56.8 81.8 70.0 86.5 41.5 61.0 56.5 84.3 60.9
(h) 81.1 67.9 52.6 55.4 61.4 51.2 80.5 69.1 86.4 41.2 56.0 56.2 83.7 59.9


key: chroma:document
string_value:
(i) 82.5 73.2 54.7 62.7 64.5 56.6 82.2 71.3 87.5 43.0 62.0 59.3 85.7 62.4
(j) 66.1 46.6 42.5 35.8 41.1 45.5 75.4 58.3 83.9 39.8 47.3 35.6 69.2 49.0
(k) 80.0 65.9 52.8 56.1 61.0 56.9 81.4 69.6 88.4 49.0 59.2 56.4 84.7 62.8
(l) 81.6 70.8 52.9 59.6 63.1 59.9 82.1 70.5 88.9 50.6 63.7 57.5 86.1 64.1
(m) 86.3 85.4 58.6 71.0 72.6 82.0 87.9 80.7 91.8 58.5 77.4 66.3 89.1 71.3
(n) 87.8 88.3 61.3 74.8 74.7 87.2 89.0 83.7 92.3 58.8 80.5 69.4 90.5 74.0


key: chroma:document
string_value:
(o) 85.1 87.4 58.1 70.4 73.1 83.5 85.5 80.9 90.8 54.1 78.9 61.1 89.0 70.4
(p) 87.7 88.6 60.3 80.1 74.4 85.9 88.2 84.6 92.1 60.3 80.5 66.2 91.3 73.5
(q) 87.6 88.6 60.7 78.2 73.6 86.0 87.4 83.8 92.3 59.3 81.0 66.8 91.3 74.0
(r) 87.5 88.1 60.4 76.9 74.8 85.8 88.1 84.3 92.2 59.5 79.3 65.8 90.8 73.5
(s) 87.3 87.6 60.1 72.3 75.3 85.2 86.9 82.6 91.9 58.5 77.9 66.5 90.5 73.4
(t) 85.7 87.9 58.3 74.2 73.9 84.7 86.6 82.0 91.0 55.8 79.2 62.1 89.3 71.0


key: chroma:document
string_value:
(u) 85.3 87.3 58.1 70.0 73.4 83.5 86.0 80.8 90.9 53.9 78.1 61.2 88.8 70.6
(v) 85.0 84.9 57.8 65.9 69.8 79.5 82.9 77.4 89.2 42.8 71.7 60.2 86.3 67.8
(w) 87.2 86.5 59.5 72.4 74.1 81.7 86.0 82.3 90.8 48.9 73.7 66.8 89.6 71.0
(x) 87.6 88.4 60.2 76.9 75.4 85.5 88.0 83.4 92.1 61.1 83.1 68.5 91.9 74.2
(y) 87.8 88.6 59.0 77.2 73.1 85.9 88.3 83.5 91.8 59.9 81.4 68.3 93.0 74.1
(z) 86.6 87.5 59.1 70.0 72.9 84.6 86.7 83.6 89.4 57.0 81.5 64.8 90.4 73.4


key: chroma:document
string_value:
(a) 86.8 85.6 59.9 72.0 73.4 81.4 88.6 80.5 92.1 60.6 77.3 66.4 89.3 73.3
(b) 87.7 88.4 61.2 76.9 76.6 84.9 89.1 82.9 92.4 61.9 80.9 68.7 91.5 75.1
(g) 91.5 91.9 64.1 76.3 74.9 89.7 92.2 86.9 95.2 60.7 82.9 68.0 95.5 74.4
TABLE 2
VOC 2007 results (continued from previous page)
applicable).
4.5 Scenario 3: Deep representation (CNN) with
pre-training and ﬁne-tuning
We ﬁne-tuned our CNN-S architecture on VOC2007 using the ranking hinge loss, and achieved
a signiﬁcant improvement: 2:7%([g]vs. [n]). This
demonstrates that in spite of the small amount
of VOC training data (5,011 images), ﬁne-tuning
is able to adjust the learnt deep representation to
better suit the dataset in question.4.6 Combinations
For the CNN-M 2048 representation [x], stacking


key: chroma:document
string_value:
is able to adjust the learnt deep representation to
better suit the dataset in question.4.6 Combinations
For the CNN-M 2048 representation [x], stacking
deep and shallow representations to form a higherdimensional descriptor makes little difference ([x]
vs. [b]). For the weaker CNN-F it results in a small
boost of0:8%([m] vs. [a]).
4.7 Comparison with the state of the art
In Table 3 we report our results on ILSVRC-2012,
VOC-2007, VOC-2012, Caltech-101, and Caltech-256
datasets, and compare them to the state of the art.
First, we note that the ILSVRC error rates of our
CNN-F, CNN-M, and CNN-S networks are better
9
ILSVRC-2012 VOC-2007 VOC-2012 Caltech-101 Caltech-256
(top-5 error) (mAP) (mAP) (accuracy) (accuracy)
(a) FK IN 512 - 68.0 – – –
(b) CNN F 16.7 77.4 79.9 – –
(c) CNN M 13.7 79.9 82.5 87.15 0.80 77.030.46


key: chroma:document
string_value:
(c) CNN M 13.7 79.9 82.5 87.15 0.80 77.030.46
(d) CNN M 2048 13.5 80.1 82.4 86.64 0.53 76.880.35
(e) CNN S 13.1 79.7 82.9 87.76 0.66 77.610.12
(f) CNN S TUNE-CLS 13.1 - 83.0 88.350.56 77.330.56
(g) CNN S TUNE-RNK 13.1 82.4 83.2 – –
(h) Zeiler & Fergus [19] 16.1 - 79.0 86.5 0.5 74.20.3
(i) Razavian et al. [9], [10] 14.7 77.2 – – –
(j) Oquab et al. [8] 18 77.7 78.7 (82.8*) – –
(k) Oquab et al. [16] - - 86.3*– –
(l) Wei et al. [17] - 81.5 ( 85.2*) 81.7 ( 90.3*) – –


key: chroma:document
string_value:
(l) Wei et al. [17] - 81.5 ( 85.2*) 81.7 ( 90.3*) – –
(m) He et al. [29] 13.6 80.1 - 91.40.7 –
TABLE 3
Comparison with the state of the art on ILSVRC2012, VOC2007, VOC2012, Caltech-101, and
Caltech-256. Results marked with * were achieved using models pre-trained on the extended ILSVRC
datasets (1512 classes in [8], [16], 2000 classes in [17]). All other results were achieved using CNNs
pre-trained on ILSVRC-2012 (1000 classes).
than those reported by [13], [19], and [10] for the
related conﬁgurations. This validates our implementation, and the difference is likely to be due to
the sampling of image crops from the uncropped
image plane (instead of the centre). When using
our CNN features on other datasets, the relative
performance generally follows the same pattern
as on ILSVRC, where the nets are trained – the
CNN-F architecture exhibits the worst performance,
with CNN-M and CNN-S performing considerably
better.
Further ﬁne-tuning of CNN-S on the VOC


key: chroma:document
string_value:
CNN-F architecture exhibits the worst performance,
with CNN-M and CNN-S performing considerably
better.
Further ﬁne-tuning of CNN-S on the VOC
datasets turns out to be beneﬁcial; on VOC-2012,
using the ranking loss is marginally better than
the classiﬁcation loss ([g] vs. [f]), which can be
explained by the ranking-based VOC evaluation
criterion. Fine-tuning on Caltech-101 also yields a
small improvement, but no gain is observed over
Caltech-256.
Our CNN-S net is competitive with recent CNNbased approaches [19], [9], [8], [16], [17], [29] and
on a number of datasets (VOC-2007, VOC-2012,
Caltech-101, Caltech-256) and sets the state of the
art on VOC-2007 and VOC-2012 across methods
pre-trained solely on ILSVRC-2012 dataset. While
the CNN-based methods of [16], [17] achieve better


key: chroma:document
string_value:
pre-trained solely on ILSVRC-2012 dataset. While
the CNN-based methods of [16], [17] achieve better
performance on VOC (86.3% and 90.3% respectively), they were trained using extended ILSVRCdatasets, enriched with additional categories semantically close to the ones in VOC. Additionally, [17] used a signiﬁcantly more complex classiﬁcation pipeline, driven by bounding box proposals [30], pre-trained on ILSVRC-2013 detection
dataset. Their best reported result on VOC-2012
(90.3%) was achieved by the late fusion with a
complex hand-crafted method of [31]; without fusion, they get 84.2%. On Caltech-101, [29] achieves
the state of the art using spatial pyramid pooling
of conv5 layer features, while we used full7 layer
features consistently across all datasets (for full7
features, they report 87:08%).
In addition to achieving performance comparable
to the state of the art with a very simple approach
(but powerful CNN-based features), with the modiﬁcations outlined in the paper (primarily the use
of data augmentation similar to the CNN-based
methods) we are able to improve the performance


key: chroma:document
string_value:
of data augmentation similar to the CNN-based
methods) we are able to improve the performance
of shallow IFV to 68.02% (Table 2, [i]).
4.8 Performance Evolution on VOC-2007
A comparative plot of the evolution in the performance of the methods evaluated in this paper,
along with a selection from our earlier review of
shallow methods [12] is presented in Fig. 1. Classiﬁcation accuracy over PASCAL VOC was 54.48%
mAP for the BoVW model in 2008, 61.7% for the
10
      BOW
32K
–FK-BL
327K
–
[I]FK
327K
–
[a]FK-IN
84K
f s
[i]DeCAF
4K
t t
[II]CNN-F
4K
f s
[m]CNN-M 2K
2K
f s
[y]CNN-S
4K (TN)
f s
[γ]   
                           
545658606264666870727476788082mAP 68.02
54.4861.6963.6673.4177.1580.132008 2010 2013 2014 ...
82.42
Method
Dim.
Aug.
Ref.


key: chroma:document
string_value:
54.4861.6963.6673.4177.1580.132008 2010 2013 2014 ...
82.42
Method
Dim.
Aug.
Ref.
Fig. 1. Evolution of Performance on PASCAL VOC-2007 over the recent years. Please refer to Table 2
for details and references.
IFV in 2010 [12], and 73.41% for DeCAF [7] and
similar [8], [9] CNN-based methods introduced in
late 2013. Our best performing CNN-based method
(CNN-S with ﬁne-tuning) achieves 82.42%, comparable to the most recent state-of-the-art.
4.9 Timings and dimensionality
One of our best-performing CNN representations
CNN-M-2048 [x] is 42more compact than the
best performing IFV [i] (84K vs. 2K) and CNN-M
features are also 50faster to compute ( 120s
vs.2:4sper image with augmentation enabled,
over a single CPU core). Non-augmented CNN-M
features [o] take around 0:3sper image, compared
to0:4sfor CNN-S features and 0:13sfor CNNF features.
5 C ONCLUSION


key: chroma:document
string_value:
to0:4sfor CNN-S features and 0:13sfor CNNF features.
5 C ONCLUSION
In this paper we presented a rigorous empirical evaluation of CNN-based methods for image
classiﬁcation, along with a comparison with more
traditional shallow feature encoding methods. We
have demonstrated that the performance of shallow representations can be signiﬁcantly improved
by adopting data augmentation, typically used indeep learning. In spite of this improvement, deep
architectures still outperform the shallow methods
by a large margin. We have shown that the performance of deep representations on the ILSVRC
dataset is a good indicator of their performance
on other datasets, and that ﬁne-tuning can further
improve on already very strong results achieved
using the combination of deep representations and
a linear SVM. Source code and CNN models to
reproduce the experiments presented in the paper
are available on the project website [32] in the
hope that it would provide common ground for
future comparisons, and good baselines for image
representation research.
ACKNOWLEDGEMENTS
This work was supported by the EPSRC and ERC
grant VisRec no. 228180. We gratefully acknowledge the support of NVIDIA Corporation with the
donation of the GPUs used for this research.
REFERENCES


key: chroma:document
string_value:
grant VisRec no. 228180. We gratefully acknowledge the support of NVIDIA Corporation with the
donation of the GPUs used for this research.
REFERENCES
[1] G. Csurka, C. Bray, C. Dance, and L. Fan, “Visual categorization with bags of keypoints,” in Workshop on Statistical
Learning in Computer Vision, ECCV , 2004, pp. 1–22.
11
[2] J. Sivic and A. Zisserman, “Video Google: A text retrieval
approach to object matching in videos,” in Proc. ICCV ,
vol. 2, 2003, pp. 1470–1477.
[3] F. Perronnin, J. S ´anchez, and T. Mensink, “Improving the
Fisher kernel for large-scale image classiﬁcation,” in Proc.
ECCV , 2010.
[4] Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E.
Howard, W. Hubbard, and L. D. Jackel, “Backpropagation applied to handwritten zip code recognition,” Neural
Computation , vol. 1, no. 4, pp. 541–551, 1989.


key: chroma:document
string_value:
Computation , vol. 1, no. 4, pp. 541–551, 1989.
[5] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei,
“Imagenet: A large-scale hierarchical image database,” in
Proc. CVPR , 2009.
[6] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn,
and A. Zisserman, “The PASCAL Visual Object Classes
(VOC) challenge,” IJCV , vol. 88, no. 2, pp. 303–338, 2010.
[7] J. Donahue, Y. Jia, O. Vinyals, J. Hoffman, N. Zhang,
E. Tzeng, and T. Darrell, “Decaf: A deep convolutional
activation feature for generic visual recognition,” CoRR ,
vol. abs/1310.1531, 2013.
[8] M. Oquab, L. Bottou, I. Laptev, and J. Sivic, “Learning
and Transferring Mid-Level Image Representations using
Convolutional Neural Networks,” in Proc. CVPR , 2014.


key: chroma:document
string_value:
and Transferring Mid-Level Image Representations using
Convolutional Neural Networks,” in Proc. CVPR , 2014.
[9] A. Razavian, H. Azizpour, J. Sullivan, and S. Carlsson,
“CNN Features off-the-shelf: an Astounding Baseline for
Recognition,” CoRR , vol. abs/1403.6382, 2014.
[10] P . Sermanet, D. Eigen, X. Zhang, M. Mathieu, R. Fergus,
and Y. LeCun, “OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks,” in
Proc. ICLR , 2014.
[11] R. B. Girshick, J. Donahue, T. Darrell, and J. Malik, “Rich
feature hierarchies for accurate object detection and semantic segmentation,” in Proc. CVPR , 2014.
[12] K. Chatﬁeld, V . Lempitsky, A. Vedaldi, and A. Zisserman,
“The devil is in the details: an evaluation of recent feature
encoding methods,” in Proc. BMVC. , 2011.


key: chroma:document
string_value:
“The devil is in the details: an evaluation of recent feature
encoding methods,” in Proc. BMVC. , 2011.
[13] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet
classiﬁcation with deep convolutional neural networks,”
inNIPS , 2012, pp. 1106–1114.
[14] M. Paulin, J. Revaud, Z. Harchaoui, F. Perronnin, and
C. Schmid, “Transformation Pursuit for Image Classiﬁcation,” in Proc. CVPR , 2014.
[15] F. Perronnin, Z. Akata, Z. Harchaoui, and C. Schmid,
“Towards good practice in large-scale learning for image
classiﬁcation,” in Proc. CVPR , 2012, pp. 3482–3489.
[16] M. Oquab, L. Bottou, I. Laptev, and J. Sivic, “Weakly
supervised object recognition with convolutional neural
networks,” INRIA, Tech. Rep. HAL-01015140, 2014.


key: chroma:document
string_value:
supervised object recognition with convolutional neural
networks,” INRIA, Tech. Rep. HAL-01015140, 2014.
[17] Y. Wei, W. Xia, J. Huang, B. Ni, J. Dong, Y. Zhao, and
S. Yan, “CNN: Single-label to multi-label,” CoRR , vol.
abs/1406.5726, 2014.
[18] H. J ´egou, F. Perronnin, M. Douze, J. S ´anchez, P . P ´erez, and
C. Schmid, “Aggregating local images descriptors into
compact codes,” IEEE P AMI , 2012.
[19] M. D. Zeiler and R. Fergus, “Visualizing and understanding convolutional networks,” CoRR , vol. abs/1311.2901,
2013.
[20] Y. Jia, “Caffe: An open source convolutional architecture
for fast feature embedding,” http://caffe.berkeleyvision.
org/, 2013.
[21] L. Fei-Fei, R. Fergus, and P . Perona, “Learning generative


key: chroma:document
string_value:
org/, 2013.
[21] L. Fei-Fei, R. Fergus, and P . Perona, “Learning generative
visual models from few training examples: An incremental bayesian approach tested on 101 object categories,”
inIEEE CVPR Workshop of Generative Model Based Vision ,
2004.[22] G. Grifﬁn, A. Holub, and P . Perona, “Caltech-256 object
category dataset,” California Institute of Technology,
Tech. Rep. 7694, 2007. [Online]. Available: http://
authors.library.caltech.edu/7694
[23] J. S ´anchez, F. Perronnin, and T. Em ´ıdio de Campos,
“Modeling the spatial layout of images beyond spatial
pyramids,” Pattern Recognition Letters , vol. 33, no. 16, pp.
2216–2223, 2012.
[24] R. Arandjelovi ´c and A. Zisserman, “Three things everyone should know to improve object retrieval,” in Proc.
CVPR , 2012.
[25] A. Vedaldi and A. Zisserman, “Efﬁcient additive kernels
via explicit feature maps,” IEEE P AMI , 2011.


key: chroma:document
string_value:
[25] A. Vedaldi and A. Zisserman, “Efﬁcient additive kernels
via explicit feature maps,” IEEE P AMI , 2011.
[26] S. Lazebnik, C. Schmid, and J. Ponce, “Beyond Bags
of Features: Spatial Pyramid Matching for Recognizing
Natural Scene Categories,” in Proc. CVPR , 2006.
[27] R. Arandjelovi ´c and A. Zisserman, “All about VLAD,” in
Proc. CVPR , 2013.
[28] H. J ´egou, M. Douze, and C. Schmid, “On the burstiness
of visual elements,” in Proc. CVPR , Jun 2009.
[29] K. He, A. Zhang, S. Ren, and J. Sun, “Spatial pyramid
pooling in deep convolutional networks for visual recognition,” in Proc. ECCV , 2014.
[30] M.-M. Cheng, Z. Zhang, W.-Y. Lin, and P . H. S. Torr,
“BING: Binarized normed gradients for objectness estimation at 300fps,” in Proc. CVPR , 2014.


key: chroma:document
string_value:
“BING: Binarized normed gradients for objectness estimation at 300fps,” in Proc. CVPR , 2014.
[31] S. Yan, J. Dong, Q. Chen, Z. Song, Y. Pan, W. Xia,
H. Zhongyang, Y. Hua, and S. Shen, “Generalized hierarchical matching for subcategory aware object classiﬁcation,” in The P ASCAL Visual Object Classes Challenge
Workshop , 2012.
[32] K. Chatﬁeld, K. Simonyan, A. Vedaldi, and A. Zisserman,
“Return of the devil in the details: delving deep into
convolutional nets webpage,” 2014. [Online]. Available:
http://www.robots.ox.ac.uk/vgg/research/deep eval


key: chroma:document
string_value:
Learning Phrase Representations using RNN Encoder–Decoder
for Statistical Machine Translation
Kyunghyun Cho
Bart van Merri ¨enboer Caglar Gulcehre
Universit ´e de Montr ´eal
firstname.lastname@umontreal.caDzmitry Bahdanau
Jacobs University, Germany
d.bahdanau@jacobs-university.de
Fethi Bougares Holger Schwenk
Universit ´e du Maine, France
firstname.lastname@lium.univ-lemans.frYoshua Bengio
Universit ´e de Montr ´eal, CIFAR Senior Fellow
find.me@on.the.web
Abstract
In this paper, we propose a novel neural network model called RNN Encoder–
Decoder that consists of two recurrent
neural networks (RNN). One RNN encodes a sequence of symbols into a ﬁxedlength vector representation, and the other
decodes the representation into another sequence of symbols. The encoder and decoder of the proposed model are jointly
trained to maximize the conditional probability of a target sequence given a source


key: chroma:document
string_value:
decodes the representation into another sequence of symbols. The encoder and decoder of the proposed model are jointly
trained to maximize the conditional probability of a target sequence given a source
sequence. The performance of a statistical machine translation system is empirically found to improve by using the conditional probabilities of phrase pairs computed by the RNN Encoder–Decoder as an
additional feature in the existing log-linear
model. Qualitatively, we show that the
proposed model learns a semantically and
syntactically meaningful representation of
linguistic phrases.
1 Introduction
Deep neural networks have shown great success in
various applications such as objection recognition
(see, e.g., (Krizhevsky et al., 2012)) and speech
recognition (see, e.g., (Dahl et al., 2012)). Furthermore, many recent works showed that neural networks can be successfully used in a number of tasks in natural language processing (NLP).
These include, but are not limited to, language
modeling (Bengio et al., 2003), paraphrase detection (Socher et al., 2011) and word embedding extraction (Mikolov et al., 2013). In the ﬁeld of statistical machine translation (SMT), deep neural
networks have begun to show promising results.
(Schwenk, 2012) summarizes a successful usage
of feedforward neural networks in the framework


key: chroma:document
string_value:
networks have begun to show promising results.
(Schwenk, 2012) summarizes a successful usage
of feedforward neural networks in the framework
of phrase-based SMT system.Along this line of research on using neural networks for SMT, this paper focuses on a novel neural network architecture that can be used as a part
of the conventional phrase-based SMT system.
The proposed neural network architecture, which
we will refer to as an RNN Encoder–Decoder , consists of two recurrent neural networks (RNN) that
act as an encoder and a decoder pair. The encoder maps a variable-length source sequence to a
ﬁxed-length vector, and the decoder maps the vector representation back to a variable-length target
sequence. The two networks are trained jointly to
maximize the conditional probability of the target
sequence given a source sequence. Additionally,
we propose to use a rather sophisticated hidden
unit in order to improve both the memory capacity
and the ease of training.
The proposed RNN Encoder–Decoder with a
novel hidden unit is empirically evaluated on the
task of translating from English to French. We
train the model to learn the translation probability of an English phrase to a corresponding French
phrase. The model is then used as a part of a standard phrase-based SMT system by scoring each


key: chroma:document
string_value:
train the model to learn the translation probability of an English phrase to a corresponding French
phrase. The model is then used as a part of a standard phrase-based SMT system by scoring each
phrase pair in the phrase table. The empirical evaluation reveals that this approach of scoring phrase
pairs with an RNN Encoder–Decoder improves
the translation performance.
We qualitatively analyze the trained RNN
Encoder–Decoder by comparing its phrase scores
with those given by the existing translation model.
The qualitative analysis shows that the RNN
Encoder–Decoder is better at capturing the linguistic regularities in the phrase table, indirectly
explaining the quantitative improvements in the
overall translation performance. The further analysis of the model reveals that the RNN Encoder–
Decoder learns a continuous space representation
of a phrase that preserves both the semantic and
syntactic structure of the phrase.arXiv:1406.1078v3  [cs.CL]  3 Sep 2014
2 RNN Encoder–Decoder
2.1 Preliminary: Recurrent Neural Networks
A recurrent neural network (RNN) is a neural network that consists of a hidden state hand an
optional output ywhich operates on a variablelength sequence x= (x1;:::;x T). At each time
stept, the hidden state hhtiof the RNN is updated
by


key: chroma:document
string_value:
stept, the hidden state hhtiof the RNN is updated
by
hhti=f
hht1i;xt
; (1)
wherefis a non-linear activation function.fmay be as simple as an elementwise logistic sigmoid function and as complex as a long short-term memory (LSTM)
unit (Hochreiter and Schmidhuber, 1997).
An RNN can learn a probability distribution
over a sequence by being trained to predict the
next symbol in a sequence. In that case, the output
at each timestep tis the conditional distribution
p(xtjxt1;:::;x 1). For example, a multinomial
distribution ( 1-of-Kcoding) can be output using a
softmax activation function
p(xt;j= 1jxt1;:::;x 1) =exp
wjhhti
PK
j0=1exp
wj0hhti;
(2)
for all possible symbols j= 1;:::;K , where wj
are the rows of a weight matrix W. By combining
these probabilities, we can compute the probability of the sequence xusing
p(x) =TY


key: chroma:document
string_value:
are the rows of a weight matrix W. By combining
these probabilities, we can compute the probability of the sequence xusing
p(x) =TY
t=1p(xtjxt1;:::;x 1): (3)
From this learned distribution, it is straightforward to sample a new sequence by iteratively sampling a symbol at each time step.
2.2 RNN Encoder–Decoder
In this paper, we propose a novel neural network
architecture that learns to encode a variable-length
sequence into a ﬁxed-length vector representation
and to decode a given ﬁxed-length vector representation back into a variable-length sequence.
From a probabilistic perspective, this new model
is a general method to learn the conditional distribution over a variable-length sequence conditioned on yet another variable-length sequence,
e.g.p(y1;:::;y T0jx1;:::;x T), where one
x1x2 xTyT' y2 y1
cDecoder
EncoderFigure 1: An illustration of the proposed RNN
Encoder–Decoder.
should note that the input and output sequence
lengthsTandT0may differ.
The encoder is an RNN that reads each symbol
of an input sequence xsequentially. As it reads


key: chroma:document
string_value:
lengthsTandT0may differ.
The encoder is an RNN that reads each symbol
of an input sequence xsequentially. As it reads
each symbol, the hidden state of the RNN changes
according to Eq. (1). After reading the end of
the sequence (marked by an end-of-sequence symbol), the hidden state of the RNN is a summary c
of the whole input sequence.
The decoder of the proposed model is another
RNN which is trained to generate the output sequence by predicting the next symbol ytgiven the
hidden state hhti. However, unlike the RNN described in Sec. 2.1, both ytandhhtiare also conditioned onyt1and on the summary cof the input
sequence. Hence, the hidden state of the decoder
at timetis computed by,
hhti=f
hht1i;yt1;c
;
and similarly, the conditional distribution of the
next symbol is
P(ytjyt1;yt2;:::;y 1;c) =g
hhti;yt1;c
:
for given activation functions fandg(the latter
must produce valid probabilities, e.g. with a softmax).
See Fig. 1 for a graphical depiction of the proposed model architecture.


key: chroma:document
string_value:
for given activation functions fandg(the latter
must produce valid probabilities, e.g. with a softmax).
See Fig. 1 for a graphical depiction of the proposed model architecture.
The two components of the proposed RNN
Encoder–Decoder are jointly trained to maximize
the conditional log-likelihood
max
1
NNX
n=1logp(ynjxn); (4)
where is the set of the model parameters and
each (xn;yn)is an (input sequence, output sequence) pair from the training set. In our case,
as the output of the decoder, starting from the input, is differentiable, we can use a gradient-based
algorithm to estimate the model parameters.
Once the RNN Encoder–Decoder is trained, the
model can be used in two ways. One way is to use
the model to generate a target sequence given an
input sequence. On the other hand, the model can
be used to score a given pair of input and output
sequences, where the score is simply a probability
p(yjx)from Eqs. (3) and (4).
2.3 Hidden Unit that Adaptively Remembers
and Forgets
In addition to a novel model architecture, we also
propose a new type of hidden unit ( fin Eq. (1))


key: chroma:document
string_value:
and Forgets
In addition to a novel model architecture, we also
propose a new type of hidden unit ( fin Eq. (1))
that has been motivated by the LSTM unit but is
much simpler to compute and implement.1Fig. 2
shows the graphical depiction of the proposed hidden unit.
Let us describe how the activation of the j-th
hidden unit is computed. First, the reset gaterjis
computed by
rj=
[Wrx]j+
Urhht1i
j
; (5)
whereis the logistic sigmoid function, and [:]j
denotes the j-th element of a vector. xandht1
are the input and the previous hidden state, respectively. WrandUrare weight matrices which are
learned.
Similarly, the update gatezjis computed by
zj=
[Wzx]j+
Uzhht1i
j
: (6)
The actual activation of the proposed unit hjis
then computed by
hhti
j=zjhht1i
j + (1zj)~hhti
j; (7)
where
~hhti
j=
[Wx]j+
U


key: chroma:document
string_value:
j; (7)
where
~hhti
j=
[Wx]j+
U
rhht1i
j
:(8)
In this formulation, when the reset gate is close
to 0, the hidden state is forced to ignore the previous hidden state and reset with the current input
1The LSTM unit, which has shown impressive results in
several applications such as speech recognition, has a memory cell and four gating units that adaptively control the information ﬂow inside the unit, compared to only two gating
units in the proposed hidden unit. For details on LSTM networks, see, e.g., (Graves, 2012).
z
rh h~xFigure 2: An illustration of the proposed hidden
activation function. The update gate zselects
whether the hidden state is to be updated with
a new hidden state ~h. The reset gate rdecides
whether the previous hidden state is ignored. See
Eqs. (5)–(8) for the detailed equations of r,z,h
and~h.
only. This effectively allows the hidden state to
drop any information that is found to be irrelevant
later in the future, thus, allowing a more compact
representation.
On the other hand, the update gate controls how
much information from the previous hidden state


key: chroma:document
string_value:
later in the future, thus, allowing a more compact
representation.
On the other hand, the update gate controls how
much information from the previous hidden state
will carry over to the current hidden state. This
acts similarly to the memory cell in the LSTM
network and helps the RNN to remember longterm information. Furthermore, this may be considered an adaptive variant of a leaky-integration
unit (Bengio et al., 2013).
As each hidden unit has separate reset and update gates, each hidden unit will learn to capture
dependencies over different time scales. Those
units that learn to capture short-term dependencies
will tend to have reset gates that are frequently active, but those that capture longer-term dependencies will have update gates that are mostly active.
In our preliminary experiments, we found that
it is crucial to use this new unit with gating units.
We were not able to get meaningful result with an
oft-used tanh unit without any gating.
3 Statistical Machine Translation
In a commonly used statistical machine translation
system (SMT), the goal of the system (decoder,
speciﬁcally) is to ﬁnd a translation fgiven a source
sentence e, which maximizes
p(fje)/p(ejf)p(f);
where the ﬁrst term at the right hand side is called


key: chroma:document
string_value:
sentence e, which maximizes
p(fje)/p(ejf)p(f);
where the ﬁrst term at the right hand side is called
translation model and the latter language model
(see, e.g., (Koehn, 2005)). In practice, however,
most SMT systems model logp(fje)as a loglinear model with additional features and corresponding weights:
logp(fje) =NX
n=1wnfn(f;e) + logZ(e);(9)
wherefnandwnare then-th feature and weight,
respectively. Z(e)is a normalization constant that
does not depend on the weights. The weights are
often optimized to maximize the BLEU score on a
development set.
In the phrase-based SMT framework
introduced in (Koehn et al., 2003) and
(Marcu and Wong, 2002), the translation model
logp(ejf)is factorized into the translation
probabilities of matching phrases in the source
and target sentences.2These probabilities are
once again considered additional features in the
log-linear model (see Eq. (9)) and are weighted
accordingly to maximize the BLEU score.
Since the neural net language model was proposed in (Bengio et al., 2003), neural networks


key: chroma:document
string_value:
accordingly to maximize the BLEU score.
Since the neural net language model was proposed in (Bengio et al., 2003), neural networks
have been used widely in SMT systems. In
many cases, neural networks have been used to
rescore translation hypotheses ( n-best lists) (see,
e.g., (Schwenk et al., 2006)). Recently, however,
there has been interest in training neural networks
to score the translated sentence (or phrase pairs)
using a representation of the source sentence as
an additional input. See, e.g., (Schwenk, 2012),
(Son et al., 2012) and (Zou et al., 2013).
3.1 Scoring Phrase Pairs with RNN
Encoder–Decoder
Here we propose to train the RNN Encoder–
Decoder (see Sec. 2.2) on a table of phrase pairs
and use its scores as additional features in the loglinear model in Eq. (9) when tuning the SMT decoder.
When we train the RNN Encoder–Decoder, we
ignore the (normalized) frequencies of each phrase
pair in the original corpora. This measure was
taken in order (1) to reduce the computational expense of randomly selecting phrase pairs from a


key: chroma:document
string_value:
pair in the original corpora. This measure was
taken in order (1) to reduce the computational expense of randomly selecting phrase pairs from a
large phrase table according to the normalized frequencies and (2) to ensure that the RNN Encoder–
Decoder does not simply learn to rank the phrase
pairs according to their numbers of occurrences.
One underlying reason for this choice was that the
existing translation probability in the phrase table already reﬂects the frequencies of the phrase
2Without loss of generality, from here on, we refer to
p(ejf)for each phrase pair as a translation model as wellpairs in the original corpus. With a ﬁxed capacity
of the RNN Encoder–Decoder, we try to ensure
that most of the capacity of the model is focused
toward learning linguistic regularities, i.e., distinguishing between plausible and implausible translations, or learning the “manifold” (region of probability concentration) of plausible translations.
Once the RNN Encoder–Decoder is trained, we
add a new score for each phrase pair to the existing phrase table. This allows the new scores to enter into the existing tuning algorithm with minimal
additional overhead in computation.
As Schwenk pointed out in (Schwenk, 2012),
it is possible to completely replace the existing


key: chroma:document
string_value:
additional overhead in computation.
As Schwenk pointed out in (Schwenk, 2012),
it is possible to completely replace the existing
phrase table with the proposed RNN Encoder–
Decoder. In that case, for a given source phrase,
the RNN Encoder–Decoder will need to generate
a list of (good) target phrases. This requires, however, an expensive sampling procedure to be performed repeatedly. In this paper, thus, we only
consider rescoring the phrase pairs in the phrase
table.
3.2 Related Approaches: Neural Networks in
Machine Translation
Before presenting the empirical results, we discuss
a number of recent works that have proposed to
use neural networks in the context of SMT.
Schwenk in (Schwenk, 2012) proposed a similar approach of scoring phrase pairs. Instead of the
RNN-based neural network, he used a feedforward
neural network that has ﬁxed-size inputs (7 words
in his case, with zero-padding for shorter phrases)
and ﬁxed-size outputs (7 words in the target language). When it is used speciﬁcally for scoring
phrases for the SMT system, the maximum phrase
length is often chosen to be small. However, as the
length of phrases increases or as we apply neural


key: chroma:document
string_value:
phrases for the SMT system, the maximum phrase
length is often chosen to be small. However, as the
length of phrases increases or as we apply neural
networks to other variable-length sequence data,
it is important that the neural network can handle variable-length input and output. The proposed RNN Encoder–Decoder is well-suited for
these applications.
Similar to (Schwenk, 2012), Devlin et al.
(Devlin et al., 2014) proposed to use a feedforward neural network to model a translation model,
however, by predicting one word in a target phrase
at a time. They reported an impressive improvement, but their approach still requires the maximum length of the input phrase (or context words)
to be ﬁxed a priori.
Although it is not exactly a neural network they
train, the authors of (Zou et al., 2013) proposed
to learn a bilingual embedding of words/phrases.
They use the learned embedding to compute the
distance between a pair of phrases which is used
as an additional score of the phrase pair in an SMT
system.
In (Chandar et al., 2014), a feedforward neural
network was trained to learn a mapping from a
bag-of-words representation of an input phrase to
an output phrase. This is closely related to both the


key: chroma:document
string_value:
network was trained to learn a mapping from a
bag-of-words representation of an input phrase to
an output phrase. This is closely related to both the
proposed RNN Encoder–Decoder and the model
proposed in (Schwenk, 2012), except that their input representation of a phrase is a bag-of-words.
A similar approach of using bag-of-words representations was proposed in (Gao et al., 2013) as
well. Earlier, a similar encoder–decoder model using two recursive neural networks was proposed
in (Socher et al., 2011), but their model was restricted to a monolingual setting, i.e. the model
reconstructs an input sentence. More recently, another encoder–decoder model using an RNN was
proposed in (Auli et al., 2013), where the decoder is conditioned on a representation of either
a source sentence or a source context.
One important difference between the proposed RNN Encoder–Decoder and the approaches
in (Zou et al., 2013) and (Chandar et al., 2014) is
that the order of the words in source and target phrases is taken into account. The RNN
Encoder–Decoder naturally distinguishes between
sequences that have the same words but in a different order, whereas the aforementioned approaches
effectively ignore order information.
The closest approach related to the proposed


key: chroma:document
string_value:
sequences that have the same words but in a different order, whereas the aforementioned approaches
effectively ignore order information.
The closest approach related to the proposed
RNN Encoder–Decoder is the Recurrent Continuous Translation Model (Model 2) proposed in
(Kalchbrenner and Blunsom, 2013). In their paper, they proposed a similar model that consists
of an encoder and decoder. The difference with
our model is that they used a convolutional n-gram
model (CGM) for the encoder and the hybrid of
an inverse CGM and a recurrent neural network
for the decoder. They, however, evaluated their
model on rescoring the n-best list proposed by the
conventional SMT system and computing the perplexity of the gold standard translations.
4 Experiments
We evaluate our approach on the English/French
translation task of the WMT’14 workshop.4.1 Data and Baseline System
Large amounts of resources are available to build
an English/French SMT system in the framework
of the WMT’14 translation task. The bilingual
corpora include Europarl (61M words), news commentary (5.5M), UN (421M), and two crawled
corpora of 90M and 780M words respectively.
The last two corpora are quite noisy. To train


key: chroma:document
string_value:
corpora of 90M and 780M words respectively.
The last two corpora are quite noisy. To train
the French language model, about 712M words of
crawled newspaper material is available in addition to the target side of the bitexts. All the word
counts refer to French words after tokenization.
It is commonly acknowledged that training statistical models on the concatenation of all this
data does not necessarily lead to optimal performance, and results in extremely large models which are difﬁcult to handle. Instead, one
should focus on the most relevant subset of the
data for a given task. We have done so by
applying the data selection method proposed in
(Moore and Lewis, 2010), and its extension to bitexts (Axelrod et al., 2011). By these means we
selected a subset of 418M words out of more
than 2G words for language modeling and a
subset of 348M out of 850M words for training the RNN Encoder–Decoder. We used the
test set newstest2012 and 2013 for data
selection and weight tuning with MERT, and
newstest2014 as our test set. Each set has
more than 70 thousand words and a single reference translation.
For training the neural networks, including the
proposed RNN Encoder–Decoder, we limited the
source and target vocabulary to the most frequent


key: chroma:document
string_value:
For training the neural networks, including the
proposed RNN Encoder–Decoder, we limited the
source and target vocabulary to the most frequent
15,000 words for both English and French. This
covers approximately 93% of the dataset. All the
out-of-vocabulary words were mapped to a special
token ( [UNK ]).
The baseline phrase-based SMT system was
built using Moses with default settings. This system achieves a BLEU score of 30.64 and 33.3 on
the development and test sets, respectively (see Table 1).
4.1.1 RNN Encoder–Decoder
The RNN Encoder–Decoder used in the experiment had 1000 hidden units with the proposed
gates at the encoder and at the decoder. The input matrix between each input symbol xhtiand the
hidden unit is approximated with two lower-rank
matrices, and the output matrix is approximated
ModelsBLEU
dev test
Baseline 30.64 33.30
RNN 31.20 33.87
CSLM + RNN 31.48 34.64
CSLM + RNN + WP 31.50 34.54
Table 1: BLEU scores computed on the development and test sets using different combinations of
approaches. WP denotes a word penalty , where
we penalizes the number of unknown words to


key: chroma:document
string_value:
Table 1: BLEU scores computed on the development and test sets using different combinations of
approaches. WP denotes a word penalty , where
we penalizes the number of unknown words to
neural networks.
similarly. We used rank-100 matrices, equivalent
to learning an embedding of dimension 100 for
each word. The activation function used for ~hin
Eq. (8) is a hyperbolic tangent function. The computation from the hidden state in the decoder to
the output is implemented as a deep neural network (Pascanu et al., 2014) with a single intermediate layer having 500 maxout units each pooling
2 inputs (Goodfellow et al., 2013).
All the weight parameters in the RNN Encoder–
Decoder were initialized by sampling from an
isotropic zero-mean (white) Gaussian distribution
with its standard deviation ﬁxed to 0:01, except
for the recurrent weight parameters. For the recurrent weight matrices, we ﬁrst sampled from a
white Gaussian distribution and used its left singular vectors matrix, following (Saxe et al., 2014).
We used Adadelta and stochastic gradient
descent to train the RNN Encoder–Decoder
with hyperparameters = 106and=
0:95(Zeiler, 2012). At each update, we used 64


key: chroma:document
string_value:
with hyperparameters = 106and=
0:95(Zeiler, 2012). At each update, we used 64
randomly selected phrase pairs from a phrase table (which was created from 348M words). The
model was trained for approximately three days.
Details of the architecture used in the experiments are explained in more depth in the supplementary material.
4.1.2 Neural Language Model
In order to assess the effectiveness of scoring
phrase pairs with the proposed RNN Encoder–
Decoder, we also tried a more traditional approach
of using a neural network for learning a target
language model (CSLM) (Schwenk, 2007). Especially, the comparison between the SMT system
using CSLM and that using the proposed approach
of phrase scoring by RNN Encoder–Decoder will
clarify whether the contributions from multiple
neural networks in different parts of the SMT sys-tem add up or are redundant.
We trained the CSLM model on 7-grams
from the target corpus. Each input word
was projected into the embedding space R512,
and they were concatenated to form a 3072dimensional vector. The concatenated vector was
fed through two rectiﬁed layers (of size 1536 and
1024) (Glorot et al., 2011). The output layer was


key: chroma:document
string_value:
fed through two rectiﬁed layers (of size 1536 and
1024) (Glorot et al., 2011). The output layer was
a simple softmax layer (see Eq. (2)). All the
weight parameters were initialized uniformly between0:01and0:01, and the model was trained
until the validation perplexity did not improve for
10 epochs. After training, the language model
achieved a perplexity of 45.80. The validation set
was a random selection of 0.1% of the corpus. The
model was used to score partial translations during the decoding process, which generally leads to
higher gains in BLEU score than n-best list rescoring (Vaswani et al., 2013).
To address the computational complexity of
using a CSLM in the decoder a buffer was
used to aggregate n-grams during the stacksearch performed by the decoder. Only when
the buffer is full, or a stack is about to
be pruned, the n-grams are scored by the
CSLM. This allows us to perform fast matrixmatrix multiplication on GPU using Theano
(Bergstra et al., 2010; Bastien et al., 2012).
−60 −50 −40 −30 −20 −10 0−14−12−10−8−6−4−20


key: chroma:document
string_value:
−60 −50 −40 −30 −20 −10 0−14−12−10−8−6−4−20
RNN Scores (log)TM Scores (log)
Figure 3: The visualization of phrase pairs according to their scores (log-probabilities) by the RNN
Encoder–Decoder and the translation model.
4.2 Quantitative Analysis
We tried the following combinations:
1. Baseline conﬁguration
2. Baseline + RNN
3. Baseline + CSLM + RNN
4. Baseline + CSLM + RNN + Word penalty
Source Translation Model RNN Encoder–Decoder
at the end of the [a la ﬁn de la] [ ´r la ﬁn des ann ´ees] [ ˆetre supprim ´es`a la ﬁn de la][`a la ﬁn du] [ `a la ﬁn des] [ `a la ﬁn de la]
for the ﬁrst time [rc
pour la premi r¨ere fois] [ ´et´e donn ´es pour
la premi `ere fois] [ ´et´e comm ´emor ´ee pour la


key: chroma:document
string_value:
la premi `ere fois] [ ´et´e comm ´emor ´ee pour la
premi `ere fois][pour la premi `ere fois] [pour la premi `ere fois ,]
[pour la premi `ere fois que]
in the United States
and[?aux?tats-Unis et] [ ´et´e ouvertes aux ´EtatsUnis et] [ ´et´e constat ´ees aux ´Etats-Unis et][aux Etats-Unis et] [des Etats-Unis et] [des
´Etats-Unis et]
, as well as [?s , qu’] [ ?s , ainsi que] [ ?re aussi bien que] [, ainsi qu’] [, ainsi que] [, ainsi que les]
one of the most [?t?l’ un des plus] [ ?l’ un des plus] [ ˆetre retenue
comme un de ses plus][l’ un des] [le] [un des]
(a) Long, frequent source phrases
Source Translation Model RNN Encoder–Decoder


key: chroma:document
string_value:
(a) Long, frequent source phrases
Source Translation Model RNN Encoder–Decoder
, Minister of Communications and Transport[Secr ´etaire aux communications et aux transports :] [Secr ´etaire aux communications et aux
transports][Secr ´etaire aux communications et aux transports] [Secr ´etaire aux communications et aux
transports :]
did not comply with
the[vestimentaire , ne correspondaient pas `a des]
[susmentionn ´ee n’ ´etait pas conforme aux]
[pr´esent ´ees n’ ´etaient pas conformes `a la][n’ ont pas respect ´e les] [n’ ´etait pas conforme
aux] [n’ ont pas respect ´e la]
parts of the world . [c
gions du monde .] [r ´egions du monde consid´er´ees .] [r ´egion du monde consid ´er´ee .][parties du monde .] [les parties du monde .]
[des parties du monde .]
the past few days . [le petit texte .] [cours des tout derniers jours .]


key: chroma:document
string_value:
[des parties du monde .]
the past few days . [le petit texte .] [cours des tout derniers jours .]
[les tout derniers jours .][ces derniers jours .] [les derniers jours .] [cours
des derniers jours .]
on Friday and Saturday[vendredi et samedi `a la] [vendredi et samedi `a]
[se d ´eroulera vendredi et samedi ,][le vendredi et le samedi] [le vendredi et samedi]
[vendredi et samedi]
(b) Long, rare source phrases
Table 2: The top scoring target phrases for a small set of source phrases according to the translation
model (direct translation probability) and by the RNN Encoder–Decoder. Source phrases were randomly
selected from phrases with 4 or more words. ?denotes an incomplete (partial) character. ris a Cyrillic
letter ghe.
The results are presented in Table 1. As expected, adding features computed by neural networks consistently improves the performance over
the baseline performance.
The best performance was achieved when we
used both CSLM and the phrase scores from the
RNN Encoder–Decoder. This suggests that the


key: chroma:document
string_value:
the baseline performance.
The best performance was achieved when we
used both CSLM and the phrase scores from the
RNN Encoder–Decoder. This suggests that the
contributions of the CSLM and the RNN Encoder–
Decoder are not too correlated and that one can
expect better results by improving each method independently. Furthermore, we tried penalizing the
number of words that are unknown to the neural
networks (i.e. words which are not in the shortlist). We do so by simply adding the number of
unknown words as an additional feature the loglinear model in Eq. (9).3However, in this case we
3To understand the effect of the penalty, consider the set
of all words in the 15,000 large shortlist, SL. All words xi=2
SL are replaced by a special token [UNK]before being scored
by the neural networks. Hence, the conditional probability of
anyxi
t=2SL is actually given by the model as
p(xt= [UNK]jx<t) =p(xt=2SLjx<t)
=X
xj
t=2SLp
xj
tjx<t
p
xi
tjx<t
;


key: chroma:document
string_value:
=X
xj
t=2SLp
xj
tjx<t
p
xi
tjx<t
;
where x<tis a shorthand notation for xt1; : : : ; x 1.were not able to achieve better performance on the
test set, but only on the development set.
4.3 Qualitative Analysis
In order to understand where the performance improvement comes from, we analyze the phrase pair
scores computed by the RNN Encoder–Decoder
against the corresponding p(fje)from the translation model. Since the existing translation model
relies solely on the statistics of the phrase pairs in
the corpus, we expect its scores to be better estimated for the frequent phrases but badly estimated
for rare phrases. Also, as we mentioned earlier
in Sec. 3.1, we further expect the RNN Encoder–
Decoder which was trained without any frequency
information to score the phrase pairs based rather
on the linguistic regularities than on the statistics
of their occurrences in the corpus.
We focus on those pairs whose source phrase is
long (more than 3 words per source phrase) and
As a result, the probability of words not in the shortlist is
always overestimated. It is possible to address this issue by
backing off to an existing model that contain non-shortlisted


key: chroma:document
string_value:
always overestimated. It is possible to address this issue by
backing off to an existing model that contain non-shortlisted
words (see (Schwenk, 2007)) In this paper, however, we opt
for introducing a word penalty instead, which counteracts the
word probability overestimation.
Source Samples from RNN Encoder–Decoder
at the end of the [`a la ﬁn de la] (11)
for the ﬁrst time [pour la premi `ere fois] (24) [pour la premi `ere fois que] (2)
in the United States and [aux ´Etats-Unis et] (6) [dans les ´Etats-Unis et] (4)
, as well as [, ainsi que] [,] [ainsi que] [, ainsi qu’] [et UNK]
one of the most [l’ un des plus] (9) [l’ un des] (5) [l’ une des plus] (2)
(a) Long, frequent source phrases
Source Samples from RNN Encoder–Decoder
, Minister of Communications and Transport[ , ministre des communications et le transport] ( 13)


key: chroma:document
string_value:
Source Samples from RNN Encoder–Decoder
, Minister of Communications and Transport[ , ministre des communications et le transport] ( 13)
did not comply with the [n’ tait pas conforme aux] [n’ a pas respect l’] ( 2) [n’ a pas respect la] ( 3)
parts of the world . [arts du monde .] (11) [des arts du monde .] ( 7)
the past few days . [quelques jours .] (5) [les derniers jours .] ( 5) [ces derniers jours .] ( 2)
on Friday and Saturday [vendredi et samedi] ( 5) [le vendredi et samedi] ( 7) [le vendredi et le samedi] ( 4)
(b) Long, rare source phrases
Table 3: Samples generated from the RNN Encoder–Decoder for each source phrase used in Table 2. We
show the top-5 target phrases out of 50 samples. They are sorted by the RNN Encoder–Decoder scores.
Figure 4: 2–D embedding of the learned word representation. The left one shows the full embedding


key: chroma:document
string_value:
Figure 4: 2–D embedding of the learned word representation. The left one shows the full embedding
space, while the right one shows a zoomed-in view of one region (color–coded). For more plots, see the
supplementary material.
frequent . For each such source phrase, we look
at the target phrases that have been scored high
either by the translation probability p(fje)or
by the RNN Encoder–Decoder. Similarly, we perform the same procedure with those pairs whose
source phrase is long butrare in the corpus.
Table 2 lists the top- 3target phrases per source
phrase favored either by the translation model
or by the RNN Encoder–Decoder. The source
phrases were randomly chosen among long ones
having more than 4 or 5 words.
In most cases, the choices of the target phrases
by the RNN Encoder–Decoder are closer to actual or literal translations. We can observe that the
RNN Encoder–Decoder prefers shorter phrases in
general.
Interestingly, many phrase pairs were scored
similarly by both the translation model and the
RNN Encoder–Decoder, but there were as manyother phrase pairs that were scored radically different (see Fig. 3). This could arise from the
proposed approach of training the RNN Encoder–


key: chroma:document
string_value:
proposed approach of training the RNN Encoder–
Decoder on a set of unique phrase pairs, discouraging the RNN Encoder–Decoder from learning
simply the frequencies of the phrase pairs from the
corpus, as explained earlier.
Furthermore, in Table 3, we show for each of
the source phrases in Table 2, the generated samples from the RNN Encoder–Decoder. For each
source phrase, we generated 50 samples and show
the top-ﬁve phrases accordingly to their scores.
We can see that the RNN Encoder–Decoder is
able to propose well-formed target phrases without looking at the actual phrase table. Importantly,
the generated phrases do not overlap completely
with the target phrases from the phrase table. This
encourages us to further investigate the possibility
of replacing the whole or a part of the phrase table
Figure 5: 2–D embedding of the learned phrase representation. The top left one shows the full representation space (5000 randomly selected points), while the other three ﬁgures show the zoomed-in view of
speciﬁc regions (color–coded).
with the proposed RNN Encoder–Decoder in the
future.
4.4 Word and Phrase Representations
Since the proposed RNN Encoder–Decoder is not


key: chroma:document
string_value:
with the proposed RNN Encoder–Decoder in the
future.
4.4 Word and Phrase Representations
Since the proposed RNN Encoder–Decoder is not
speciﬁcally designed only for the task of machine
translation, here we brieﬂy look at the properties
of the trained model.
It has been known for some time that
continuous space language models using
neural networks are able to learn semantically meaningful embeddings (See, e.g.,
(Bengio et al., 2003; Mikolov et al., 2013)). Since
the proposed RNN Encoder–Decoder also projects
to and maps back from a sequence of words into
a continuous space vector, we expect to see a
similar property with the proposed model as well.
The left plot in Fig. 4 shows the 2–D embedding
of the words using the word embedding matrix
learned by the RNN Encoder–Decoder. The projection was done by the recently proposed BarnesHut-SNE (van der Maaten, 2013). We can clearly
see that semantically similar words are clusteredwith each other (see the zoomed-in plots in Fig. 4).
The proposed RNN Encoder–Decoder naturally
generates a continuous-space representation of a
phrase. The representation ( cin Fig. 1) in this


key: chroma:document
string_value:
The proposed RNN Encoder–Decoder naturally
generates a continuous-space representation of a
phrase. The representation ( cin Fig. 1) in this
case is a 1000-dimensional vector. Similarly to the
word representations, we visualize the representations of the phrases that consists of four or more
words using the Barnes-Hut-SNE in Fig. 5.
From the visualization, it is clear that the RNN
Encoder–Decoder captures both semantic and syntactic structures of the phrases. For instance, in
the bottom-left plot, most of the phrases are about
the duration of time, while those phrases that are
syntactically similar are clustered together. The
bottom-right plot shows the cluster of phrases that
are semantically similar (countries or regions). On
the other hand, the top-right plot shows the phrases
that are syntactically similar.
5 Conclusion
In this paper, we proposed a new neural network
architecture, called an RNN Encoder–Decoder
that is able to learn the mapping from a sequence
of an arbitrary length to another sequence, possibly from a different set, of an arbitrary length. The
proposed RNN Encoder–Decoder is able to either
score a pair of sequences (in terms of a conditional
probability) or generate a target sequence given a
source sequence. Along with the new architecture,


key: chroma:document
string_value:
score a pair of sequences (in terms of a conditional
probability) or generate a target sequence given a
source sequence. Along with the new architecture,
we proposed a novel hidden unit that includes a reset gate and an update gate that adaptively control
how much each hidden unit remembers or forgets
while reading/generating a sequence.
We evaluated the proposed model with the task
of statistical machine translation, where we used
the RNN Encoder–Decoder to score each phrase
pair in the phrase table. Qualitatively, we were
able to show that the new model is able to capture linguistic regularities in the phrase pairs well
and also that the RNN Encoder–Decoder is able to
propose well-formed target phrases.
The scores by the RNN Encoder–Decoder were
found to improve the overall translation performance in terms of BLEU scores. Also, we
found that the contribution by the RNN Encoder–
Decoder is rather orthogonal to the existing approach of using neural networks in the SMT system, so that we can improve further the performance by using, for instance, the RNN Encoder–
Decoder and the neural net language model together.
Our qualitative analysis of the trained model
shows that it indeed captures the linguistic regularities in multiple levels i.e. at the word level as
well as phrase level. This suggests that there may


key: chroma:document
string_value:
Our qualitative analysis of the trained model
shows that it indeed captures the linguistic regularities in multiple levels i.e. at the word level as
well as phrase level. This suggests that there may
be more natural language related applications that
may beneﬁt from the proposed RNN Encoder–
Decoder.
The proposed architecture has large potential
for further improvement and analysis. One approach that was not investigated here is to replace the whole, or a part of the phrase table by
letting the RNN Encoder–Decoder propose target
phrases. Also, noting that the proposed model is
not limited to being used with written language,
it will be an important future research to apply the
proposed architecture to other applications such as
speech transcription.
Acknowledgments
KC, BM, CG, DB and YB would like to thank
NSERC, Calcul Qu ´ebec, Compute Canada, the
Canada Research Chairs and CIFAR. FB and HS
were partially funded by the European Commis-sion under the project MateCat, and by DARPA
under the BOLT project.
References
[Auli et al.2013] Michael Auli, Michel Galley, Chris
Quirk, and Geoffrey Zweig. 2013. Joint language
and translation modeling with recurrent neural networks. In Proceedings of the ACL Conference on
Empirical Methods in Natural Language Processing


key: chroma:document
string_value:
Quirk, and Geoffrey Zweig. 2013. Joint language
and translation modeling with recurrent neural networks. In Proceedings of the ACL Conference on
Empirical Methods in Natural Language Processing
(EMNLP) , pages 1044–1054.
[Axelrod et al.2011] Amittai Axelrod, Xiaodong He,
and Jianfeng Gao. 2011. Domain adaptation via
pseudo in-domain data selection. In Proceedings of
the ACL Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 355–362.
[Bastien et al.2012] Fr ´ed´eric Bastien, Pascal Lamblin,
Razvan Pascanu, James Bergstra, Ian J. Goodfellow,
Arnaud Bergeron, Nicolas Bouchard, and Yoshua
Bengio. 2012. Theano: new features and speed improvements. Deep Learning and Unsupervised Feature Learning NIPS 2012 Workshop.
[Bengio et al.2003] Yoshua Bengio, R ´ejean Ducharme,
Pascal Vincent, and Christian Janvin. 2003. A neural probabilistic language model. J. Mach. Learn.
Res., 3:1137–1155, March.


key: chroma:document
string_value:
Pascal Vincent, and Christian Janvin. 2003. A neural probabilistic language model. J. Mach. Learn.
Res., 3:1137–1155, March.
[Bengio et al.2013] Y . Bengio, N. BoulangerLewandowski, and R. Pascanu. 2013. Advances
in optimizing recurrent networks. In Proceedings
of the 38th International Conference on Acoustics,
Speech, and Signal Processing (ICASSP 2013) ,
May.
[Bergstra et al.2010] James Bergstra, Olivier Breuleux,
Fr´ed´eric Bastien, Pascal Lamblin, Razvan Pascanu,
Guillaume Desjardins, Joseph Turian, David WardeFarley, and Yoshua Bengio. 2010. Theano: a CPU
and GPU math expression compiler. In Proceedings
of the Python for Scientiﬁc Computing Conference
(SciPy) , June. Oral Presentation.
[Chandar et al.2014] Sarath Chandar, Stanislas Lauly,
Hugo Larochelle, Mitesh Khapra, Balaraman Ravindran, Vikas Raykar, and Amrita Saha. 2014. An autoencoder approach to learning bilingual word representations. arXiv: 1402.1454 [cs.CL] , February.


key: chroma:document
string_value:
[Dahl et al.2012] George E. Dahl, Dong Yu, Li Deng,
and Alex Acero. 2012. Context-dependent pretrained deep neural networks for large vocabulary
speech recognition. IEEE Transactions on Audio,
Speech, and Language Processing , 20(1):33–42.
[Devlin et al.2014] Jacob Devlin, Rabih Zbib,
Zhongqiang Huang, Thomas Lamar, Richard
Schwartz, , and John Makhoul. 2014. Fast and
robust neural network joint models for statistical
machine translation. In Proceedings of the ACL
2014 Conference , ACL ’14, pages 1370–1380.
[Gao et al.2013] Jianfeng Gao, Xiaodong He, Wen tau
Yih, and Li Deng. 2013. Learning semantic representations for the phrase translation model. Technical report, Microsoft Research.
[Glorot et al.2011] X. Glorot, A. Bordes, and Y . Bengio. 2011. Deep sparse rectiﬁer neural networks. In
AISTATS’2011 .
[Goodfellow et al.2013] Ian J. Goodfellow, David
Warde-Farley, Mehdi Mirza, Aaron Courville, and
Yoshua Bengio. 2013. Maxout networks. In
ICML’2013 .


key: chroma:document
string_value:
Warde-Farley, Mehdi Mirza, Aaron Courville, and
Yoshua Bengio. 2013. Maxout networks. In
ICML’2013 .
[Graves2012] Alex Graves. 2012. Supervised Sequence Labelling with Recurrent Neural Networks .
Studies in Computational Intelligence. Springer.
[Hochreiter and Schmidhuber1997] S. Hochreiter and
J. Schmidhuber. 1997. Long short-term memory.
Neural Computation , 9(8):1735–1780.
[Kalchbrenner and Blunsom2013] Nal Kalchbrenner
and Phil Blunsom. 2013. Two recurrent continuous
translation models. In Proceedings of the ACL Conference on Empirical Methods in Natural Language
Processing (EMNLP) , pages 1700–1709.
[Koehn et al.2003] Philipp Koehn, Franz Josef Och,
and Daniel Marcu. 2003. Statistical phrase-based
translation. In Proceedings of the 2003 Conference
of the North American Chapter of the Association
for Computational Linguistics on Human Language
Technology - Volume 1 , NAACL ’03, pages 48–54.
[Koehn2005] P. Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In Machine


key: chroma:document
string_value:
[Koehn2005] P. Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In Machine
Translation Summit X , pages 79–86, Phuket, Thailand.
[Krizhevsky et al.2012] Alex Krizhevsky, Ilya
Sutskever, and Geoffrey Hinton. 2012. ImageNet classiﬁcation with deep convolutional neural
networks. In Advances in Neural Information
Processing Systems 25 (NIPS’2012) .
[Marcu and Wong2002] Daniel Marcu and William
Wong. 2002. A phrase-based, joint probability
model for statistical machine translation. In Proceedings of the ACL-02 Conference on Empirical
Methods in Natural Language Processing - Volume
10, EMNLP ’02, pages 133–139.
[Mikolov et al.2013] Tomas Mikolov, Ilya Sutskever,
Kai Chen, Greg Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and
their compositionality. In Advances in Neural Information Processing Systems 26 , pages 3111–3119.
[Moore and Lewis2010] Robert C. Moore and William
Lewis. 2010. Intelligent selection of language
model training data. In Proceedings of the ACL
2010 Conference Short Papers , ACLShort ’10,


key: chroma:document
string_value:
Lewis. 2010. Intelligent selection of language
model training data. In Proceedings of the ACL
2010 Conference Short Papers , ACLShort ’10,
pages 220–224, Stroudsburg, PA, USA.[Pascanu et al.2014] R. Pascanu, C. Gulcehre, K. Cho,
and Y . Bengio. 2014. How to construct deep recurrent neural networks. In Proceedings of the Second
International Conference on Learning Representations (ICLR 2014) , April.
[Saxe et al.2014] Andrew M. Saxe, James L. McClelland, and Surya Ganguli. 2014. Exact solutions
to the nonlinear dynamics of learning in deep linear neural networks. In Proceedings of the Second
International Conference on Learning Representations (ICLR 2014) , April.
[Schwenk et al.2006] Holger Schwenk, Marta R. CostaJuss`a, and Jos ´e A. R. Fonollosa. 2006. Continuous
space language models for the iwslt 2006 task. In
IWSLT , pages 166–173.
[Schwenk2007] Holger Schwenk. 2007. Continuous
space language models. Comput. Speech Lang. ,
21(3):492–518, July.
[Schwenk2012] Holger Schwenk. 2012. Continuous


key: chroma:document
string_value:
space language models. Comput. Speech Lang. ,
21(3):492–518, July.
[Schwenk2012] Holger Schwenk. 2012. Continuous
space translation models for phrase-based statistical machine translation. In Martin Kay and Christian Boitet, editors, Proceedings of the 24th International Conference on Computational Linguistics
(COLIN) , pages 1071–1080.
[Socher et al.2011] Richard Socher, Eric H. Huang, Jeffrey Pennington, Andrew Y . Ng, and Christopher D.
Manning. 2011. Dynamic pooling and unfolding
recursive autoencoders for paraphrase detection. In
Advances in Neural Information Processing Systems
24.
[Son et al.2012] Le Hai Son, Alexandre Allauzen, and
Franc ¸ois Yvon. 2012. Continuous space translation models with neural networks. In Proceedings of
the 2012 Conference of the North American Chapter of the Association for Computational Linguistics:
Human Language Technologies , NAACL HLT ’12,
pages 39–48, Stroudsburg, PA, USA.
[van der Maaten2013] Laurens van der Maaten. 2013.
Barnes-hut-sne. In Proceedings of the First International Conference on Learning Representations
(ICLR 2013) , May.


key: chroma:document
string_value:
Barnes-hut-sne. In Proceedings of the First International Conference on Learning Representations
(ICLR 2013) , May.
[Vaswani et al.2013] Ashish Vaswani, Yinggong Zhao,
Victoria Fossum, and David Chiang. 2013. Decoding with large-scale neural language models improves translation. Proceedings of the Conference
on Empirical Methods in Natural Language Processing , pages 1387–1392.
[Zeiler2012] Matthew D. Zeiler. 2012. ADADELTA:
an adaptive learning rate method. Technical report,
arXiv 1212.5701.
[Zou et al.2013] Will Y . Zou, Richard Socher,
Daniel M. Cer, and Christopher D. Manning.
2013. Bilingual word embeddings for phrase-based
machine translation. In Proceedings of the ACL
Conference on Empirical Methods in Natural
Language Processing (EMNLP) , pages 1393–1398.
A RNN Encoder–Decoder
In this document, we describe in detail the architecture of the RNN Encoder–Decoder used in the experiments.
Let us denote an source phrase by X= (x1;x2;:::;xN)and a target phrase by Y=


key: chroma:document
string_value:
Let us denote an source phrase by X= (x1;x2;:::;xN)and a target phrase by Y=
(y1;y2;:::;yM). Each phrase is a sequence of K-dimensional one-hot vectors, such that only one
element of the vector is 1and all the others are 0. The index of the active ( 1) element indicates the word
represented by the vector.
A.1 Encoder
Each word of the source phrase is embedded in a 500-dimensional vector space: e(xi)2R500.e(x)is
used in Sec. 4.4 to visualize the words.
The hidden state of an encoder consists of 1000 hidden units, and each one of them at time tis
computed by
hhti
j=zjhht1i
j + (1zj)~hhti
j;
where
~hhti
j= tanh
[We(xt)]j+
U
rhht1i
j
;
zj=
[Wze(xt)]j+
Uzhht1i
j
;
rj=
[Wre(xt)]j+
Urhht1i
j
:


key: chroma:document
string_value:
j
;
rj=
[Wre(xt)]j+
Urhht1i
j
:
andare a logistic sigmoid function and an element-wise multiplication, respectively. To make the
equations uncluttered, we omit biases. The initial hidden state hh0i
jis ﬁxed to 0.
Once the hidden state at the Nstep (the end of the source phrase) is computed, the representation of
the source phrase cis
c= tanh
VhhNi
:
A.1.1 Decoder
The decoder starts by initializing the hidden state with
h0h0i= tanh
V0c
;
where we will use0to distinguish parameters of the decoder from those of the encoder.
The hidden state at time tof the decoder is computed by
h0hti
j=z0jh0ht1i
j + (1z0j)~h0hti
j;
where
~h0hti
j= tanh
W0e(yt1)
j+r0j
U0h0
ht1i+Cc
;
z0j=


key: chroma:document
string_value:
j+r0j
U0h0
ht1i+Cc
;
z0j=
W0ze(yt1)
j+
U0zh0
ht1i
j+ [Czc]j
;
r0j=
W0re(yt1)
j+
U0rh0
ht1i
j+ [Crc]j
;
ande(y0)is an all-zero vector. Similarly to the case of the encoder, e(y)is an embedding of a target
word.
Unlike the encoder which simply encodes the source phrase, the decoder is learned to generate a target
phrase. At each time t, the decoder computes the probability of generating j-th word by
p(yt;j= 1jyt1;:::;y1;X) =exp
gjshti
PK
j0=1exp
gj0shti;
where thei-element of shtiis
shti
i= maxn
s0hti
2i1;s0hti
2io
and
s0hti=Ohh0hti+Oyyt1+Occ:


key: chroma:document
string_value:
s0hti
2i1;s0hti
2io
and
s0hti=Ohh0hti+Oyyt1+Occ:
In short, the shti
iis a so-called maxout unit.
For the computational efﬁciency, instead of a single-matrix output weight G, we use a product of two
matrices such that
G=GlGr;
where Gl2RK500andGr2R5001000.
B Word and Phrase Representations
Here, we show enlarged plots of the word and phrase representations in Figs. 4–5.
Figure 6: 2–D embedding of the learned word representation. The top left one shows the full embedding space, while the other three ﬁgures show the zoomed-in
view of speciﬁc regions (color–coded).
Figure 7: 2–D embedding of the learned phrase representation. The top left one shows the full representation space (1000 randomly selected points), while the
other three ﬁgures show the zoomed-in view of speciﬁc regions (color–coded).


key: chroma:document
string_value:
1
Spatial Pyramid Pooling in Deep Convolutional
Networks for Visual Recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun
Abstract —Existing deep convolutional neural networks (CNNs) require a ﬁxed-size ( e.g., 224224) input image. This requirement is “artiﬁcial” and may reduce the recognition accuracy for the images or sub-images of an arbitrary size/scale. In this
work, we equip the networks with another pooling strategy, “spatial pyramid pooling”, to eliminate the above requirement. The
new network structure, called SPP-net, can generate a ﬁxed-length representation regardless of image size/scale. Pyramid
pooling is also robust to object deformations. With these advantages, SPP-net should in general improve all CNN-based image
classiﬁcation methods. On the ImageNet 2012 dataset, we demonstrate that SPP-net boosts the accuracy of a variety of CNN
architectures despite their different designs. On the Pascal VOC 2007 and Caltech101 datasets, SPP-net achieves state-of-theart classiﬁcation results using a single full-image representation and no ﬁne-tuning.


key: chroma:document
string_value:
The power of SPP-net is also signiﬁcant in object detection. Using SPP-net, we compute the feature maps from the entire
image only once, and then pool features in arbitrary regions (sub-images) to generate ﬁxed-length representations for training
the detectors. This method avoids repeatedly computing the convolutional features. In processing test images, our method is
24-102faster than the R-CNN method, while achieving better or comparable accuracy on Pascal VOC 2007.
In ImageNet Large Scale Visual Recognition Challenge (ILSVRC) 2014, our methods rank #2 in object detection and #3 in
image classiﬁcation among all 38 teams. This manuscript also introduces the improvement made for this competition.
Index Terms —Convolutional Neural Networks, Spatial Pyramid Pooling, Image Classiﬁcation, Object Detection
F
1 I NTRODUCTION
We are witnessing a rapid, revolutionary change in
our vision community, mainly caused by deep convolutional neural networks (CNNs) [1] and the availability of large scale training data [2]. Deep-networksbased approaches have recently been substantially
improving upon the state of the art in image classiﬁcation [3], [4], [5], [6], object detection [7], [8], [5],


key: chroma:document
string_value:
improving upon the state of the art in image classiﬁcation [3], [4], [5], [6], object detection [7], [8], [5],
many other recognition tasks [9], [10], [11], [12], and
even non-recognition tasks.
However, there is a technical issue in the training
and testing of the CNNs: the prevalent CNNs require
aﬁxed input image size ( e.g., 224224), which limits
both the aspect ratio and the scale of the input image.
When applied to images of arbitrary sizes, current
methods mostly ﬁt the input image to the ﬁxed size,
either via cropping [3], [4] or via warping [13], [7],
as shown in Figure 1 (top). But the cropped region
may not contain the entire object, while the warped
content may result in unwanted geometric distortion.
Recognition accuracy can be compromised due to the
content loss or distortion. Besides, a pre-deﬁned scale
K. He and J. Sun are with Microsoft Research, Beijing, China. E-mail:
fkahe,jiansung@microsoft.com
X. Zhang is with Xi’an Jiaotong University, Xi’an, China. Email:


key: chroma:document
string_value:
fkahe,jiansung@microsoft.com
X. Zhang is with Xi’an Jiaotong University, Xi’an, China. Email:
xyz.clx@stu.xjtu.edu.cn
S. Ren is with University of Science and Technology of China, Hefei,
China. Email: sqren@mail.ustc.edu.cn
This work was done when X. Zhang and S. Ren were interns at Microsoft
Research.
crop warp
spatial pyramid poolingcrop / warp
conv layers image fc layers outputimage conv layers fc layers outputFigure 1: Top: cropping or warping to ﬁt a ﬁxed
size. Middle: a conventional CNN. Bottom: our spatial
pyramid pooling network structure.
may not be suitable when object scales vary. Fixing
input sizes overlooks the issues involving scales.
So why do CNNs require a ﬁxed input size? A CNN
mainly consists of two parts: convolutional layers,
and fully-connected layers that follow. The convolutional layers operate in a sliding-window manner
and output feature maps which represent the spatial
arrangement of the activations (Figure 2). In fact, convolutional layers do not require a ﬁxed image size and


key: chroma:document
string_value:
and output feature maps which represent the spatial
arrangement of the activations (Figure 2). In fact, convolutional layers do not require a ﬁxed image size and
can generate feature maps of any sizes. On the other
hand, the fully-connected layers need to have ﬁxedsize/length input by their deﬁnition. Hence, the ﬁxedsize constraint comes only from the fully-connected
layers, which exist at a deeper stage of the network.
In this paper, we introduce a spatial pyramid pooling (SPP) [14], [15] layer to remove the ﬁxed-size
constraint of the network. Speciﬁcally, we add anarXiv:1406.4729v4  [cs.CV]  23 Apr 2015
2
SPP layer on top of the last convolutional layer. The
SPP layer pools the features and generates ﬁxedlength outputs, which are then fed into the fullyconnected layers (or other classiﬁers). In other words,
we perform some information “aggregation” at a
deeper stage of the network hierarchy (between convolutional layers and fully-connected layers) to avoid
the need for cropping or warping at the beginning.


key: chroma:document
string_value:
deeper stage of the network hierarchy (between convolutional layers and fully-connected layers) to avoid
the need for cropping or warping at the beginning.
Figure 1 (bottom) shows the change of the network
architecture by introducing the SPP layer. We call the
new network structure SPP-net .
Spatial pyramid pooling [14], [15] (popularly
known as spatial pyramid matching or SPM [15]), as
an extension of the Bag-of-Words (BoW) model [16],
is one of the most successful methods in computer
vision. It partitions the image into divisions from
ﬁner to coarser levels, and aggregates local features
in them. SPP has long been a key component in the
leading and competition-winning systems for classiﬁcation ( e.g., [17], [18], [19]) and detection ( e.g., [20])
before the recent prevalence of CNNs. Nevertheless,
SPP has not been considered in the context of CNNs.
We note that SPP has several remarkable properties
for deep CNNs: 1) SPP is able to generate a ﬁxedlength output regardless of the input size, while the
sliding window pooling used in the previous deep
networks [3] cannot; 2) SPP uses multi-level spatial


key: chroma:document
string_value:
sliding window pooling used in the previous deep
networks [3] cannot; 2) SPP uses multi-level spatial
bins, while the sliding window pooling uses only
a single window size. Multi-level pooling has been
shown to be robust to object deformations [15]; 3) SPP
can pool features extracted at variable scales thanks
to the ﬂexibility of input scales. Through experiments
we show that all these factors elevate the recognition
accuracy of deep networks.
SPP-net not only makes it possible to generate representations from arbitrarily sized images/windows
for testing, but also allows us to feed images with
varying sizes or scales during training. Training with
variable-size images increases scale-invariance and
reduces over-ﬁtting. We develop a simple multi-size
training method. For a single network to accept
variable input sizes, we approximate it by multiple
networks that share all parameters, while each of
these networks is trained using a ﬁxed input size. In
each epoch we train the network with a given input
size, and switch to another input size for the next
epoch. Experiments show that this multi-size training
converges just as the traditional single-size training,
and leads to better testing accuracy.


key: chroma:document
string_value:
epoch. Experiments show that this multi-size training
converges just as the traditional single-size training,
and leads to better testing accuracy.
The advantages of SPP are orthogonal to the speciﬁc
CNN designs. In a series of controlled experiments on
the ImageNet 2012 dataset, we demonstrate that SPP
improves four different CNN architectures in existing
publications [3], [4], [5] (or their modiﬁcations), over
the no-SPP counterparts. These architectures have
various ﬁlter numbers/sizes, strides, depths, or other
designs. It is thus reasonable for us to conjecture
that SPP should improve more sophisticated (deeperand larger) convolutional architectures. SPP-net also
shows state-of-the-art classiﬁcation results on Caltech101 [21] and Pascal VOC 2007 [22] using only a
single full-image representation and no ﬁne-tuning.
SPP-net also shows great strength in object detection. In the leading object detection method R-CNN
[7], the features from candidate windows are extracted
via deep convolutional networks. This method shows
remarkable detection accuracy on both the VOC and
ImageNet datasets. But the feature computation in RCNN is time-consuming, because it repeatedly applies


key: chroma:document
string_value:
remarkable detection accuracy on both the VOC and
ImageNet datasets. But the feature computation in RCNN is time-consuming, because it repeatedly applies
the deep convolutional networks to the raw pixels
of thousands of warped regions per image. In this
paper, we show that we can run the convolutional
layers only once on the entire image (regardless of
the number of windows), and then extract features
by SPP-net on the feature maps. This method yields
a speedup of over one hundred times over R-CNN.
Note that training/running a detector on the feature
maps (rather than image regions) is actually a more
popular idea [23], [24], [20], [5]. But SPP-net inherits
the power of the deep CNN feature maps and also the
ﬂexibility of SPP on arbitrary window sizes, which
leads to outstanding accuracy and efﬁciency. In our
experiment, the SPP-net-based system (built upon the
R-CNN pipeline) computes features 24-102 faster
than R-CNN, while has better or comparable accuracy.
With the recent fast proposal method of EdgeBoxes
[25], our system takes 0.5 seconds processing an image
(including all steps ). This makes our method practical
for real-world applications.


key: chroma:document
string_value:
[25], our system takes 0.5 seconds processing an image
(including all steps ). This makes our method practical
for real-world applications.
A preliminary version of this manuscript has been
published in ECCV 2014. Based on this work, we
attended the competition of ILSVRC 2014 [26], and
ranked #2 in object detection and #3 in image classiﬁcation (both are provided-data-only tracks) among
all 38 teams. There are a few modiﬁcations made
for ILSVRC 2014. We show that the SPP-nets can
boost various networks that are deeper and larger
(Sec. 3.1.2-3.1.4) over the no-SPP counterparts. Further, driven by our detection framework, we ﬁnd
that multi-view testing on feature maps with ﬂexibly
located/sized windows (Sec. 3.1.5) can increase the
classiﬁcation accuracy. This manuscript also provides
the details of these modiﬁcations.
We have released the code to facilitate future research ( http://research.microsoft.com/en-us/um/people/kahe/ ).
2 D EEP NETWORKS WITH SPATIAL PYRAMIDPOOLING
2.1 Convolutional Layers and Feature Maps


key: chroma:document
string_value:
2 D EEP NETWORKS WITH SPATIAL PYRAMIDPOOLING
2.1 Convolutional Layers and Feature Maps
Consider the popular seven-layer architectures [3], [4].
The ﬁrst ﬁve layers are convolutional, some of which
are followed by pooling layers. These pooling layers
can also be considered as “convolutional”, in the sense
that they are using sliding windows. The last two
3
  
  
filter #175
filter #55
(a) image (b) feature maps (c) strongest activations
  
  
filter #66
filter #118
(a) image (b) feature maps (c) strongest activations
Figure 2: Visualization of the feature maps. (a) Two images in Pascal VOC 2007. (b) The feature maps of some
conv 5ﬁlters. The arrows indicate the strongest responses and their corresponding positions in the images.
(c) The ImageNet images that have the strongest responses of the corresponding ﬁlters. The green rectangles
mark the receptive ﬁelds of the strongest responses.
layers are fully connected, with an N-way softmax as
the output, where N is the number of categories.
The deep network described above needs a ﬁxed


key: chroma:document
string_value:
layers are fully connected, with an N-way softmax as
the output, where N is the number of categories.
The deep network described above needs a ﬁxed
image size. However, we notice that the requirement
of ﬁxed sizes is only due to the fully-connected layers
that demand ﬁxed-length vectors as inputs. On the
other hand, the convolutional layers accept inputs of
arbitrary sizes. The convolutional layers use sliding
ﬁlters, and their outputs have roughly the same aspect
ratio as the inputs. These outputs are known as feature
maps [1] - they involve not only the strength of the
responses, but also their spatial positions.
In Figure 2, we visualize some feature maps. They
are generated by some ﬁlters of the conv 5layer. Figure 2(c) shows the strongest activated images of these
ﬁlters in the ImageNet dataset. We see a ﬁlter can be
activated by some semantic content. For example, the
55-th ﬁlter (Figure 2, bottom left) is most activated by
a circle shape; the 66-th ﬁlter (Figure 2, top right) is


key: chroma:document
string_value:
a circle shape; the 66-th ﬁlter (Figure 2, top right) is
most activated by a ^-shape; and the 118-th ﬁlter (Figure 2, bottom right) is most activated by a _-shape.
These shapes in the input images (Figure 2(a)) activate
the feature maps at the corresponding positions (the
arrows in Figure 2).
It is worth noticing that we generate the feature
maps in Figure 2 without ﬁxing the input size. These
feature maps generated by deep convolutional layers are analogous to the feature maps in traditional
methods [27], [28]. In those methods, SIFT vectors
[29] or image patches [28] are densely extracted and
then encoded, e.g., by vector quantization [16], [15],
[30], sparse coding [17], [18], or Fisher kernels [19].
These encoded features consist of the feature maps,
and are then pooled by Bag-of-Words (BoW) [16] or
spatial pyramids [14], [15]. Analogously, the deep
convolutional features can be pooled in a similar way.
2.2 The Spatial Pyramid Pooling Layer
The convolutional layers accept arbitrary input sizes,


key: chroma:document
string_value:
convolutional features can be pooled in a similar way.
2.2 The Spatial Pyramid Pooling Layer
The convolutional layers accept arbitrary input sizes,
but they produce outputs of variable sizes. The classiﬁers (SVM/softmax) or fully-connected layers require
convolutional layersfeature maps of conv 5
(arbitrary size)
fixed-length representation
input image16×256-d 4×256-d 256-d…...
…...
spatial pyramid pooling layerfully-connected layers (fc 6, fc 7)Figure 3: A network structure with a spatial pyramid
pooling layer . Here 256 is the ﬁlter number of the
conv 5layer, and conv 5is the last convolutional layer.
ﬁxed-length vectors. Such vectors can be generated
by the Bag-of-Words (BoW) approach [16] that pools
the features together. Spatial pyramid pooling [14],
[15] improves BoW in that it can maintain spatial
information by pooling in local spatial bins. These
spatial bins have sizes proportional to the image size,
so the number of bins is ﬁxed regardless of the image
size. This is in contrast to the sliding window pooling
of the previous deep networks [3], where the number


key: chroma:document
string_value:
so the number of bins is ﬁxed regardless of the image
size. This is in contrast to the sliding window pooling
of the previous deep networks [3], where the number
of sliding windows depends on the input size.
To adopt the deep network for images of arbitrary sizes, we replace the last pooling layer ( e.g.,
pool 5, after the last convolutional layer) with a spatial
pyramid pooling layer . Figure 3 illustrates our method.
In each spatial bin, we pool the responses of each
ﬁlter (throughout this paper we use max pooling).
The outputs of the spatial pyramid pooling are kMdimensional vectors with the number of bins denoted
asM(kis the number of ﬁlters in the last convolutional layer). The ﬁxed-dimensional vectors are the
input to the fully-connected layer.
With spatial pyramid pooling, the input image can
4
be of any sizes. This not only allows arbitrary aspect
ratios, but also allows arbitrary scales. We can resize
the input image to any scale ( e.g.,min(w;h)=180, 224,
...) and apply the same deep network. When the
input image is at different scales, the network (with
the same ﬁlter sizes) will extract features at different


key: chroma:document
string_value:
...) and apply the same deep network. When the
input image is at different scales, the network (with
the same ﬁlter sizes) will extract features at different
scales. The scales play important roles in traditional
methods, e.g., the SIFT vectors are often extracted at
multiple scales [29], [27] (determined by the sizes of
the patches and Gaussian ﬁlters). We will show that
the scales are also important for the accuracy of deep
networks.
Interestingly, the coarsest pyramid level has a single
bin that covers the entire image. This is in fact a
“global pooling” operation, which is also investigated
in several concurrent works. In [31], [32] a global
average pooling is used to reduce the model size
and also reduce overﬁtting; in [33], a global average
pooling is used on the testing stage after all fc layers
to improve accuracy; in [34], a global max pooling is
used for weakly supervised object recognition. The
global pooling operation corresponds to the traditional Bag-of-Words method.
2.3 Training the Network
Theoretically, the above network structure can be
trained with standard back-propagation [1], regardless of the input image size. But in practice the GPU


key: chroma:document
string_value:
Theoretically, the above network structure can be
trained with standard back-propagation [1], regardless of the input image size. But in practice the GPU
implementations (such as cuda-convnet [3] and Caffe
[35]) are preferably run on ﬁxed input images. Next
we describe our training solution that takes advantage
of these GPU implementations while still preserving
the spatial pyramid pooling behaviors.
Single-size training
As in previous works, we ﬁrst consider a network taking a ﬁxed-size input (224 224) cropped from images.
The cropping is for the purpose of data augmentation.
For an image with a given size, we can pre-compute
the bin sizes needed for spatial pyramid pooling.
Consider the feature maps after conv 5that have a size
ofaa(e.g., 1313). With a pyramid level of nn
bins, we implement this pooling level as a sliding
window pooling, where the window size win=da=ne
and stride str=ba=ncwithdeandbcdenoting
ceiling and ﬂoor operations. With an l-level pyramid,
we implement lsuch layers. The next fully-connected
layer (fc 6) will concatenate the loutputs. Figure 4


key: chroma:document
string_value:
we implement lsuch layers. The next fully-connected
layer (fc 6) will concatenate the loutputs. Figure 4
shows an example conﬁguration of 3-level pyramid
pooling (33, 22, 11) in the cuda-convnet style [3].
The main purpose of our single-size training is to
enable the multi-level pooling behavior. Experiments
show that this is one reason for the gain of accuracy.
Multi-size training
Our network with SPP is expected to be applied on
images of any sizes. To address the issue of varying
[fc6]
type=fcoutputs=4096inputs=pool3x3,pool2x2,pool1x1[pool1x1]
type=poolpool=maxinputs=conv5sizeX=13stride=13[pool3x3]
type=poolpool=maxinputs=conv5sizeX=5stride=4[pool2x2]
type=poolpool=maxinputs=conv5sizeX=7stride=6Figure 4: An example 3-level pyramid pooling in the
cuda-convnet style [3]. Here sizeX is the size of the
pooling window. This conﬁguration is for a network


key: chroma:document
string_value:
cuda-convnet style [3]. Here sizeX is the size of the
pooling window. This conﬁguration is for a network
whose feature map size of conv 5is 1313, so the
pool 33, pool 22, and pool 11layers will have 3 3,
22, and 11 bins respectively.
image sizes in training, we consider a set of predeﬁned sizes. We consider two sizes: 180 180 in addition to 224224. Rather than crop a smaller 180 180
region, we resize the aforementioned 224 224 region
to 180180. So the regions at both scales differ only
in resolution but not in content/layout. For the network to accept 180 180 inputs, we implement another
ﬁxed-size-input (180 180) network. The feature map
size after conv 5isaa= 1010 in this case. Then we
still usewin=da=neandstr=ba=ncto implement
each pyramid pooling level. The output of the spatial
pyramid pooling layer of this 180-network has the
same ﬁxed length as the 224-network. As such, this
180-network has exactly the same parameters as the


key: chroma:document
string_value:
same ﬁxed length as the 224-network. As such, this
180-network has exactly the same parameters as the
224-network in each layer. In other words, during
training we implement the varying-input-size SPP-net
by two ﬁxed-size networks that share parameters.
To reduce the overhead to switch from one network
(e.g., 224) to the other ( e.g., 180), we train each full
epoch on one network, and then switch to the other
one (keeping all weights) for the next full epoch. This
is iterated. In experiments, we ﬁnd the convergence
rate of this multi-size training to be similar to the
above single-size training.
The main purpose of our multi-size training is to
simulate the varying input sizes while still leveraging
the existing well-optimized ﬁxed-size implementations. Besides the above two-scale implementation, we
have also tested a variant using ssas input where
sis randomly and uniformly sampled from [180;224]
at each epoch. We report the results of both variants
in the experiment section.
Note that the above single/multi-size solutions are
for training only. At the testing stage, it is straightforward to apply SPP-net on images of any sizes.
5


key: chroma:document
string_value:
Note that the above single/multi-size solutions are
for training only. At the testing stage, it is straightforward to apply SPP-net on images of any sizes.
5
model conv 1 conv 2 conv 3 conv 4 conv 5 conv 6 conv 7
ZF-5 9672, str 2 25652, str 2 384323843225632
LRN, pool 32, str 2 LRN, pool 32, str 2 - map size 5555 2727 1313 1313 1313
Convnet*-5 96112, str 4 25652384323843225632
LRN, LRN, pool 32, str 2 pool 32, 2 - map size 5555 2727 1313 1313 1313
Overfeat-5/7 9672, str 2 256525123251232512325123251232
pool 32, str 3, LRN pool 22, str 2
map size 3636 1818 1818 1818 1818 1818 1818
Table 1: Network architectures: ﬁlter number ﬁlter size ( e.g.,9672), ﬁlter stride ( e.g., str 2), pooling window


key: chroma:document
string_value:
size ( e.g., pool 32), and the output feature map size ( e.g., map size 5555). LRN represents Local Response
Normalization. The padding is adjusted to produce the expected output feature map size.
3 SPP- NET FOR IMAGE CLASSIFICATION
3.1 Experiments on ImageNet 2012 Classiﬁcation
We train the networks on the 1000-category training
set of ImageNet 2012. Our training algorithm follows
the practices of previous work [3], [4], [36]. The images are resized so that the smaller dimension is 256,
and a 224224 crop is picked from the center or the
four corners from the entire image1. The data are augmented by horizontal ﬂipping and color altering [3].
Dropout [3] is used on the two fully-connected layers.
The learning rate starts from 0.01, and is divided by 10
(twice) when the error plateaus. Our implementation
is based on the publicly available code of cuda-convnet
[3] and Caffe [35]. All networks in this paper can be
trained on a single GeForce GTX Titan GPU (6 GB
memory) within two to four weeks.
3.1.1 Baseline Network Architectures
The advantages of SPP are independent of the convolutional network architectures used. We investigate


key: chroma:document
string_value:
memory) within two to four weeks.
3.1.1 Baseline Network Architectures
The advantages of SPP are independent of the convolutional network architectures used. We investigate
four different network architectures in existing publications [3], [4], [5] (or their modiﬁcations), and we
show SPP improves the accuracy of all these architectures. These baseline architectures are in Table 1 and
brieﬂy introduced below:
ZF-5 : this architecture is based on Zeiler and Fergus’s (ZF) “fast” (smaller) model [4]. The number
indicates ﬁve convolutional layers.
Convnet*-5 : this is a modiﬁcation on Krizhevsky
et al.’s network [3]. We put the two pooling layers
after conv 2and conv 3(instead of after conv 1and
conv 2). As a result, the feature maps after each
layer have the same size as ZF-5.
Overfeat-5/7 : this architecture is based on the
Overfeat paper [5], with some modiﬁcations as in
[6]. In contrast to ZF-5/Convnet*-5, this architecture produces a larger feature map ( 1818instead


key: chroma:document
string_value:
[6]. In contrast to ZF-5/Convnet*-5, this architecture produces a larger feature map ( 1818instead
of1313) before the last pooling layer. A larger
ﬁlter number (512) is used in conv 3and the following convolutional layers. We also investigate
1. In [3], the four corners are picked from the corners of the
central 256256 crop.a deeper architecture with 7 convolutional layers,
where conv 3to conv 7have the same structures.
In the baseline models, the pooling layer after the last
convolutional layer generates 66feature maps, with
two 4096-d fc layers and a 1000-way softmax layer
following. Our replications of these baseline networks
are in Table 2 (a). We train 70 epochs for ZF-5 and
90 epochs for the others. Our replication of ZF-5 is
better than the one reported in [4]. This gain is because
the corner crops are from the entire image, as is also
reported in [36].
3.1.2 Multi-level Pooling Improves Accuracy
In Table 2 (b) we show the results using singlesize training. The training and testing sizes are both
224224. In these networks, the convolutional layers


key: chroma:document
string_value:
In Table 2 (b) we show the results using singlesize training. The training and testing sizes are both
224224. In these networks, the convolutional layers
have the same structures as the corresponding baseline models, whereas the pooling layer after the ﬁnal
convolutional layer is replaced with the SPP layer. For
the results in Table 2, we use a 4-level pyramid. The
pyramid isf66, 33, 22, 11g(totally 50 bins).
For fair comparison, we still use the standard 10view prediction with each view a 224 224 crop. Our
results in Table 2 (b) show considerable improvement
over the no-SPP baselines in Table 2 (a). Interestingly,
the largest gain of top-1 error (1.65%) is given by the
most accurate architecture. Since we are still using the
same 10 cropped views as in (a), these gains are solely
because of multi-level pooling.
It is worth noticing that the gain of multi-level
pooling is notsimply due to more parameters; rather,
it is because the multi-level pooling is robust to the
variance in object deformations and spatial layout
[15]. To show this, we train another ZF-5 network with


key: chroma:document
string_value:
it is because the multi-level pooling is robust to the
variance in object deformations and spatial layout
[15]. To show this, we train another ZF-5 network with
a different 4-level pyramid: f44, 33, 22, 11g
(totally 30 bins). This network has fewer parameters
than its no-SPP counterpart, because its fc 6layer has
30256-d inputs instead of 36 256-d. The top-1/top5 errors of this network are 35.06/14.04. This result
is similar to the 50-bin pyramid above (34.98/14.14),
but considerably better than the no-SPP counterpart
(35.99/14.76).
6
top-1 error (%)
ZF-5 Convnet*-5 Overfeat-5 Overfeat-7
(a) no SPP 35.99 34.93 34.13 32.01
(b) SPP single-size trained 34.98 (1.01) 34.38 (0.55) 32.87 (1.26) 30.36 (1.65)
(c) SPP multi-size trained 34.60 (1.39) 33.94 (0.99) 32.26 (1.87) 29.68 (2.33)


key: chroma:document
string_value:
top-5 error (%)
ZF-5 Convnet*-5 Overfeat-5 Overfeat-7
(a) no SPP 14.76 13.92 13.52 11.97
(b) SPP single-size trained 14.14 (0.62) 13.54 (0.38) 12.80 (0.72) 11.12 (0.85)
(c) SPP multi-size trained 13.64 (1.12) 13.33 (0.59) 12.33 (1.19) 10.95 (1.02)
Table 2: Error rates in the validation set of ImageNet 2012. All the results are obtained using standard 10-view
testing. In the brackets are the gains over the “no SPP” baselines.
SPP on test view top-1 val
ZF-5, single-size trained 1 crop 38.01
ZF-5, single-size trained 1 full 37.55
ZF-5, multi-size trained 1 crop 37.57
ZF-5, multi-size trained 1 full 37.07
Overfeat-7, single-size trained 1 crop 33.18
Overfeat-7, single-size trained 1 full 32.72
Overfeat-7, multi-size trained 1 crop 32.57


key: chroma:document
string_value:
Overfeat-7, single-size trained 1 full 32.72
Overfeat-7, multi-size trained 1 crop 32.57
Overfeat-7, multi-size trained 1 full 31.25
Table 3: Error rates in the validation set of ImageNet
2012 using a single view. The images are resized so
min(w;h) = 256 . The crop view is the central 224 224
of the image.
3.1.3 Multi-size Training Improves Accuracy
Table 2 (c) shows our results using multi-size training.
The training sizes are 224 and 180, while the testing
size is still 224. We still use the standard 10-view
prediction. The top-1/top-5 errors of all architectures
further drop. The top-1 error of SPP-net (Overfeat-7)
drops to 29.68%, which is 2.33% better than its noSPP counterpart and 0.68% better than its single-size
trained counterpart.
Besides using the two discrete sizes of 180 and
224, we have also evaluated using a random size
uniformly sampled from [180;224]. The top-1/5 error
of SPP-net (Overfeat-7) is 30.06%/10.96%. The top1 error is slightly worse than the two-size version,


key: chroma:document
string_value:
of SPP-net (Overfeat-7) is 30.06%/10.96%. The top1 error is slightly worse than the two-size version,
possibly because the size of 224 (which is used for
testing) is visited less. But the results are still better
the single-size version.
There are previous CNN solutions [5], [36] that deal
with various scales/sizes, but they are mostly based
on testing. In Overfeat [5] and Howard’s method [36],
the single network is applied at multiple scales in the
testing stage, and the scores are averaged. Howard
further trains two different networks on low/highresolution image regions and averages the scores. To
our knowledge, our method is the ﬁrst one that trains
a single network with input images of multiple sizes.3.1.4 Full-image Representations Improve Accuracy
Next we investigate the accuracy of the full-image
views. We resize the image so that min(w;h)=256
while maintaining its aspect ratio. The SPP-net is
applied on this full image to compute the scores of
the full view. For fair comparison, we also evaluate
the accuracy of the single view in the center 224 224
crop (which is used in the above evaluations). The
comparisons of single-view testing accuracy are in


key: chroma:document
string_value:
the accuracy of the single view in the center 224 224
crop (which is used in the above evaluations). The
comparisons of single-view testing accuracy are in
Table 3. Here we evaluate ZF-5/Overfeat-7. The top-1
error rates are all reduced by the full-view representation. This shows the importance of maintaining the
complete content. Even though our network is trained
using square images only, it generalizes well to other
aspect ratios.
Comparing Table 2 and Table 3, we ﬁnd that the
combination of multiple views is substantially better
than the single full-image view. However, the fullimage representations are still of good merits. First,
we empirically ﬁnd that (discussed in the next subsection) even for the combination of dozens of views,
the additional two full-image views (with ﬂipping)
can still boost the accuracy by about 0.2%. Second,
the full-image view is methodologically consistent
with the traditional methods [15], [17], [19] where the
encoded SIFT vectors of the entire image are pooled
together. Third, in other applications such as image
retrieval [37], an image representation, rather than a
classiﬁcation score, is required for similarity ranking.
A full-image representation can be preferred.


key: chroma:document
string_value:
retrieval [37], an image representation, rather than a
classiﬁcation score, is required for similarity ranking.
A full-image representation can be preferred.
3.1.5 Multi-view Testing on Feature Maps
Inspired by our detection algorithm (described in
the next section), we further propose a multi-view
testing method on the feature maps. Thanks to the
ﬂexibility of SPP , we can easily extract the features
from windows (views) of arbitrary sizes from the
convolutional feature maps.
On the testing stage, we resize an image so
min(w;h) =swheresrepresents a predeﬁned scale
(like 256). Then we compute the convolutional feature maps from the entire image. For the usage of
7
method test scales test views top-1 val top-5 val top-5 test
Krizhevsky et al. [3] 1 10 40.7 18.2
Overfeat (fast) [5] 1 - 39.01 16.97
Overfeat (fast) [5] 6 - 38.12 16.27
Overfeat (big) [5] 4 - 35.74 14.18
Howard (base) [36] 3 162 37.0 15.8
Howard (high-res) [36] 3 162 36.8 16.2


key: chroma:document
string_value:
Howard (base) [36] 3 162 37.0 15.8
Howard (high-res) [36] 3 162 36.8 16.2
Zeiler & Fergus (ZF) (fast) [4] 1 10 38.4 16.5
Zeiler & Fergus (ZF) (big) [4] 1 10 37.5 16.0
Chatﬁeld et al. [6] 1 10 - 13.1
ours (SPP O-7) 1 10 29.68 10.95
ours (SPP O-7) 6 96+2full 27.86 9.14 9.08
Table 4: Error rates in ImageNet 2012. All the results are based on a single network . The number of views in
Overfeat depends on the scales and strides, for which there are several hundreds at the ﬁnest scale.
ﬂipped views, we also compute the feature maps of
the ﬂipped image. Given any view (window) in the
image, we map this window to the feature maps (the
way of mapping is in Appendix), and then use SPP
to pool the features from this window (see Figure 5).
The pooled features are then fed into the fc layers
to compute the softmax score of this window. These
scores are averaged for the ﬁnal prediction. For the


key: chroma:document
string_value:
The pooled features are then fed into the fc layers
to compute the softmax score of this window. These
scores are averaged for the ﬁnal prediction. For the
standard 10-view, we use s= 256 and the views
are 224224 windows on the corners or center. Experiments show that the top-5 error of the 10-view
prediction on feature maps is within 0.1% around the
original 10-view prediction on image crops.
We further apply this method to extract multiple
views from multiple scales. We resize the image to six
scaless2 f224;256;300;360;448;560gand compute
the feature maps on the entire image for each scale.
We use 224224 as the view size for any scale,
so these views have different relative sizes on the
original image for different scales. We use 18 views
for each scale: one at the center, four at the corners,
and four on the middle of each side, with/without
ﬂipping (when s= 224 there are 6 different views).
The combination of these 96 views reduces the top-5
error from 10.95% to 9.36%. Combining the two fullimage views (with ﬂipping) further reduces the top-5
error to 9.14%.


key: chroma:document
string_value:
error from 10.95% to 9.36%. Combining the two fullimage views (with ﬂipping) further reduces the top-5
error to 9.14%.
In the Overfeat paper [5], the views are also extracted from the convolutional feature maps instead
of image crops. However, their views cannot have arbitrary sizes; rather, the windows are those where the
pooled features match the desired dimensionality. We
empirically ﬁnd that these restricted windows are less
beneﬁcial than our ﬂexibly located/sized windows.
3.1.6 Summary and Results for ILSVRC 2014
In Table 4 we compare with previous state-of-theart methods. Krizhevsky et al .’s [3] is the winning
method in ILSVRC 2012; Overfeat [5], Howard’s [36],
and Zeiler and Fergus’s [4] are the leading methodsrank team top-5 test
1 GoogLeNet [32] 6.66
2 VGG [33] 7.32
3 ours 8.06
4 Howard 8.11
5 DeeperVision 9.50
6 NUS-BST 9.79
7 TTIC ECP 10.22


key: chroma:document
string_value:
3 ours 8.06
4 Howard 8.11
5 DeeperVision 9.50
6 NUS-BST 9.79
7 TTIC ECP 10.22
Table 5: The competition results of ILSVRC 2014 classiﬁcation [26]. The best entry of each team is listed.
in ILSVRC 2013. We only consider single-network
performance for manageable comparisons.
Our best single network achieves 9.14% top-5 error
on the validation set. This is exactly the single-model
entry we submitted to ILSVRC 2014 [26]. The top-5
error is 9.08% on the testing set (ILSVRC 2014 has
the same training/validation/testing data as ILSVRC
2012). After combining eleven models, our team’s result ( 8.06%) is ranked #3 among all 38 teams attending
ILSVRC 2014 (Table 5). Since the advantages of SPPnet should be in general independent of architectures,
we expect that it will further improve the deeper and
larger convolutional architectures [33], [32].
3.2 Experiments on VOC 2007 Classiﬁcation
Our method can generate a full-view image representation. With the above networks pre-trained on
ImageNet, we extract these representations from the


key: chroma:document
string_value:
Our method can generate a full-view image representation. With the above networks pre-trained on
ImageNet, we extract these representations from the
images in the target datasets and re-train SVM classiﬁers [38]. In the SVM training, we intentionally do
not use any data augmentation (ﬂip/multi-view). We
l2-normalize the features for SVM training.
The classiﬁcation task in Pascal VOC 2007 [22]
involves 9,963 images in 20 categories. 5,011 images
are for training, and the rest are for testing. The
performance is evaluated by mean Average Precision
(mAP). Table 6 summarizes the results.
8
(a) (b) (c) (d) (e)
model no SPP (ZF-5) SPP (ZF-5) SPP (ZF-5) SPP (ZF-5) SPP (Overfeat-7)
crop crop full full full
size 224224 224224 224- 392- 364conv 4 59.96 57.28 - - conv 5 66.34 65.43 - - pool 5=7(66) 69.14 68.76 70.82 71.67 76.09


key: chroma:document
string_value:
fc6=8 74.86 75.55 77.32 78.78 81.58
fc7=9 75.90 76.45 78.39 80.10 82.44
Table 6: Classiﬁcation mAP in Pascal VOC 2007. For SPP-net, the pool 5=7layer uses the 66 pyramid level.
(a) (b) (c) (d)
model no SPP (ZF-5) SPP (ZF-5) SPP (ZF-5) SPP (Overfeat-7)
crop crop full full
size 224224 224224 224- 224conv 4 80.12 81.03 - conv 5 84.40 83.76 - pool 5=7(66) 87.98 87.60 89.46 91.46
SPP pool 5=7 - 89.47 91.44 93.42
fc6=8 87.86 88.54 89.50 91.83
fc7=9 85.30 86.10 87.08 90.00
Table 7: Classiﬁcation accuracy in Caltech101. For SPP-net, the pool 5=7layer uses the 66 pyramid level.
We start from a baseline in Table 6 (a). The model is


key: chroma:document
string_value:
We start from a baseline in Table 6 (a). The model is
ZF-5 without SPP . To apply this model, we resize the
image so that its smaller dimension is 224, and crop
the center 224224 region. The SVM is trained via
the features of a layer. On this dataset, the deeper the
layer is, the better the result is. In Table 6 (b), we replace the no-SPP net with our SPP-net. As a ﬁrst-step
comparison, we still apply the SPP-net on the center
224224 crop. The results of the fc layers improve.
This gain is mainly due to multi-level pooling.
Table 6 (c) shows our results on full images, where
the images are resized so that the shorter side is 224.
We ﬁnd that the results are considerably improved
(78.39% vs. 76.45%). This is due to the full-image
representation that maintains the complete content.
Because the usage of our network does not depend
on scale, we resize the images so that the smaller
dimension is sand use the same network to extract
features. We ﬁnd that s= 392 gives the best results
(Table 6 (d)) based on the validation set. This is mainly
because the objects occupy smaller regions in VOC


key: chroma:document
string_value:
(Table 6 (d)) based on the validation set. This is mainly
because the objects occupy smaller regions in VOC
2007 but larger regions in ImageNet, so the relative
object scales are different between the two sets. These
results indicate scale matters in the classiﬁcation tasks,
and SPP-net can partially address this “scale mismatch” issue.
In Table 6 (e) the network architecture is replaced
with our best model (Overfeat-7, multi-size trained),
and the mAP increases to 82.44% . Table 8 summarizes
our results and the comparisons with the state-of-theart methods. Among these methods, VQ [15], LCC
[18], and FK [19] are all based on spatial pyramids
matching, and [13], [4], [34], [6] are based on deepnetworks. In these results, Oquab et al.’s (77.7%) and
Chatﬁeld et al .’s (82.42%) are obtained by network
ﬁne-tuning and multi-view testing. Our result is comparable with the state of the art, using only a single
full-image representation and without ﬁne-tuning.
3.3 Experiments on Caltech101


key: chroma:document
string_value:
full-image representation and without ﬁne-tuning.
3.3 Experiments on Caltech101
The Caltech101 dataset [21] contains 9,144 images in
102 categories (one background). We randomly sample 30 images per category for training and up to 50
images per category for testing. We repeat 10 random
splits and average the accuracy. Table 7 summarizes
our results.
There are some common observations in the Pascal
VOC 2007 and Caltech101 results: SPP-net is better
than the no-SPP net (Table 7 (b) vs. (a)), and the fullview representation is better than the crop ((c) vs. (b)).
But the results in Caltech101 have some differences
with Pascal VOC. The fully-connected layers are less
accurate, and the SPP layers are better. This is possibly
because the object categories in Caltech101 are less
related to those in ImageNet, and the deeper layers
are more category-specialized. Further, we ﬁnd that
the scale 224 has the best performance among the
scales we tested on this dataset. This is mainly because
the objects in Caltech101 also occupy large regions of
the images, as is the case of ImageNet.
Besides cropping, we also evaluate warping the
image to ﬁt the 224 224 size. This solution maintains


key: chroma:document
string_value:
the images, as is the case of ImageNet.
Besides cropping, we also evaluate warping the
image to ﬁt the 224 224 size. This solution maintains
the complete content, but introduces distortion. On
the SPP (ZF-5) model, the accuracy is 89.91% using
the SPP layer as features - lower than 91.44% which
uses the same model on the undistorted full image.
9
method VOC 2007 Caltech101
VQ [15]y56.07 74.41 1.0
LLC [18]y57.66 76.95 0.4
FK [19]y61.69 77.78 0.6
DeCAF [13] - 86.91 0.7
Zeiler & Fergus [4] 75.90z86.50.5
Oquab et al. [34] 77.7 Chatﬁeld et al. [6] 82.42 88.540.3
ours 82.44 93.42 0.5
Table 8: Classiﬁcation results for Pascal VOC 2007
(mAP) and Caltech101 (accuracy).ynumbers reported
by [27].zour implementation as in Table 6 (a).
Table 8 summarizes our results compared with the


key: chroma:document
string_value:
(mAP) and Caltech101 (accuracy).ynumbers reported
by [27].zour implementation as in Table 6 (a).
Table 8 summarizes our results compared with the
state-of-the-art methods on Caltech101. Our result
(93.42% ) exceeds the previous record (88.54%) by a
substantial margin (4.88%).
4 SPP- NET FOR OBJECT DETECTION
Deep networks have been used for object detection.
We brieﬂy review the recent state-of-the-art R-CNN
method [7]. R-CNN ﬁrst extracts about 2,000 candidate windows from each image via selective search
[20]. Then the image region in each window is warped
to a ﬁxed size (227 227). A pre-trained deep network
is used to extract the feature of each window. A
binary SVM classiﬁer is then trained on these features
for detection. R-CNN generates results of compelling
quality and substantially outperforms previous methods. However, because R-CNN repeatedly applies the
deep convolutional network to about 2,000 windows
per image, it is time-consuming. Feature extraction is
the major timing bottleneck in testing.
Our SPP-net can also be used for object detection.
We extract the feature maps from the entire image


key: chroma:document
string_value:
the major timing bottleneck in testing.
Our SPP-net can also be used for object detection.
We extract the feature maps from the entire image
only once (possibly at multiple scales). Then we apply the spatial pyramid pooling on each candidate
window of the feature maps to pool a ﬁxed-length
representation of this window (see Figure 5). Because
the time-consuming convolutions are only applied
once, our method can run orders of magnitude faster.
Our method extracts window-wise features from
regions of the feature maps, while R-CNN extracts
directly from image regions. In previous works, the
Deformable Part Model (DPM) [23] extracts features
from windows in HOG [24] feature maps, and the
Selective Search (SS) method [20] extracts from windows in encoded SIFT feature maps. The Overfeat
detection method [5] also extracts from windows of
deep convolutional feature maps, but needs to predeﬁne the window size. On the contrary, our method
enables feature extraction in arbitrary windows from
the deep convolutional feature maps.
spatial pyramid 
pooling layer
feature maps of conv 5
convolutional layersfixed-length representation
input imagewindow…...fully-connected layers (fc 6, fc 7)Figure 5: Pooling features from arbitrary windows


key: chroma:document
string_value:
convolutional layersfixed-length representation
input imagewindow…...fully-connected layers (fc 6, fc 7)Figure 5: Pooling features from arbitrary windows
on feature maps. The feature maps are computed
from the entire image. The pooling is performed in
candidate windows.
4.1 Detection Algorithm
We use the “fast” mode of selective search [20] to
generate about 2,000 candidate windows per image.
Then we resize the image such that min(w;h) =s,
and extract the feature maps from the entire image.
We use the SPP-net model of ZF-5 (single-size trained)
for the time being. In each candidate window, we use
a 4-level spatial pyramid (1 1, 22, 33, 66, totally
50 bins) to pool the features. This generates a 12,800d (25650) representation for each window. These
representations are provided to the fully-connected
layers of the network. Then we train a binary linear
SVM classiﬁer for each category on these features.
Our implementation of the SVM training follows
[20], [7]. We use the ground-truth windows to generate the positive samples. The negative samples are
those overlapping a positive window by at most 30%


key: chroma:document
string_value:
[20], [7]. We use the ground-truth windows to generate the positive samples. The negative samples are
those overlapping a positive window by at most 30%
(measured by the intersection-over-union (IoU) ratio).
Any negative sample is removed if it overlaps another
negative sample by more than 70%. We apply the standard hard negative mining [23] to train the SVM. This
step is iterated once. It takes less than 1 hour to train
SVMs for all 20 categories. In testing, the classiﬁer
is used to score the candidate windows. Then we use
non-maximum suppression [23] (threshold of 30%) on
the scored windows.
Our method can be improved by multi-scale feature
extraction. We resize the image such that min(w;h) =
s2S=f480;576;688;864;1200g, and compute the
feature maps of conv 5for each scale. One strategy of
combining the features from these scales is to pool
them channel-by-channel. But we empirically ﬁnd
that another strategy provides better results. For each
candidate window, we choose a single scale s2S
such that the scaled candidate window has a number
of pixels closest to 224 224. Then we only use the
feature maps extracted from this scale to compute


key: chroma:document
string_value:
such that the scaled candidate window has a number
of pixels closest to 224 224. Then we only use the
feature maps extracted from this scale to compute
10
the feature of this window. If the pre-deﬁned scales
are dense enough and the window is approximately
square, our method is roughly equivalent to resizing
the window to 224 224 and then extracting features
from it. Nevertheless, our method only requires computing the feature maps once (at each scale) from the
entire image, regardless of the number of candidate
windows.
We also ﬁne-tune our pre-trained network, following [7]. Since our features are pooled from the conv 5
feature maps from windows of any sizes, for simplicity we only ﬁne-tune the fully-connected layers.
In this case, the data layer accepts the ﬁxed-length
pooled features after conv 5, and the fc 6;7layers and
a new 21-way (one extra negative category) fc 8layer
follow. The fc 8weights are initialized with a Gaussian
distribution of =0.01. We ﬁx all the learning rates to
1e-4 and then adjust to 1e-5 for all three layers. During


key: chroma:document
string_value:
distribution of =0.01. We ﬁx all the learning rates to
1e-4 and then adjust to 1e-5 for all three layers. During
ﬁne-tuning, the positive samples are those overlapping with a ground-truth window by [0:5;1], and
the negative samples by [0:1;0:5). In each mini-batch,
25% of the samples are positive. We train 250k minibatches using the learning rate 1e-4, and then 50k
mini-batches using 1e-5. Because we only ﬁne-tune
the fc layers, the training is very fast and takes about
2 hours on the GPU (excluding pre-caching feature
maps which takes about 1 hour). Also following [7],
we use bounding box regression to post-process the
prediction windows. The features used for regression
are the pooled features from conv 5(as a counterpart
of the pool 5features used in [7]). The windows used
for the regression training are those overlapping with
a ground-truth window by at least 50%.
4.2 Detection Results
We evaluate our method on the detection task of the
Pascal VOC 2007 dataset. Table 9 shows our results
on various layers, by using 1-scale ( s=688) or 5-scale.


key: chroma:document
string_value:
Pascal VOC 2007 dataset. Table 9 shows our results
on various layers, by using 1-scale ( s=688) or 5-scale.
Here the R-CNN results are as reported in [7] using
the AlexNet [3] with 5 conv layers. Using the pool 5
layers (in our case the pooled features), our result
(44.9%) is comparable with R-CNN’s result (44.2%).
But using the non-ﬁne-tuned fc 6layers, our results are
inferior. An explanation is that our fc layers are pretrained using image regions, while in the detection
case they are used on the feature map regions. The
feature map regions can have strong activations near
the window boundaries, while the image regions may
not. This difference of usages can be addressed by
ﬁne-tuning. Using the ﬁne-tuned fc layers (ftfc 6;7), our
results are comparable with or slightly better than
the ﬁne-tuned results of R-CNN. After bounding box
regression, our 5-scale result ( 59.2%) is 0.7% better
than R-CNN (58.5%), and our 1-scale result (58.0%)
is 0.5% worse.


key: chroma:document
string_value:
than R-CNN (58.5%), and our 1-scale result (58.0%)
is 0.5% worse.
In Table 10 we further compare with R-CNN using the same pre-trained model of SPPnet (ZF-5). InSPP (1-sc) SPP (5-sc) R-CNN
(ZF-5) (ZF-5) (Alex-5)
pool 5 43.0 44.9 44.2
fc6 42.5 44.8 46.2
ftfc6 52.3 53.7 53.1
ftfc7 54.5 55.2 54.2
ftfc7bb 58.0 59.2 58.5
conv time (GPU) 0.053s 0.293s 8.96s
fc time (GPU) 0.089s 0.089s 0.07s
total time (GPU) 0.142s 0.382s 9.03s
speedup ( vs. RCNN) 64 24 Table 9: Detection results (mAP) on Pascal VOC 2007.
“ft” and “bb” denote ﬁne-tuning and bounding box
regression.
SPP (1-sc) SPP (5-sc) R-CNN


key: chroma:document
string_value:
regression.
SPP (1-sc) SPP (5-sc) R-CNN
(ZF-5) (ZF-5) (ZF-5)
ftfc7 54.5 55.2 55.1
ftfc7bb 58.0 59.2 59.2
conv time (GPU) 0.053s 0.293s 14.37s
fc time (GPU) 0.089s 0.089s 0.089s
total time (GPU) 0.142s 0.382s 14.46s
speedup ( vs. RCNN) 102 38 Table 10: Detection results (mAP) on Pascal VOC 2007,
using the same pre-trained model of SPP (ZF-5).
this case, our method and R-CNN have comparable
averaged scores. The R-CNN result is boosted by
this pre-trained model. This is because of the better
architecture of ZF-5 than AlexNet, and also because
of the multi-level pooling of SPPnet (if using the noSPP ZF-5, the R-CNN result drops). Table 11 shows
the results for each category.
Table 11 also includes additional methods. Selective
Search (SS) [20] applies spatial pyramid matching on


key: chroma:document
string_value:
the results for each category.
Table 11 also includes additional methods. Selective
Search (SS) [20] applies spatial pyramid matching on
SIFT feature maps. DPM [23] and Regionlet [39] are
based on HOG features [24]. The Regionlet method
improves to 46.1% [8] by combining various features including conv 5. DetectorNet [40] trains a deep
network that outputs pixel-wise object masks. This
method only needs to apply the deep network once
to the entire image, as is the case for our method. But
this method has lower mAP (30.5%).
4.3 Complexity and Running Time
Despite having comparable accuracy, our method is
much faster than R-CNN. The complexity of the convolutional feature computation in R-CNN is O(n
2272)with the window number n(2000). This complexity of our method is O(rs2)at a scales, where
ris the aspect ratio. Assume ris about 4/3. In the
single-scale version when s= 688 , this complexity is
11
method mAP areo bike bird boat bottle bus car cat chair cow table dog horse mbike person plant sheep sofa train tv


key: chroma:document
string_value:
11
method mAP areo bike bird boat bottle bus car cat chair cow table dog horse mbike person plant sheep sofa train tv
DPM [23] 33.7 33.2 60.3 10.2 16.1 27.3 54.3 58.2 23.0 20.0 24.1 26.7 12.7 58.1 48.2 43.2 12.0 21.1 36.1 46.0 43.5
SS [20] 33.8 43.5 46.5 10.4 12.0 9.3 49.4 53.7 39.4 12.5 36.9 42.2 26.4 47.0 52.4 23.5 12.1 29.9 36.3 42.2 48.8
Regionlet [39] 41.7 54.2 52.0 20.3 24.0 20.1 55.5 68.7 42.6 19.2 44.2 49.1 26.6 57.0 54.5 43.4 16.4 36.6 37.7 59.4 52.3


key: chroma:document
string_value:
DetNet [40] 30.5 29.2 35.2 19.4 16.7 3.7 53.2 50.2 27.2 10.2 34.8 30.2 28.2 46.6 41.7 26.2 10.3 32.8 26.8 39.8 47.0
RCNN ftfc 7(A5) 54.2 64.2 69.7 50.0 41.9 32.0 62.6 71.0 60.7 32.7 58.5 46.5 56.1 60.6 66.8 54.2 31.5 52.8 48.9 57.9 64.7
RCNN ftfc 7(ZF5) 55.1 64.8 68.4 47.0 39.5 30.9 59.8 70.5 65.3 33.5 62.5 50.3 59.5 61.6 67.9 54.1 33.4 57.3 52.9 60.2 62.9
SPP ftfc 7(ZF5) 55.2 65.5 65.9 51.7 38.4 32.7 62.6 68.6 69.7 33.1 66.6 53.1 58.2 63.6 68.8 50.4 27.4 53.7 48.2 61.7 64.7


key: chroma:document
string_value:
RCNN bb (A5) 58.5 68.1 72.8 56.8 43.0 36.8 66.3 74.2 67.6 34.4 63.5 54.5 61.2 69.1 68.6 58.7 33.4 62.9 51.1 62.5 64.8
RCNN bb (ZF5) 59.2 68.4 74.0 54.0 40.9 35.2 64.1 74.4 69.8 35.5 66.9 53.8 64.2 69.9 69.6 58.9 36.8 63.4 56.0 62.8 64.9
SPP bb (ZF5) 59.2 68.6 69.7 57.1 41.2 40.5 66.3 71.3 72.5 34.4 67.3 61.7 63.1 71.0 69.8 57.6 29.7 59.0 50.2 65.2 68.0
Table 11: Comparisons of detection results on Pascal VOC 2007.
method mAP areo bike bird boat bottle bus car cat chair cow table dog horse mbike person plant sheep sofa train tv


key: chroma:document
string_value:
Table 11: Comparisons of detection results on Pascal VOC 2007.
method mAP areo bike bird boat bottle bus car cat chair cow table dog horse mbike person plant sheep sofa train tv
SPP-net (1) 59.2 68.6 69.7 57.1 41.2 40.5 66.3 71.3 72.5 34.4 67.3 61.7 63.1 71.0 69.8 57.6 29.7 59.0 50.2 65.2 68.0
SPP-net (2) 59.1 65.7 71.4 57.4 42.4 39.9 67.0 71.4 70.6 32.4 66.7 61.7 64.8 71.7 70.4 56.5 30.8 59.9 53.2 63.9 64.6
combination 60.9 68.5 71.7 58.7 41.9 42.5 67.7 72.1 73.8 34.7 67.0 63.4 66.0 72.5 71.3 58.9 32.8 60.9 56.1 67.9 68.8
Table 12: Detection results on VOC 2007 using model combination. The results of both models use “ftfc 7bb”.
about 1/160 of R-CNN’s; in the 5-scale version, this


key: chroma:document
string_value:
about 1/160 of R-CNN’s; in the 5-scale version, this
complexity is about 1/24 of R-CNN’s.
In Table 10, we provide a fair comparison on the
running time of the feature computation using the
same SPP (ZF-5) model . The implementation of RCNN is from the code published by the authors implemented in Caffe [35]. We also implement our feature
computation in Caffe . In Table 10 we evaluate the
average time of 100 random VOC images using GPU.
R-CNN takes 14.37s per image for convolutions, while
our 1-scale version takes only 0.053s per image. So
ours is 270faster than R-CNN. Our 5-scale version
takes 0.293s per image for convolutions, so is 49 
faster than R-CNN. Our convolutional feature computation is so fast that the computational time of fc layers
takes a considerable portion. Table 10 shows that the
GPU time of computing the 4,096-d fc 7features is
0.089s per image. Considering both convolutional and
fully-connected features, our 1-scale version is 102
faster than R-CNN and is 1.2% inferior; our 5-scale


key: chroma:document
string_value:
fully-connected features, our 1-scale version is 102
faster than R-CNN and is 1.2% inferior; our 5-scale
version is 38faster and has comparable results.
We also compares the running time in Table 9 where
R-CNN uses AlexNet [3] as is in the original paper
[7]. Our method is 24 to 64faster. Note that the
AlexNet [3] has the same number of ﬁlters as our ZF5 on each conv layer. The AlexNet is faster because
it uses splitting on some layers, which was designed
for two GPUs in [3].
We further achieve an efﬁcient full system with the
help of the recent window proposal method [25]. The
Selective Search (SS) proposal [20] takes about 1-2 seconds per image on a CPU. The method of EdgeBoxes
[25] only takes0.2s. Note that it is sufﬁcient to use
a fast proposal method during testing only. Using the
same model trained as above (using SS), we test proposals generated by EdgeBoxes only. The mAP is 52.8
without bounding box regression. This is reasonableconsidering that EdgeBoxes are not used for training.
Then we use both SS and EdgeBox as proposals in


key: chroma:document
string_value:
without bounding box regression. This is reasonableconsidering that EdgeBoxes are not used for training.
Then we use both SS and EdgeBox as proposals in
the training stage, and adopt only EdgeBoxes in the
testing stage. The mAP is 56.3 without bounding box
regression, which is better than 55.2 (Table 10) due to
additional training samples. In this case, the overall
testing time is0.5s per image including all steps
(proposal and recognition). This makes our method
practical for real-world applications.
4.4 Model Combination for Detection
Model combination is an important strategy for boosting CNN-based classiﬁcation accuracy [3]. We propose a simple combination method for detection.
We pre-train another network in ImageNet, using
the same structure but different random initializations. Then we repeat the above detection algorithm.
Table 12 (SPP-net (2)) shows the results of this network. Its mAP is comparable with the ﬁrst network
(59.1% vs. 59.2%), and outperforms the ﬁrst network
in 11 categories.
Given the two models, we ﬁrst use either model
to score all candidate windows on the test image.
Then we perform non-maximum suppression on the
union of the two sets of candidate windows (with


key: chroma:document
string_value:
to score all candidate windows on the test image.
Then we perform non-maximum suppression on the
union of the two sets of candidate windows (with
their scores). A more conﬁdent window given by
one method can suppress those less conﬁdent given
by the other method. After combination, the mAP
is boosted to 60.9% (Table 12). In 17 out of all 20
categories the combination performs better than either
individual model. This indicates that the two models
are complementary.
We further ﬁnd that the complementarity is mainly
because of the convolutional layers. We have tried to
combine two randomly initialized ﬁne-tuned results
of the same convolutional model, and found no gain.
12
4.5 ILSVRC 2014 Detection
The ILSVRC 2014 detection [26] task involves 200
categories. There are 450k/20k/40k images in the
training/validation/testing sets. We focus on the task
of the provided-data-only track (the 1000-category
CLS training data is not allowed to use).
There are three major differences between the detection (DET) and classiﬁcation (CLS) training datasets,
which greatly impacts the pre-training quality. First,


key: chroma:document
string_value:
There are three major differences between the detection (DET) and classiﬁcation (CLS) training datasets,
which greatly impacts the pre-training quality. First,
the DET training data is merely 1/3 of the CLS
training data. This seems to be a fundamental challenge of the provided-data-only DET task. Second, the
category number of DET is 1/5 of CLS. To overcome
this problem, we harness the provided subcategory
labels2for pre-training. There are totally 499 nonoverlapping subcategories ( i.e., the leaf nodes in the
provided category hierarchy). So we pre-train a 499category network on the DET training set. Third, the
distributions of object scales are different between
DET/CLS training sets. The dominant object scale in
CLS is about 0.8 of the image length, but in DET is
about 0.5. To address the scale difference, we resize
each training image to min(w;h) = 400 (instead of
256), and randomly crop 224224views for training.
A crop is only used when it overlaps with a ground
truth object by at least 50%.
We verify the effect of pre-training on Pascal VOC
2007. For a CLS-pre-training baseline, we consider


key: chroma:document
string_value:
truth object by at least 50%.
We verify the effect of pre-training on Pascal VOC
2007. For a CLS-pre-training baseline, we consider
the pool 5features (mAP 43.0% in Table 9). Replaced
with a 200-category network pre-trained on DET, the
mAP signiﬁcantly drops to 32.7%. A 499-category
pre-trained network improves the result to 35.9%.
Interestingly, even if the amount of training data
do not increase, training a network of more categories boosts the feature quality. Finally, training with
min(w;h) = 400 instead of 256 further improves the
mAP to 37.8%. Even so, we see that there is still a
considerable gap to the CLS-pre-training result. This
indicates the importance of big data to deep learning.
For ILSVRC 2014, we train a 499-category Overfeat7 SPP-net. The remaining steps are similar to the
VOC 2007 case. Following [7], we use the validation
set to generate the positive/negative samples, with
windows proposed by the selective search fast mode.
The training set only contributes positive samples
using the ground truth windows. We ﬁne-tune the fc
layers and then train the SVMs using the samples in


key: chroma:document
string_value:
The training set only contributes positive samples
using the ground truth windows. We ﬁne-tune the fc
layers and then train the SVMs using the samples in
both validation and training sets. The bounding box
regression is trained on the validation set.
Our single model leads to 31.84% mAP in the
ILSVRC 2014 testing set [26]. We combine six similar
models using the strategy introduced in this paper.
The mAP is 35.11% in the testing set [26]. This result
ranks #2 in the provided-data-only track of ILSVRC
2014 (Table 13) [26]. The winning result is 37.21% from
2. Using the provided subcategory labels is allowed, as is explicitly stated in the competition introduction.rank team mAP
1 NUS 37.21
2 ours 35.11
3 UvA 32.02
- (our single-model) (31.84)
4 Southeast-CASIA 30.47
5 1-HKUST 28.86
6 CASIA CRIPAC 2 28.61
Table 13: The competition results of ILSVRC 2014
detection (provided-data-only track) [26]. The best
entry of each team is listed.
NUS, which uses contextual information.
Our system still shows great advantages on speed


key: chroma:document
string_value:
entry of each team is listed.
NUS, which uses contextual information.
Our system still shows great advantages on speed
for this dataset. It takes our single model 0.6 seconds
(0.5 for conv, 0.1 for fc, excluding proposals) per testing image on a GPU extracting convolutional features
from all 5 scales. Using the same model, it takes 32
seconds per image in the way of RCNN. For the 40k
testing images, our method requires 8 GPU hours to
compute convolutional features, while RCNN would
require 15 GPUdays.
5 C ONCLUSION
SPP is a ﬂexible solution for handling different scales,
sizes, and aspect ratios. These issues are important in
visual recognition, but received little consideration in
the context of deep networks. We have suggested a solution to train a deep network with a spatial pyramid
pooling layer. The resulting SPP-net shows outstanding accuracy in classiﬁcation/detection tasks and
greatly accelerates DNN-based detection. Our studies
also show that many time-proven techniques/insights
in computer vision can still play important roles in
deep-networks-based recognition.
APPENDIX A
In the appendix, we describe some implementation
details:
Mean Subtraction.
The 224224 cropped training/testing images are


key: chroma:document
string_value:
APPENDIX A
In the appendix, we describe some implementation
details:
Mean Subtraction.
The 224224 cropped training/testing images are
often pre-processed by subtracting the per-pixel mean
[3]. When input images are in any sizes, the ﬁxedsize mean image is not directly applicable. In the
ImageNet dataset, we warp the 224 224 mean image
to the desired size and then subtract it. In Pascal VOC
2007 and Caltech101, we use the constant mean (128)
in all the experiments.
Implementation of Pooling Bins.
We use the following implementation to handle all
bins when applying the network. Denote the width
and height of the conv 5feature maps (can be the
full image or a window) as wandh. For a pyramid level with nnbins, the (i;j)-th bin is in the
range of [bi1
nwc;di
nwe][bj1
nhc;dj
nhe]. Intuitively,
13
bottle: 0. 24person:1.20
sheep:1.52
chair:0.21
diningtable:0.78person:1.16pers on:1.05


key: chroma:document
string_value:
sheep:1.52
chair:0.21
diningtable:0.78person:1.16pers on:1.05
pottedplant:0.21chair:4.79pottedplant:0.73
c hair: 0. 33diningtable: 0.34
chair:0.89
bus:0.56
car:3.24
ca r:3.4 5pers on:1.52
train:0.31
train:1.62
pottedplant:0.33
pottedplant:0.78
sofa:0.55tvmonitor:1.77
aeroplane: 0.45
aeroplane:1.40aeroplane:1.01
aeroplane:0.94aeroplane:0.93
aeroplane:0.91aeroplane:0.79
aeroplane:0.57 aeroplane:0.54
person:0.93
person:0.68
horse:1.73pers on:1.91
boat:0.60pers on:3.20
car:2.52
bus:2.01
car:0.93per son:4.16person:0.79
pers on:0.32
hors e:1.68
hors e:0.61
horse:1.29


key: chroma:document
string_value:
pers on:0.32
hors e:1.68
hors e:0.61
horse:1.29
per son:1.23
c hair:0.87pers on:1.79
per son:0.91
sofa:0.22person:0.85
sofa:0.58
c ow:2.36
cow:1.88co w:1 .86
cow:1.82cow:1.39 cow:1.31
ca t:0.5 2person:1.02
person:0.40
bicy cle:2.85bicy cle:2.71
bicycle:2.04bicy cle:0.67person:3.35
person:2.39
pers on:2.11
pers on:0.27
person:0.22bus:1.42
person:3.29person:1.18
bottle:1.15pottedplant:0.81
s heep:1.81s heep:1.17
sheep:0.81
bird:0.24pott edplant:0.35 pot tedplant: 0. 20
car:1.31person:1.60
person:0.62
dog:0.37person:0.38
dog:0.99person:1.48


key: chroma:document
string_value:
car:1.31person:1.60
person:0.62
dog:0.37person:0.38
dog:0.99person:1.48
pers on:0.22
cow:0.80person:3.29
person:2.69pers on:2.42
person:1.05
person:0.92pers on:0.76
bird:1.39
bird:0.84
bottle:1.20diningtable:0.96pers on:1.53pers on:1.52
pers on:0.73
car:0.12
car:0.11
car:0.04car:0.03
car:3.98car:1.95car:1.39
car:0.50
bird:1.47
sofa:0.41 person:2.15pers on:0 .86tvmonitor:2.24
motorbike:1.11motorbike:0.74person:1.36
pers on:1.10
Figure 6: Example detection results of “SPP-net ftfc 7bb” on the Pascal VOC 2007 testing set (59.2% mAP).
All windows with scores >0 are shown. The predicted category/score are marked. The window color is


key: chroma:document
string_value:
All windows with scores >0 are shown. The predicted category/score are marked. The window color is
associated with the predicted category. These images are manually selected because we ﬁnd them impressive.
Visit our project website to see all 4,952 detection results in the testing set.
if rounding is needed, we take the ﬂoor operation on
the left/top boundary and ceiling on the right/bottom
boundary.
Mapping a Window to Feature Maps.
In the detection algorithm (and multi-view testing
on feature maps), a window is given in the image
domain, and we use it to crop the convolutional feature maps ( e.g., conv 5) which have been sub-sampled
several times. So we need to align the window on the
feature maps.
In our implementation, we project the corner point
of a window onto a pixel in the feature maps, such
that this corner point in the image domain is closest
to the center of the receptive ﬁeld of that feature mappixel. The mapping is complicated by the padding
of all convolutional and pooling layers. To simplify
the implementation, during deployment we pad bp=2c
pixels for a layer with a ﬁlter size of p. As such, for
a response centered at (x0;y0), its effective receptive


key: chroma:document
string_value:
pixels for a layer with a ﬁlter size of p. As such, for
a response centered at (x0;y0), its effective receptive
ﬁeld in the image domain is centered at (x;y) =
(Sx0;Sy0)whereSis the product of all previous
strides. In our models, S= 16 for ZF-5 on conv 5,
andS= 12 for Overfeat-5/7 on conv 5=7. Given a
window in the image domain, we project the left (top)
boundary by: x0=bx=Sc+ 1 and the right (bottom)
boundaryx0=dx=Se1. If the padding is not bp=2c,
we need to add a proper offset to x.
14
REFERENCES
[1] Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard,
W. Hubbard, and L. D. Jackel, “Backpropagation applied to
handwritten zip code recognition,” Neural computation , 1989.


key: chroma:document
string_value:
W. Hubbard, and L. D. Jackel, “Backpropagation applied to
handwritten zip code recognition,” Neural computation , 1989.
[2] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. FeiFei, “Imagenet: A large-scale hierarchical image database,” in
CVPR , 2009.
[3] A. Krizhevsky, I. Sutskever, and G. Hinton, “Imagenet classiﬁcation with deep convolutional neural networks,” in NIPS ,
2012.
[4] M. D. Zeiler and R. Fergus, “Visualizing and understanding
convolutional neural networks,” arXiv:1311.2901 , 2013.
[5] P . Sermanet, D. Eigen, X. Zhang, M. Mathieu, R. Fergus,
and Y. LeCun, “Overfeat: Integrated recognition, localization
and detection using convolutional networks,” arXiv:1312.6229 ,
2013.
[6] A. V . K. Chatﬁeld, K. Simonyan and A. Zisserman, “Return of


key: chroma:document
string_value:
2013.
[6] A. V . K. Chatﬁeld, K. Simonyan and A. Zisserman, “Return of
the devil in the details: Delving deep into convolutional nets,”
inArXiv:1405.3531 , 2014.
[7] R. Girshick, J. Donahue, T. Darrell, and J. Malik, “Rich feature
hierarchies for accurate object detection and semantic segmentation,” in CVPR , 2014.
[8] W. Y. Zou, X. Wang, M. Sun, and Y. Lin, “Generic object detection with dense neural patterns and regionlets,” in
ArXiv:1404.4316 , 2014.
[9] A. S. Razavian, H. Azizpour, J. Sullivan, and S. Carlsson, “Cnn
features off-the-shelf: An astounding baseline for recogniton,”
inCVPR 2014, DeepVision Workshop , 2014.
[10] Y. Taigman, M. Yang, M. Ranzato, and L. Wolf, “Deepface:
Closing the gap to human-level performance in face veriﬁcation,” in CVPR , 2014.


key: chroma:document
string_value:
Closing the gap to human-level performance in face veriﬁcation,” in CVPR , 2014.
[11] N. Zhang, M. Paluri, M. Ranzato, T. Darrell, and L. Bourdevr,
“Panda: Pose aligned networks for deep attribute modeling,”
inCVPR , 2014.
[12] Y. Gong, L. Wang, R. Guo, and S. Lazebnik, “Multi-scale
orderless pooling of deep convolutional activation features,”
inArXiv:1403.1840 , 2014.
[13] J. Donahue, Y. Jia, O. Vinyals, J. Hoffman, N. Zhang, E. Tzeng,
and T. Darrell, “Decaf: A deep convolutional activation feature
for generic visual recognition,” arXiv:1310.1531 , 2013.
[14] K. Grauman and T. Darrell, “The pyramid match kernel:
Discriminative classiﬁcation with sets of image features,” in
ICCV , 2005.
[15] S. Lazebnik, C. Schmid, and J. Ponce, “Beyond bags of features: Spatial pyramid matching for recognizing natural scene


key: chroma:document
string_value:
[15] S. Lazebnik, C. Schmid, and J. Ponce, “Beyond bags of features: Spatial pyramid matching for recognizing natural scene
categories,” in CVPR , 2006.
[16] J. Sivic and A. Zisserman, “Video google: a text retrieval
approach to object matching in videos,” in ICCV , 2003.
[17] J. Yang, K. Yu, Y. Gong, and T. Huang, “Linear spatial pyramid
matching using sparse coding for image classiﬁcation,” in
CVPR , 2009.
[18] J. Wang, J. Yang, K. Yu, F. Lv, T. Huang, and Y. Gong, “Localityconstrained linear coding for image classiﬁcation,” in CVPR ,
2010.
[19] F. Perronnin, J. S ´anchez, and T. Mensink, “Improving the ﬁsher
kernel for large-scale image classiﬁcation,” in ECCV , 2010.
[20] K. E. van de Sande, J. R. Uijlings, T. Gevers, and A. W. Smeulders, “Segmentation as selective search for object recognition,”


key: chroma:document
string_value:
inICCV , 2011.
[21] L. Fei-Fei, R. Fergus, and P . Perona, “Learning generative
visual models from few training examples: An incremental
bayesian approach tested on 101 object categories,” CVIU ,
2007.
[22] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and
A. Zisserman, “The PASCAL Visual Object Classes Challenge
2007 (VOC2007) Results,” 2007.
[23] P . F. Felzenszwalb, R. B. Girshick, D. McAllester, and D. Ramanan, “Object detection with discriminatively trained partbased models,” P AMI , 2010.
[24] N. Dalal and B. Triggs, “Histograms of oriented gradients for
human detection,” in CVPR , 2005.
[25] C. L. Zitnick and P . Doll ´ar, “Edge boxes: Locating object
proposals from edges,” in ECCV , 2014.
[26] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh,


key: chroma:document
string_value:
[26] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh,
S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein
et al. , “Imagenet large scale visual recognition challenge,”
arXiv:1409.0575 , 2014.
[27] K. Chatﬁeld, V . Lempitsky, A. Vedaldi, and A. Zisserman, “The
devil is in the details: an evaluation of recent feature encoding
methods,” in BMVC , 2011.
[28] A. Coates and A. Ng, “The importance of encoding versus
training with sparse coding and vector quantization,” in ICML ,
2011.
[29] D. G. Lowe, “Distinctive image features from scale-invariant
keypoints,” IJCV , 2004.
[30] J. C. van Gemert, J.-M. Geusebroek, C. J. Veenman, and A. W.
Smeulders, “Kernel codebooks for scene categorization,” in


key: chroma:document
string_value:
Smeulders, “Kernel codebooks for scene categorization,” in
ECCV , 2008.[31] M. Lin, Q. Chen, and S. Yan, “Network in network,”
arXiv:1312.4400 , 2013.
[32] C. Szegedy, W. Liu, Y. Jia, P . Sermanet, S. Reed, D. Anguelov,
D. Erhan, V . Vanhoucke, and A. Rabinovich, “Going deeper
with convolutions,” arXiv:1409.4842 , 2014.
[33] K. Simonyan and A. Zisserman, “Very deep convolutional
networks for large-scale image recognition,” arXiv:1409.1556 ,
2014.
[34] M. Oquab, L. Bottou, I. Laptev, J. Sivic et al. , “Learning and
transferring mid-level image representations using convolutional neural networks,” in CVPR , 2014.
[35] Y. Jia, “Caffe: An open source convolutional architecture
for fast feature embedding,” http://caffe.berkeleyvision.org/,
2013.


key: chroma:document
string_value:
for fast feature embedding,” http://caffe.berkeleyvision.org/,
2013.
[36] A. G. Howard, “Some improvements on deep convolutional
neural network based image classiﬁcation,” ArXiv:1312.5402 ,
2013.
[37] H. Jegou, F. Perronnin, M. Douze, J. Sanchez, P . Perez, and
C. Schmid, “Aggregating local image descriptors into compact
codes,” TP AMI , vol. 34, no. 9, pp. 1704–1716, 2012.
[38] C.-C. Chang and C.-J. Lin, “Libsvm: a library for support
vector machines,” ACM Transactions on Intelligent Systems and
Technology (TIST) , 2011.
[39] X. Wang, M. Yang, S. Zhu, and Y. Lin, “Regionlets for generic
object detection,” in ICCV , 2013.
[40] C. Szegedy, A. Toshev, and D. Erhan, “Deep neural networks
for object detection,” in NIPS , 2013.
CHANGELOG
arXiv v1 . Initial technical report for ECCV 2014 paper.


key: chroma:document
string_value:
for object detection,” in NIPS , 2013.
CHANGELOG
arXiv v1 . Initial technical report for ECCV 2014 paper.
arXiv v2 . Submitted version for TPAMI. Includes extra
experiments of SPP on various architectures. Includes
details for ILSVRC 2014.
arXiv v3 . Accepted version for TPAMI. Includes comparisons with R-CNN using the same architecture.
Includes detection experiments using EdgeBoxes.
arXiv v4 . Revised “Mapping a Window to Feature
Maps” in Appendix for easier implementation.


key: chroma:document
string_value:
Accepted as a workshop contribution at ICLR 2015
LEARNING DEEPSTRUCTURED MODELS
Liang-Chieh Chen
Department of Computer Science, UCLA, lcchen@cs.ucla.edu
Alexander G. Schwing
Department of Computer Science, University of Toronto, aschwing@cs.toronto.edu
Alan L. Yuille
Department of Statistics, UCLA, yuille@stat.ucla.edu
Raquel Urtasun
Department of Computer Science, University of Toronto, urtasun@cs.toronto.edu
ABSTRACT
Many problems in real-world applications involve predicting several random variables which are statistically related. Markov random ﬁelds (MRFs) are a great
mathematical tool to encode such relationships. The goal of this paper is to combine MRFs with deep learning algorithms to estimate complex representations
while taking into account the dependencies between the output random variables.
Towards this goal, we propose a training algorithm that is able to learn structured
models jointly with deep features that form the MRF potentials. Our approach is
efﬁcient as it blends learning and inference and makes use of GPU acceleration.
We demonstrate the effectiveness of our algorithm in the tasks of predicting words
from noisy images, as well as multi-class classiﬁcation of Flickr photographs. We


key: chroma:document
string_value:
We demonstrate the effectiveness of our algorithm in the tasks of predicting words
from noisy images, as well as multi-class classiﬁcation of Flickr photographs. We
show that joint learning of the deep features and the MRF parameters results in
signiﬁcant performance gains.
1 I NTRODUCTION
Deep learning algorithms attempt to model high-level abstractions of the data using architectures composed of multiple non-linear transformations. A multiplicity of variants have been proposed (Hinton et al., 1984; LeCun et al., 1998; Hinton & Salakhutdinov, 2006; Bengio et al., 2007;
Salakhutdinov & Hinton, 2012; Zeiler & Fergus, 2014) and shown to be extremely successful in a
wide variety of applications including object detection, speech recognition as well as natural language processing (Lee et al., 2009; Socher et al., 2012; Jia, 2013; Krizhevsky et al., 2013; Eigen
et al., 2014). Recently, state-of-the-art results have been achieved in many computer vision tasks,
outperforming competitive methods by a large margin (Krizhevsky et al., 2013; Girshick et al.,
2014).
Deep neural networks can, however, be even more powerful when combined with graphical models


key: chroma:document
string_value:
2014).
Deep neural networks can, however, be even more powerful when combined with graphical models
in order to capture the statistical dependencies between the variables of interest. For example, Deng
et al. (2014) exploit mutual exclusion, overlapping and subsumption properties of class labels in order to better predict in large scale classiﬁcation tasks. In pose estimation, more accurate predictions
can be obtained when encoding the spatial relationships between joint locations (Tompson et al.,
2014).
It is, however, an open problem how to develop scalable deep learning algorithms that can learn
higher-order knowledge taking into account the output variable’s dependencies. Existing approaches
often rely on a two-step process (Nowozin et al., 2011; Xu et al., 2014) where a non-linear classiﬁer
that employs deep features is trained ﬁrst, and its output is used to generate potentials for the structured predictor. This piece-wise training is, however, suboptimal as the deep features are learned
The ﬁrst two authors contributed equally to this work.
1arXiv:1407.2538v3  [cs.LG]  27 Apr 2015
Accepted as a workshop contribution at ICLR 2015


key: chroma:document
string_value:
1arXiv:1407.2538v3  [cs.LG]  27 Apr 2015
Accepted as a workshop contribution at ICLR 2015
while ignoring the dependencies between the variables of interest, e.g., independently learned segmentation and detection features (Hariharan et al., 2014) might be focusing on predicting the same
examples correctly. But when learned jointly they can improve their predictive power by exploiting
complementary information to ﬁx additional mistakes.
In this paper we extend deep learning algorithms to learn complex representations taking into account the dependencies between the output random variables. Towards this goal, we propose a
learning algorithm that is able to learn structured models with arbitrary graphs jointly with deep
features that form Markov random ﬁeld (MRF) potentials. Our approach is efﬁcient as it blends
learning and inference resulting in a single loop algorithm which makes use of GPU acceleration.
We demonstrate the effectiveness of our method in the tasks of predicting words from noisy images,
and multi-class classiﬁcation of Flickr photographs. We show that joint learning of deep features
and MRF parameters results in big performance gains.
2 L EARNING DEEPSTRUCTURED MODELS


key: chroma:document
string_value:
and MRF parameters results in big performance gains.
2 L EARNING DEEPSTRUCTURED MODELS
In this section we investigate how to learn ‘deep features’ that take into account the dependencies between the output variables. Let y2Y be the set of random variables y= (y1;:::;yN)
that we are interested in predicting. In this work we assume the space of valid conﬁgurations to
be a product space, i.e.,Y=QN
i=1Yi, and the domain of each individual variable yito be discrete, i.e.,Yi=f1;:::;jYijg. Given input data x2X and parameters w2RAof the function
F(x;y;w) :XY RA!R, inference amounts to ﬁnding the highest scoring conﬁguration
y= arg max yF(x;y;w):Note that ifFis a deep network and there are no connections between
the output variables to be predicted, inference corresponds to a forward pass to evaluate the function,
followed by independently ﬁnding the largest response for each variable. This can be interpreted as
inference in a graphical model with only unary potentials. However, for arbitrary graphical models


key: chroma:document
string_value:
inference in a graphical model with only unary potentials. However, for arbitrary graphical models
it is NP-hard to ﬁnd the maximizing conﬁguration ysince the inference program generally requires
a search over a space of sizeQN
i=1jYij. Note also that log-linear models are a special case of this
program, with F(x;y;w) =w>(x;y)and(x;y)denoting a feature vector computed using the
input-output pair (x;y).
In this work, we consider the general setting where F(x;y;w)is an arbitrary scalar-valued function ofwand(x;y). In our experiments Fis a function composition of non-linear base mappings
like convolutions, rectiﬁcations, pooling etc. We let the probability of an arbitrary conﬁguration ^y
be given by the annealed soft-max p(x;y)(^yjw;) =1
Z(x;w)exp(F(x;^y;w))1=:HerebyZ(x;w)
refers to the partition function, normalizing the distribution p(x;y)to lie within the probability simplexviaZ(x;w) =P


key: chroma:document
string_value:
refers to the partition function, normalizing the distribution p(x;y)to lie within the probability simplexviaZ(x;w) =P
^y2Yexp(F(x;^y;w))1=. The annealing/temperature parameter 0is
used to adjust the uniformity of the distribution. We consider general graphical models where the
computation of Z(x;w)is #P-hard.
2.1 L EARNING VIA GRADIENT DESCENT
During learning, given a training set Dof input-output pairs (x;y)2D, we are interested in ﬁnding
the parameters wof the model. We do so by maximizing the data likelihood, i.e., minimizing the
negative log-likelihood lnQ
(x;y)2Dp(x;y)(yjw;)which yields
min
wX
(x;y)2D(lnZ(x;w)F(x;y;w)): (1)
Note that this is equivalent to maximizing the cross-entropy between a target distributionp(x;y);tg(^y) =(^y=y)placing all its mass on the groundtruth label,


key: chroma:document
string_value:
and the model distribution p(x;y)(^yjw;). Hence Eq. (1) is equivalently obtained by
maxwP
(x;y);^y2Yp(x;y);tg(^y) lnp(x;y)(^yjw;). It is easily possible to incorporate more general target distributions into Eq. (1). Note also that = 0recovers the structured hinge loss objective.
Minimizing Eq. (1) w.r.t. wrequires computation of the gradient@
@wP
(x;y)2Dlnp(x;y)(yjw;),
which is given by a transformed difference between the distributions of the model p(x;y)(^yjw;)and
2
Accepted as a workshop contribution at ICLR 2015
Repeat until stopping criteria
1. Forward pass to compute F(x;^y;w)
2. Compute p(x;y)(^yjw;)
3. Backward pass via chain rule to obtain gradient
4. Update parameters w
Figure 1: Gradient descent algorithm for learning deep structured models.
the targetp(x;y);tg(^y):
X
(x;y)2DX
^y2Y@


key: chroma:document
string_value:
the targetp(x;y);tg(^y):
X
(x;y)2DX
^y2Y@
@wF(x;^y;w)
p(x;y)(^yjw;)p(x;y);tg(^y)
: (2)
A gradient descent algorithm for minimizing Eq. (1) will iterate between the following steps: (i) For
a givenwevaluate the function F, (ii) compute the model distribution p(x;y)(^yjw;), (iii) propagate
the difference between the model and target distribution using a backward pass (resembling the chain
rule for composite functions) and (iv) update the parameters w. This is summarized in Fig. 1.
2.2 A PPROXIMATE LEARNING
Note that for general graphical models the exact computation of p(x;y)(^yjw;)is not possible. As a
consequence it is intractable to compute the exact gradient of the cost-function given in Eq. (2) and
one has to resort to approximate solutions.
Inspired by approximations used for log-linear models, we make use of the following identity (Wainwright & Jordan, 2008; Koller & Friedman, 2009):


key: chroma:document
string_value:
Inspired by approximations used for log-linear models, we make use of the following identity (Wainwright & Jordan, 2008; Koller & Friedman, 2009):
lnZ(x;w) = max
p(x;y)(^y)2E[F(x;^y;w)] +H(p(x;y)); (3)
where Edenotes an expectation over p(x;y)(^y)andHrefers to the entropy.
For most applications, F(x;y;w)decomposes into a sum of functions, each depending on a local
subset of variables yr,i.e.,F(x;y;w) =P
r2Rfr(x;yr;w):Herebyris a restriction of the variable
tupley= (y1;:::;yN)to the subset rf1;:::;Ng,i.e.,yr= (yi)i2r. All subsets rrequired to
compute the model function Fare summarized in the set R.
Plugging this decomposition into Eq. (3), we equivalently get the log-partition function
lnZ(x;w)viamaxp(x;y)(^y)2P


key: chroma:document
string_value:
lnZ(x;w)viamaxp(x;y)(^y)2P
r;^yrp(x;y);r(^yr)fr(x;^yr;w) +H(p(x;y));where we use
marginalsp(x;y);r(^yr) =P
ynyrp(x;y)(y).
Despite the assumed locality of the scoring function, the learning task remains computationally
challenging since the entropy H(p(x;y))can only be computed exactly for a very small set of models,
e.g., models for which the joint distribution p(x;y)(y)is equivalently described by low tree-width
models. In addition, the marginalization constraints are exponential in size.
To deal with both issues a common solution in log-linear models is to approximate the true marginals
p(x;y);rwith local beliefs b(x;y);rthat are not required to fulﬁll marginalization constraints globally,
but only locally (Wainwright & Jordan, 2008). That means marginals b(x;y);rare not required to arise
from a common joint distribution p(x;y). In addition, we approximate the entropy via the fractional


key: chroma:document
string_value:
from a common joint distribution p(x;y). In addition, we approximate the entropy via the fractional
entropy (Wiegerinck & Heskes, 2003), i.e.,H(p(x;y))P
rcrH(b(x;y);r). Counting numbers cr
are employed to weight the marginal entropies. Putting all this together, we obtain the following
approximation for lnZ(x;w):
max
b(x;y)2C(x;y)X
r;^yrb(x;y);r(^yr)fr(x;^yr;w) +X
rcrH(b(x;y);r): (4)
Hereby beliefs are constrained to the local polytope
C(x;y)=(8r b (x;y);r2
8r;^yr;p2P(r)P
^ypn^yrb(x;y);p(^yp) =b(x;y);r(^yr);
3
Accepted as a workshop contribution at ICLR 2015
Let~cr;p=cp=(cr+P
p02P(r)cp0). Repeat until stopping criteria


key: chroma:document
string_value:
3
Accepted as a workshop contribution at ICLR 2015
Let~cr;p=cp=(cr+P
p02P(r)cp0). Repeat until stopping criteria
1. Forward pass to compute fr(x;^yr;w)8(x;y);r;yr
2. Get beliefs b(x;y);r/exp^fr(x;^yr;w;)
crby iterating over r:8(x;y);p2P(r);^yr
(x;y);p!r(^yr) =cplnX
^ypn^yrexpfp(x;^yp;w)P
p02P(p)(x;y);p!p0(^yp) +P
r02C(p)nr(x;y);r0!p(^yr0)
cp
(x;y);r!p(^yr)/~cr;p0
@fr(x;^yr;w) +X
c2C(r)(x;y);c!r(^yc) +X
p2P(r)(x;y);p!r(^yr)1
A(x;y);p!r(^yr)


key: chroma:document
string_value:
p2P(r)(x;y);p!r(^yr)1
A(x;y);p!r(^yr)
3. Backward pass via chain-rule for gradient g=P
(x;y);r;^yrb(x;y);r(^yr)rwfr(x;^yr;w)rwF(w)
4. Update parameters wusing stepsize viaw wg
Figure 2: Efﬁcient learning algorithm that blends learning and inference.
withP(r)the set of parents of region r,i.e.,P(r)fp2R :rpg, which subsumes those
regions for which we want the marginalization constraint to hold. Conversely, we deﬁne the set of
children asC(r) =fc2R:r2P(c)g.
We can thus rewrite the learning problem by plugging the approximations derived in Eq. (4) into
Eq. (1). This gives rise to the new approximated learning program
min
wX
(x;y)2D0
@max
b(x;y)2C(x;y)8
<
:X


key: chroma:document
string_value:
min
wX
(x;y)2D0
@max
b(x;y)2C(x;y)8
<
:X
r;^yrb(x;y);r(^yr)fr(x;^yr;w) +X
rcrH(b(x;y);r)9
=
;F(x;y;w)1
A:(5)
To iteratively update the parameters for the non-smooth approximated cost function given in Eq. (5)
we require a sub-gradient w.r.t. w, which in turn requires to solve the maximization w.r.t. the beliefs
bexactly. This is a non-trivial task in itself as inference in general graphical models is NP-hard.
Iterative message passing algorithms (Pearl, 1988; Yedidia et al., 2005; Wainwright et al., 2005;
Weiss et al., 2007; Meltzer et al., 2009) are typically employed. Importantly, note that combining
the procedure outlined in Fig. 1 with iterative message passing to approximate p(x;y)(^yjw;)results
in a double-loop algorithm which would be slow for many graphical models of interest.


key: chroma:document
string_value:
in a double-loop algorithm which would be slow for many graphical models of interest.
2.3 E FFICIENT APPROXIMATE LEARNING BY BLENDING LEARNING AND INFERENCE
In this section we propose a more efﬁcient algorithm that is based on the principle of blending
learning ( i.e., parameter updates) and inference. Thus we are interested in only performing a single
message passing iteration before updating the parameters w. Note that simply reducing the number
of iterations is generally not an option as the obtained beliefs b(x;y);rare by no means accurate.
However, assuming all counting numbers crto be positive, we can derive an algorithm that is able
to interleave minimization w.r.t. wand maximization of the beliefs b. Such a procedure is more
efﬁcient as we are able to update the parameters wmuch more frequently.
To interleave both programs we convert maximization of the beliefs into a minimization by employing the dual program as detailed in the following claim. This is possible since the maximization
problem is concave in b(x;y)if8r,cr0.
Claim 1 Assumecr08rand letF(w) =P
(x;y)2DF(x;y;w)denote the sum of empirical


key: chroma:document
string_value:
Claim 1 Assumecr08rand letF(w) =P
(x;y)2DF(x;y;w)denote the sum of empirical
function observations. Let (x;y);r!p(^yr)be the Lagrange multipliers for each marginalization
constraintP
^ypn^yrb(x;y);p(^yp) =b(x;y);r(^yr)within the polytope C(x;y). Then the approximated
general structured prediction task shown in Eq. (5)is equivalent to
min
w;X
(x;y);rcrlnX
^yrexp^fr(x;^yr;w;)
crF(w); (6)
4
Accepted as a workshop contribution at ICLR 2015
banal julep resty
drein yojan mothy
Figure 3: Samples from the Word50 dataset. High degree of rotation, scaling and translation.
where we employed the re-parameterization score ^fr(x;^yr;w;) =fr(x;^yr;w) +P
c2C(r)(x;y);c!r(^yc)P


key: chroma:document
string_value:
c2C(r)(x;y);c!r(^yc)P
p2P(r)(x;y);r!p(^yr).
Proof: To obtain the dual of the maximization w.r.t. b(x;y)we utilize its Lagrangian L(x;y)=P
r;^yrb(x;y);r(^yr)^fr(x;^yr;w;)+P
rcrH(b(x;y);r):Maximization of the Lagrangian w.r.t. the primal variables bis possible by employing the relationship stated in Eq. (3) locally 8r. We then obtain
the dual function being the ﬁrst term in Eq. (6). For strict convexity, i.e.,cr>0, we reconstruct the beliefs to be proportional to the exponentiated, loss-augmented re-parameterization score
b(x;y);r/exp^fr(x;^yr;w;)
cr:Forcr= 0the beliefs correspond to a uniform distribution over the set
of maximizers of the loss-augmented re-parameterization score ^fr(x;^yr;w;). 


key: chroma:document
string_value:
of maximizers of the loss-augmented re-parameterization score ^fr(x;^yr;w;). 
It is important to note that by applying duality we managed to convert the min-max task in Eq. (5)
into a single minimization as shown in Eq. (6). Performing block coordinate descent updates to
minimize Eq. (6), we are therefore able to interleave both, updating the weights ( i.e., learning) and
the messages ( i.e., inference). This results in a more efﬁcient algorithm, as inference does not have
to be run until convergence. Even a single update of the messages sufﬁces. We note that this is
possible only if cr08r. Strictly speaking, we require concavity only within the set of feasible
beliefsC(x;y). However, for simplicity we neglect this extension in the following.
Fig. 2 summarizes our efﬁcient deep structured prediction algorithm which iterates between the
following steps. Given parameters wwe perform a standard forward pass to compute fr(x;^yr;w)
for all regions. We then iterate through all regions rand use block-coordinate descent to ﬁnd the


key: chroma:document
string_value:
for all regions. We then iterate through all regions rand use block-coordinate descent to ﬁnd the
globally optimal value of Eq. (6) w.r.t. (x;y);r!p(^yr)8(x;y);^yr;p2P(r). This can be done in
closed form and therefore is computed very efﬁciently. We refer the reader to Schwing (2013) for
a derivation in the log-linear setting. We then compute the gradient using a standard backward pass
before we update the parameters by performing a step of size along the negative gradient.
2.4 I MPLEMENTATION DETAILS
We implemented the general algorithm presented in Fig. 2 in C++ as a library for Linux, Windows
and OS X platforms. It supports usage of the GPU for the forward and backward pass using both,
standard linear algebra packages and manually tuned GPU-kernels. In addition to standard gradient descent, we allow speciﬁcation of both mini-batches, moments and different regularizers like
2-norm and1-norm. Between iterations the step-size can be reduced based on either the negative log-likelihood or validation set performance. Contrasting available deep learning packages, our


key: chroma:document
string_value:
2-norm and1-norm. Between iterations the step-size can be reduced based on either the negative log-likelihood or validation set performance. Contrasting available deep learning packages, our
functionFis speciﬁed using a general computation tree. Hence we support an arbitrarily nested
function structure composed of data, parameters and function prototypes (convolution, afﬁne function aka fully connected, dropout, local response normalization, pooling, rectiﬁed linear, sigmoid
and softmax units). The aforementioned library is accompanied by a program performing learning,
inference and gradient checks. To accommodate for large datasets it reads data from HDF5 storage
while a second thread simultaneously performs the computation. Google protocol buffers are employed to effectively specify the function Fwithout the need to modify any source code. We will
release this library upon publication. We believe that it will be useful for many researchers.
5
Accepted as a workshop contribution at ICLR 2015
Graph MLP Method H1= 128H1= 256H1= 512H1= 768H1= 1024
1st order Markov One LayerUnary only 8.60 / 61.32 10.80 / 64.41 12.50 / 65.69 12.95 / 66.66 13.40 / 67.02


key: chroma:document
string_value:
JointTrain 16.80 / 65.28 25.20 / 70.75 31.80 / 74.90 33.05 / 76.42 34.30 / 77.02
PwTrain 12.70 / 64.35 18.00 / 68.27 22.80 / 71.29 23.25 / 72.62 26.30 / 73.96
PreTrainJoint 20.65 /67.42 25.70 /71.65 31.70 / 75.56 34.50 /77.14 35.85 /78.05
2nd order Markov One LayerJointTrain 25.50 / 67.13 34.60 / 73.19 45.55 / 79.60 51.55 /82.37 54.05 /83.57
PwTrain 10.05 / 58.90 14.10 / 63.44 18.10 / 67.31 20.40 / 70.14 22.20 / 71.25
PreTrainJoint 28.15 /69.07 36.85 /75.21 45.75 /80.09 50.10 / 82.30 52.25 / 83.39
H1= 512H2= 32H2= 64H2= 128H2= 256H2= 512


key: chroma:document
string_value:
H1= 512H2= 32H2= 64H2= 128H2= 256H2= 512
1st order Markov Two LayerUnary only 15.25 / 69.04 18.15 / 70.66 19.00 / 71.43 19.20 / 72.06 20.40 / 72.51
JointTrain 35.95 / 76.92 43.80 / 81.64 44.75 / 82.22 46.00 / 82.96 47.70 / 83.64
PwTrain 34.85 / 79.11 38.95 / 80.93 42.75 / 82.38 45.10 / 83.67 45.75 / 83.88
PreTrainJoint 42.25 /81.10 44.85 /82.96 46.85 /83.50 47.95 /84.21 47.05 / 84.08
2nd order Markov Two LayerJointTrain 54.65 / 83.98 61.80 / 87.30 66.15 / 89.09 64.85 / 88.93 68.00 / 89.96
PwTrain 39.95 / 81.14 48.25 / 84.45 52.65 / 86.24 57.10 / 87.61 62.90 / 89.49


key: chroma:document
string_value:
PwTrain 39.95 / 81.14 48.25 / 84.45 52.65 / 86.24 57.10 / 87.61 62.90 / 89.49
PreTrainJoint 62.60 /88.03 65.80 /89.32 68.75 /90.47 68.60 /90.42 69.35 /90.75
Table 1: Word / Character accuracy. Performance improves as (1) joint-training is employed, (2) the
model is more structured, and (3) deeper unary classiﬁers are utilized. The number of hidden units
for the ﬁrst and second layer are denoted as H1andH2respectively.
abcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyz
abcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyz
00.511.522.533.544.55
x 1043004005006007008009001000Neg. Log−Likelihood
Iteration  
JointTrain
PwTrain
PreTrainJoint
Unary weights distance-1 edges distance-2 edges Neg. Log-Likelihood


key: chroma:document
string_value:
Iteration  
JointTrain
PwTrain
PreTrainJoint
Unary weights distance-1 edges distance-2 edges Neg. Log-Likelihood
Figure 4: (left) Subset of the learned unary weights. Pairwise weights (middle two panels), the
darker, the larger the weight. (right) Negative log-likelihood for different learning approaches.
3 E XPERIMENTAL EVALUATION
We demonstrate the performance of our model on two tasks: word recognition and image classiﬁcation. We investigate four strategies to learn the model parameters. ‘ Unary only ’ denotes training
only unary classiﬁers while ignoring the structure of the graphical model, i.e., pairwise weights are
equal to 0. ‘JointTrain ’ initializes all weights at random and trains them jointly. ‘ PwTrain ’ uses
piecewise training by ﬁrst training the unary potentials and then keeping them ﬁxed when learning the pairwise potentials. ‘ PreTrainJoint ’ pre-trains the unaries but jointly optimizes pairwise
weights as well as unary weights in a second step.
3.1 W ORD RECOGNITION : W ORD50
Our ﬁrst task consists of word recognition from noisy images. Towards this goal, we created a


key: chroma:document
string_value:
3.1 W ORD RECOGNITION : W ORD50
Our ﬁrst task consists of word recognition from noisy images. Towards this goal, we created a
challenging dataset by randomly selecting 50 words, each consisting of ﬁve characters. We then
generated writing variations of each word as follows: we took the lower case characters from the
Chars74K dataset (de Campos et al., 2009), and inserted them in random background image patches
(similar to Larochelle et al. (2007)) by alpha matting, i.e., characters have transparency. To increase
the difﬁculty, we perturbed each character image of size 2828by scaling, rotation and translation.
As shown in Fig. 3 the task is very challenging, some characters are fairly difﬁcult to recognize even
for humans. We denote the resulting dataset ‘ Word50 .’ The training, validation and test sets have
10;000,2;000and2;000variations of words respectively.
We experimented with graphical models composed of unary and pairwise regions deﬁned over ﬁve


key: chroma:document
string_value:
We experimented with graphical models composed of unary and pairwise regions deﬁned over ﬁve
random variables, one per character. We encode unary potentials fr(x;yi;wu)using multi-layer perceptrons (MLPs) with rectiﬁed linear units (ReLU). Unless otherwise stated, we deﬁne all pairwise
6
Accepted as a workshop contribution at ICLR 2015
H1=128 H1=256 H1=512 H1=768 H1=1024161820222426283032343638
  
Linear
PairH16
PairH32
PairH64
H2=32 H2=64 H2=128 H2=256 H2=512343638404244464850
  
Linear
PairH16
PairH32
PairH64
One-Layer MLP Chain Two-Layer MLP Chain
Figure 5: Learning non-linear pairwise functions: Word recognition as a function of the number
of hidden units for the unary potential. Colors represent different number of hidden units for the
pairwise potentials. The y-axis shows the word accuracy of using Linear function, or 16 (PairH16),
32 (PairH32), and 64 (PairH64) hidden units for the pairwise function.


key: chroma:document
string_value:
32 (PairH32), and 64 (PairH64) hidden units for the pairwise function.
female/indoor/portrait sky/plant life/tree water/animals/sea animals/dog/indoor indoor/ﬂower/plant life
female/indoor/portrait sky/plant life/tree water/animals/sky animals/dog ;
Figure 6: Flickr test set images and some assigned tags as well as our predictions (bottom row).
interactions via
fr(x;yi;yj;wp) =X
mnWmn(yi=m;yj=n); (7)
wherer=fi;jg,wp=fWg,Wmnis the element of matrix W, andrefers to the indicator
function.
For all experiments, we share all unary weights across the nodes of the graphical model as well as all
pairwise weights for all edges. Note that due to the use of ReLU units, the negative log-likelihood
is non-smooth, non-linear and non-convex w.r.t. w. Because of the non-smoothness of F, we utilize
momentum based sub-gradient descent methods to estimate the weights. In particular, we use a


key: chroma:document
string_value:
momentum based sub-gradient descent methods to estimate the weights. In particular, we use a
mini-batch size of 100, a step size of 0:01and a momentum of 0:95. If the unary potential is pretrained, the initial step size is reduced to 0:001. All the unary classiﬁers are trained with 100;000
iterations over mini-batches. For all experiments, the validation set is only used to decrease the
step size, i.e., if the accuracy on the validation set decreases, we reduce the step size by 0:5. We
use= 1, setcr= 1 for all regions r, and perform 10 message passing iterations to compute the
marginal beliefs b(x;y);rat step 2 in Fig. 2 when dealing with loopy models.
We experiment with two graphical models, Markov models of ﬁrst ( i.e., there are links only between
yiandyi+1) and second order ( i.e., there are links between yiandyi+1,yi+2) as well as two types of
unary potentials with varying degree of structure. We report two metrics, the average character and
word accuracy, which correspond to Hamming loss and zero-one loss respectively. Table 1 depicts
the results for the different models, learning strategies and number of hidden units. We observe the


key: chroma:document
string_value:
word accuracy, which correspond to Hamming loss and zero-one loss respectively. Table 1 depicts
the results for the different models, learning strategies and number of hidden units. We observe the
following trends.
Joint training helps: Joint training with pre-trained unary classiﬁers (PreTrainJoint) outperforms
all the other approaches in almost all cases. The piecewise training method (PwTrain), unable to
adapt the non-linearities while learning pairwise weights, often leads to performance worse than
joint training.
Structure helps: Adding structure to the model is key to capture complex dependencies. As shown
in Table 1, more structured models ( i.e., second order Markov model) consistently improves performance.
7
Accepted as a workshop contribution at ICLR 2015
0.00 0.68 0.04 0.06 0.02 0.24 0.03 −0.00 −0.01 0.01 0.04 −0.00 −0.05 −0.01 0.07 −0.01 −0.00 −0.12 0.04 0.01 0.01 0.02 0.04 0.02


key: chroma:document
string_value:
0.68 0.00 0.06 0.06 −0.00 0.36 0.03 −0.08 −0.05 −0.03 0.02 −0.06 −0.12 −0.05 0.74 −0.04 −0.03 −0.21 0.01 −0.03 −0.03 −0.03 0.05 −0.03
0.04 0.06 0.00 0.05 −0.06 0.07 −0.12 −0.07 −0.35 −0.03 −0.46 −0.02 −0.34 0.11 0.02 −0.15 −0.14 −0.01 −0.07 −0.21 0.03 −0.08 0.06 −0.03
0.06 0.06 0.05 0.00 0.10 0.11 0.07 0.09 0.03 0.10 0.01 0.10 0.02 0.09 0.06 0.08 0.07 0.07 0.08 0.06 0.09 0.09 0.08 0.10


key: chroma:document
string_value:
0.02 −0.00 −0.06 0.10 0.00 0.04 0.08 0.05 0.16 0.17 −0.02 0.09 −0.02 0.06 0.03 0.14 0.36 0.06 0.05 0.01 0.08 0.14 0.06 0.10
0.24 0.36 0.07 0.11 0.04 0.00 0.01 0.03 −0.02 0.05 −0.02 0.04 −0.01 0.03 0.12 0.02 0.01 −0.07 0.05 0.05 0.03 0.04 0.07 0.05
0.03 0.03 −0.12 0.07 0.08 0.01 0.00 0.02 0.14 0.07 0.14 0.04 0.05 0.03 0.06 0.08 0.07 −0.03 0.36 0.10 0.04 0.05 0.04 0.07


key: chroma:document
string_value:
−0.00 −0.08 −0.07 0.09 0.05 0.03 0.02 0.00 0.02 0.07 −0.03 0.07 0.34 0.04 −0.04 0.04 0.04 0.02 0.05 0.06 0.06 0.06 0.02 0.07
−0.01 −0.05 −0.35 0.03 0.16 −0.02 0.14 0.02 0.00 0.12 0.22 0.04 0.24 −0.02 −0.00 0.44 0.12 −0.04 0.10 0.30 0.01 0.23 0.05 0.11
0.01 −0.03 −0.03 0.10 0.17 0.05 0.07 0.07 0.12 0.00 −0.00 0.09 0.09 0.07 0.01 0.12 0.26 0.06 0.06 0.10 0.07 0.12 0.07 0.18


key: chroma:document
string_value:
0.04 0.02 −0.46 0.01 −0.02 −0.02 0.14 −0.03 0.22 −0.00 0.00 0.01 0.04 −0.05 0.06 0.08 −0.04 −0.06 0.14 0.09 −0.00 0.06 0.03 0.02
−0.00 −0.06 −0.02 0.10 0.09 0.04 0.04 0.07 0.04 0.09 0.01 0.00 0.04 0.07 −0.01 0.06 0.09 0.26 0.06 0.05 0.07 0.09 0.05 0.09
−0.05 −0.12 −0.34 0.02 −0.02 −0.01 0.05 0.34 0.24 0.09 0.04 0.04 0.00 −0.03 −0.07 0.09 0.01 0.01 0.08 0.68 0.02 0.05 −0.07 0.10


key: chroma:document
string_value:
−0.01 −0.05 0.11 0.09 0.06 0.03 0.03 0.04 −0.02 0.07 −0.05 0.07 −0.03 0.00 −0.01 0.03 0.03 0.03 0.05 0.01 0.06 0.06 0.04 0.07
0.07 0.74 0.02 0.06 0.03 0.12 0.06 −0.04 −0.00 0.01 0.06 −0.01 −0.07 −0.01 0.00 0.00 −0.01 −0.10 0.04 −0.02 0.01 0.00 0.06 0.01
−0.01 −0.04 −0.15 0.08 0.14 0.02 0.08 0.04 0.44 0.12 0.08 0.06 0.09 0.03 0.00 0.00 0.09 −0.00 0.07 0.11 0.05 0.22 −0.01 0.10


key: chroma:document
string_value:
−0.00 −0.03 −0.14 0.07 0.36 0.01 0.07 0.04 0.12 0.26 −0.04 0.09 0.01 0.03 −0.01 0.09 0.00 0.05 0.02 0.03 0.05 0.10 0.03 0.27
−0.12 −0.21 −0.01 0.07 0.06 −0.07 −0.03 0.02 −0.04 0.06 −0.06 0.26 0.01 0.03 −0.10 −0.00 0.05 0.00 0.02 0.00 0.22 0.03 −0.01 0.05
0.04 0.01 −0.07 0.08 0.05 0.05 0.36 0.05 0.10 0.06 0.14 0.06 0.08 0.05 0.04 0.07 0.02 0.02 0.00 0.11 0.06 0.08 0.07 0.06


key: chroma:document
string_value:
0.01 −0.03 −0.21 0.06 0.01 0.05 0.10 0.06 0.30 0.10 0.09 0.05 0.68 0.01 −0.02 0.11 0.03 0.00 0.11 0.00 0.04 0.09 −0.00 0.12
0.01 −0.03 0.03 0.09 0.08 0.03 0.04 0.06 0.01 0.07 −0.00 0.07 0.02 0.06 0.01 0.05 0.05 0.22 0.06 0.04 0.00 0.06 0.05 0.07
0.02 −0.03 −0.08 0.09 0.14 0.04 0.05 0.06 0.23 0.12 0.06 0.09 0.05 0.06 0.00 0.22 0.10 0.03 0.08 0.09 0.06 0.00 0.06 0.10


key: chroma:document
string_value:
0.04 0.05 0.06 0.08 0.06 0.07 0.04 0.02 0.05 0.07 0.03 0.05 −0.07 0.04 0.06 −0.01 0.03 −0.01 0.07 −0.00 0.05 0.06 0.00 0.07
0.02 −0.03 −0.03 0.10 0.10 0.05 0.07 0.07 0.11 0.18 0.02 0.09 0.10 0.07 0.01 0.10 0.27 0.05 0.06 0.12 0.07 0.10 0.07 0.00female
people
indoor
baby
sea
portrait
transport
flower
sky
lake
structures
bird
plant life
food
male
clouds
water
animals
car
tree
dog
sunset
night
river
femalepeopleindoorbabyseaportraittransportflowerskylakestructuresbirdplant lifefoodmalecloudswateranimalscartreedogsunsetnightriver
0 5000 100000246810x 105
Time [s]Neg. Log−Likelihood
  
w/o blend
w blend
0 5000 100000200040006000800010000
Time [s]Training error


key: chroma:document
string_value:
Time [s]Neg. Log−Likelihood
  
w/o blend
w blend
0 5000 100000200040006000800010000
Time [s]Training error
  
w/o blend
w blend
(a) (b)
Figure 7: (a) Correlation matrix ( i.e., pairwise potentials) learned on the Flickr dataset. (b) Blending
learning and inference speeds-up training signiﬁcantly.
Deep helps: We tested our models using one layer and two-layer perceptrons with both shortrange and long-range connections in the MRF. For the two-layer MLP, the number of hidden units
in the ﬁrst layer is ﬁxed to H1= 512 , and we varied the number of hidden units H2in the second
layer. As shown in Table 1 we observe that the deeper and the more structured the model, the better
the performance we achieve. As expected, performance also grows with the number of hidden units.
Efﬁciency: Using GPUs, it takes on average 0.064s per iteration for the 1st order Markov model
and 0.104s for the 2nd order Markov model. The time employed for training the one layer vs. the


key: chroma:document
string_value:
and 0.104s for the 2nd order Markov model. The time employed for training the one layer vs. the
multi-layer models is approximately the same. Note that our approach is very efﬁcient, as this is the
time per iteration to train 831,166 weights.
Learned parameters: As shown in the left column of Fig. 4, the learned unary weights resemble
character strokes. The middle two panels show the learned pairwise weights for distance-1 edges
(i.e., edges with only neighboring connections) and distance-2 edges ( i.e., edges connecting every
other variable). For example, it shows that ‘q’ is likely to be followed by ‘u,’ and ‘e’ is likely to
be distance-2 away from ‘q’ in this dataset. On the right-most panel, we also show the negative
log-likelihood as a function of the number of joint training iterations. PreTrainJoint can achieve the
lowest cost value, while PwTrain has the highest value.
Non-linear pairwise functions: To further demonstrate the generality of our approach, we replaced
the linear pairwise function in Eq. (7) by a one-layer MLP, while keeping the other settings identical.


key: chroma:document
string_value:
the linear pairwise function in Eq. (7) by a one-layer MLP, while keeping the other settings identical.
For this experiment we utilize a 1st order Markov model. As shown in Fig. 5, our model attains best
performance when using a non-linear pairwise function. We found 16 to 64 hidden units for the
non-linear pairwise function to be sufﬁcient for modeling the bi-gram combinations in this dataset.
In this case the largest model has 974,846 weights and training takes on average 0.068s per iteration.
3.2 I MAGE CLASSIFICATION : FLICKR
We next evaluate the importance of blending learning and inference. Towards this goal, we make
use of the Flickr dataset, which consists of 10;000training and 10;000test images from Flickr. The
task is to predict which of 38 possible tags should be assigned to each image. Fig. 6 shows some
examples. The graphical model has 38 binary random variables, each denoting the presence/absence
of a particular tag. We deﬁne the non-linear unaries fr(x;yi;wu)using the 8-layer deep-net architecture from Krizhevsky et al. (2013) followed by a 76-dimensional top layer. Hence the function


key: chroma:document
string_value:
is composed out of two subsequent stacks of convolution, rectiﬁed linear (ReLU), pooling and local
response normalization units. Those are followed by three convolution–ReLU function pairs. Afterwards pooling is applied before two fully-connected–ReLU–dropout combinations are employed to
yield the input into a fully connected layer which ﬁnally computes the unary potentials. We employ
pairwise potentials similar to Eq. (7) which now fully model the correlations between any pair of
8
Accepted as a workshop contribution at ICLR 2015
output variables. This amounts to a total of 57;182;408parameters arising from the convolutional
units, fully connected units and corresponding biases as well as the pairwise weights.
We use a momentum based sub-gradient method for training with a mini-batch size of 300, a step
size of 0:0001 , a momentum of 0:95and set= 1 andcr= 18r. We initialized the deep-net
parameters using a model pre-trained on ImageNet (Deng et al., 2009). Our error metric is the
classiﬁcation error ( i.e., Hamming loss).
Joint training helps: The mean error for unary only potentials (‘Unary only’), piecewise training


key: chroma:document
string_value:
Joint training helps: The mean error for unary only potentials (‘Unary only’), piecewise training
(‘PwTrain’) and joint pretraining (‘PreTrainJoint’) is 9.36%, 7.70% and 7.25% respectively. Similar
to the Word50 dataset we observe that joint training is beneﬁcial. We provide examples for perfect
(two left-most images), roughly accurate and failing predictions (right image) in Fig. 6.
Learned pairwise weights: In Fig. 7(a) we illustrate the learned correlations for a subset of the 38
classes. We observe that the class ‘people’ correlates highly with ‘female,’ ‘male,’ and ‘portrait.’
The ‘indoor’ tag does not co-occur with ‘sky,’ ‘structures,’ ‘plant life’ and ‘tree.’ ‘Sea’ appears
typically with ‘water,’ ‘clouds,’ ‘lake’ and ‘sky.’
Efﬁciency of Blending: To illustrate that blending is indeed beneﬁcial we compare the negative


key: chroma:document
string_value:
Efﬁciency of Blending: To illustrate that blending is indeed beneﬁcial we compare the negative
log-likelihood and the training error as a function of run-time in Fig. 7(b). The standard approach
is limited to 20 iterations of message passing to avoid time-consuming, repeated computation of a
stopping criterion involving both the approximated log-partition function and its dual. As show in
Fig. 7(b) blending learning and inference speeds up parameter estimation signiﬁcantly. For larger
graphical models, we expect the differences to be even more signiﬁcant.
4 D ISCUSSION & C ONCLUSION
Joint training of neural networks and graphical models: Neural Networks have been incorporated as unary potentials in graphical models. One of the earliest works by Bridle (1990) jointly optimizes a system consisting of multilayer perceptrons and hidden Markov models for speech recognition. For document processing systems, Bottou et al. (1997) propose Graph Transformer Networks
to jointly optimize sub-tasks, such as word segmentation and character recognition. Several works
(Collobert et al., 2011; Peng et al., 2009; Ma et al., 2012; Do & Artieres, 2010; Prabhavalkar &


key: chroma:document
string_value:
(Collobert et al., 2011; Peng et al., 2009; Ma et al., 2012; Do & Artieres, 2010; Prabhavalkar &
Fosler-Lussier, 2010; Morris & Fosler-Lussier, 2008) have extended the linear unary potential in
MRFs to incorporate non-linearities. However, they assume that exact inference can be performed
either via a forward-backward pass within the graphical model or dynamic programming. In contrast, in this paper we present learning algorithms for general graphical models, where inference is
hard. Moreover, all the previous works (except Do & Artieres (2010)) do not consider max-margin
loss during training which is incorporated into our framework by choosing = 0. More recently, Li
& Zemel (2014) use a hinge loss to learn the unary term deﬁned as a neural net, but keep the pairwise
potentials ﬁxed ( i.e., no joint training). Domke (2013) considers non-linear structured prediction and
decomposes the learning problem into a subset of logistic regressors, which require the parameter
updates to be run till convergence before updating the messages. Tompson et al. (2014) also jointly
train convolutional neural networks and a graphical model for pose estimation. However, the MRF


key: chroma:document
string_value:
train convolutional neural networks and a graphical model for pose estimation. However, the MRF
inference procedure is approximated by their Spatial-Model which ignores the partition function.
Blending learning and inference: In this paper we deﬁned learning to be a min-max task. The
blending strategy, which was previously employed for learning log-linear models by (Meshi et al.,
2010; Hazan & Urtasun, 2010), amounts to converting the maximization task into a minimization
problem using its dual. Subsequently we make use of block-coordinate descent strategies to obtain
a more efﬁcient algorithm. Importantly any order of block-updates is possible. It remains an open
problem to ﬁnd the optimal tradeoff.
We have proposed an efﬁcient algorithm to learn deep models enriched to capture the dependencies
between the output variables. Our experiments on word prediction from noisy images and multiclass image classiﬁcation showed that the deeper and the more structured the model, the better the
performance we achieve. Furthermore, joint learning of all weights outperforms all other strategies. In the future we plan to learn deeper models in applications such as holistic semantic scene
understanding. We will also extend our approach to deal with hidden variables.
9
Accepted as a workshop contribution at ICLR 2015
ACKNOWLEDGMENTS


