# embedding_metadata

key: chroma:document
string_value:
understanding. We will also extend our approach to deal with hidden variables.
9
Accepted as a workshop contribution at ICLR 2015
ACKNOWLEDGMENTS
We thank NVIDIA Corporation for the donation of GPUs used in this research. This work was
partially funded by ONR-N00014-14-1-0232.
REFERENCES
Bengio, Y ., Lamblin, P., Popovici, D., and Larochelle, H. Greedy Layer-Wise Training of Deep Networks. In
Proc. NIPS , 2007.
Bottou, L., Bengio, Y ., and LeCun, Y . Global training of document processing systems using graph transformer
networks. In Proc. CVPR , 1997.
Bridle, J. S. Training stochastic model recognition algorithms as networks can lead to maximum mutual information estimation of parameters. In Proc. NIPS , 1990.
Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglu, K., and Kuksa, P. Natural language processing
(almost) from scratch. JMLR , 2011.
de Campos, T. E., Babu, B. R., and Varma, M. Character recognition in natural images. In Proc. VISAPP , 2009.


key: chroma:document
string_value:
de Campos, T. E., Babu, B. R., and Varma, M. Character recognition in natural images. In Proc. VISAPP , 2009.
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image
database. In Proc. CVPR , 2009.
Deng, J., Ding, N., Jia, Y ., Frome, A., Murphy, K., Bengio, S., Li, Y ., Neven, H., and Adam, H. Large-Scale
Object Classiﬁcation using Label Relation Graphs. In Proc. ECCV , 2014.
Do, T.-M.-T. and Artieres, T. Neural conditional random ﬁelds. In Proc. AISTATS , 2010.
Domke, J. Structured Learning via Logistic Regression. In Proc. NIPS , 2013.
Eigen, D., Rolfe, J., Fergus, R., and LeCun, Y . Understanding Deep Architectures using a Recursive Convolutional Network. In Proc. ICLR , 2014.
Girshick, R., Donahue, J., Darrell, T., and Malik, J. Rich feature hierarchies for accurate object detection and


key: chroma:document
string_value:
Girshick, R., Donahue, J., Darrell, T., and Malik, J. Rich feature hierarchies for accurate object detection and
semantic segmentation. In Proc. CVPR , 2014.
Hariharan, B., Arbel ´aez, P., Girshick, R., and Malik, J. Simultaneous detection and segmentation. In Proc.
ECCV , 2014.
Hazan, T. and Urtasun, R. A Primal-Dual Message-Passing Algorithm for Approximated Large Scale Structured Prediction. In Proc. NIPS , 2010.
Hinton, G. E. and Salakhutdinov, R. R. Reducing the dimensionality of data with neural networks. Science ,
2006.
Hinton, G. E., Sejnowski, T. J., and Ackley, D. H. Boltzmann Machines: Constraint Satisfaction Networks that
Learn. Technical report, University of Toronto, 1984.
Jia, Y . Caffe: An Open Source Convolutional Architecture for Fast Feature Embedding. http://caffe.
berkeleyvision.org/ , 2013.
Koller, D. and Friedman, N. Probabilistic Graphical Models: Principles and Techniques . MIT Press, 2009.


key: chroma:document
string_value:
berkeleyvision.org/ , 2013.
Koller, D. and Friedman, N. Probabilistic Graphical Models: Principles and Techniques . MIT Press, 2009.
Krizhevsky, A., Sutskever, I., and Hinton, G. E. ImageNet Classiﬁcation with Deep Convolutional Neural
Networks. In Proc. NIPS , 2013.
Larochelle, H., Erhan, D., Courville, A., Bergstra, J., and Bengio, Y . An empirical evaluation of deep architectures on problems with many factors of variation. In Proc. ICML , 2007.
LeCun, Y ., Bottou, L., Bengio, Y ., and Haffner, P. Gradient-based learning applied to document recognition.
Proceedings of the IEEE , 1998.
Lee, H., Grosse, R., Ranganath, R., and Ng, A. Y . Convolutional deep belief networks for scalable unsupervised
learning of hierarchical representations. In Proc. ICML , 2009.
Li, Y . and Zemel, R. High Order Regularization for Semi-Supervised Learning of Structured Output Problems.
InProc. ICML , 2014.
10
Accepted as a workshop contribution at ICLR 2015


key: chroma:document
string_value:
InProc. ICML , 2014.
10
Accepted as a workshop contribution at ICLR 2015
Ma, J., Peng, J., Wang, S., and Xu, J. A conditional neural ﬁelds model for protein threading. Bioinformatics ,
2012.
Meltzer, T., Globerson, A., and Weiss, Y . Convergent Message Passing Algorithms: a unifying view. In Proc.
UAI, 2009.
Meshi, O., Sontag, D., Jaakkola, T., and Globerson, A. Learning Efﬁciently with Approximate inference via
Dual Losses. In Proc. ICML , 2010.
Morris, J. and Fosler-Lussier, E. Conditional random ﬁelds for integrating local discriminative classiﬁers. IEEE
Trans. Audio, Speech, and Language Processing , 2008.
Nowozin, S., Rother, C., Bagon, S., Sharp, T., Yao, B., and Kohli, P. Decision tree ﬁelds. In Proc. ICCV , 2011.
Pearl, J. Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference . Morgan Kaufmann,
1988.


key: chroma:document
string_value:
Pearl, J. Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference . Morgan Kaufmann,
1988.
Peng, J., Bo, L., and Xu, J. Conditional Neural Fields. In Proc. NIPS , 2009.
Prabhavalkar, R. and Fosler-Lussier, E. Backpropagation training for multilayer conditional random ﬁeld based
phone recognition. In Proc. ICASSP , 2010.
Salakhutdinov, R. R. and Hinton, G. E. An Efﬁcient Learning Procedure for Deep Boltzmann Machines. Neural
Computation , 2012.
Schwing, A. G. Inference and Learning Algorithms with Applications to 3D Indoor Scene Understanding . PhD
thesis, ETH Zurich, 2013.
Socher, R., Huval, B., Bhat, B., Manning, C. D., and Ng, A. Y . Convolutional-Recursive Deep Learning for 3D
Object Classiﬁcation. In Proc. NIPS , 2012.
Tompson, J., Jain, A., LeCun, Y ., and Bregler, C. Joint Training of a Convolutional Network and a Graphical
Model for Human Pose Estimation. In Proc. NIPS , 2014.


key: chroma:document
string_value:
Model for Human Pose Estimation. In Proc. NIPS , 2014.
Wainwright, M. J. and Jordan, M. I. Graphical Models, Exponential Families and Variational Inference . Foundations and Trends in Machine Learning, 2008.
Wainwright, M. J., Jaakkola, T., and Willsky, A. S. A new class of upper bounds on the log partition function.
Trans. Information Theory , 2005.
Weiss, Y ., Yanover, C., and Meltzer, T. MAP Estimation, Linear Programming and Belief Propagation with
Convex Free Energies. In Proc. UAI , 2007.
Wiegerinck, W. and Heskes, T. Fractional belief propagation. In Proc. NIPS , 2003.
Xu, J., Schwing, A. G., and Urtasun, R. Tell me what you see and I will show you where it is. In Proc. CVPR ,
2014.
Yedidia, J. S., Freeman, W. T., and Weiss, Y . Constructing free-energy approximations and generalized belief
propagation algorithms. Trans. Information Theory , 2005.
Zeiler, M. D. and Fergus, R. Visualizing and Understanding Convolutional Networks. In Proc. ECCV , 2014.
11


key: chroma:document
string_value:
Published as a conference paper at ICLR 2015
NEURAL MACHINE TRANSLATION
BYJOINTLY LEARNING TO ALIGN AND TRANSLATE
Dzmitry Bahdanau
Jacobs University Bremen, Germany
KyungHyun Cho Yoshua Bengio
Universit ´e de Montr ´eal
ABSTRACT
Neural machine translation is a recently proposed approach to machine translation. Unlike the traditional statistical machine translation, the neural machine
translation aims at building a single neural network that can be jointly tuned to
maximize the translation performance. The models proposed recently for neural machine translation often belong to a family of encoder–decoders and encode
a source sentence into a ﬁxed-length vector from which a decoder generates a
translation. In this paper, we conjecture that the use of a ﬁxed-length vector is a
bottleneck in improving the performance of this basic encoder–decoder architecture, and propose to extend this by allowing a model to automatically (soft-)search
for parts of a source sentence that are relevant to predicting a target word, without
having to form these parts as a hard segment explicitly. With this new approach,
we achieve a translation performance comparable to the existing state-of-the-art
phrase-based system on the task of English-to-French translation. Furthermore,


key: chroma:document
string_value:
we achieve a translation performance comparable to the existing state-of-the-art
phrase-based system on the task of English-to-French translation. Furthermore,
qualitative analysis reveals that the (soft-)alignments found by the model agree
well with our intuition.
1 I NTRODUCTION
Neural machine translation is a newly emerging approach to machine translation, recently proposed
by Kalchbrenner and Blunsom (2013), Sutskever et al. (2014) and Cho et al. (2014b). Unlike the
traditional phrase-based translation system (see, e.g., Koehn et al. , 2003) which consists of many
small sub-components that are tuned separately, neural machine translation attempts to build and
train a single, large neural network that reads a sentence and outputs a correct translation.
Most of the proposed neural machine translation models belong to a family of encoder–
decoders (Sutskever et al. , 2014; Cho et al. , 2014a), with an encoder and a decoder for each language, or involve a language-speciﬁc encoder applied to each sentence whose outputs are then compared (Hermann and Blunsom, 2014). An encoder neural network reads and encodes a source sentence into a ﬁxed-length vector. A decoder then outputs a translation from the encoded vector. The


key: chroma:document
string_value:
whole encoder–decoder system, which consists of the encoder and the decoder for a language pair,
is jointly trained to maximize the probability of a correct translation given a source sentence.
A potential issue with this encoder–decoder approach is that a neural network needs to be able to
compress all the necessary information of a source sentence into a ﬁxed-length vector. This may
make it difﬁcult for the neural network to cope with long sentences, especially those that are longer
than the sentences in the training corpus. Cho et al. (2014b) showed that indeed the performance of
a basic encoder–decoder deteriorates rapidly as the length of an input sentence increases.
In order to address this issue, we introduce an extension to the encoder–decoder model which learns
to align and translate jointly. Each time the proposed model generates a word in a translation, it
(soft-)searches for a set of positions in a source sentence where the most relevant information is
concentrated. The model then predicts a target word based on the context vectors associated with
these source positions and all the previous generated target words.
CIFAR Senior Fellow
1arXiv:1409.0473v7  [cs.CL]  19 May 2016
Published as a conference paper at ICLR 2015


key: chroma:document
string_value:
CIFAR Senior Fellow
1arXiv:1409.0473v7  [cs.CL]  19 May 2016
Published as a conference paper at ICLR 2015
The most important distinguishing feature of this approach from the basic encoder–decoder is that
it does not attempt to encode a whole input sentence into a single ﬁxed-length vector. Instead, it encodes the input sentence into a sequence of vectors and chooses a subset of these vectors adaptively
while decoding the translation. This frees a neural translation model from having to squash all the
information of a source sentence, regardless of its length, into a ﬁxed-length vector. We show this
allows a model to cope better with long sentences.
In this paper, we show that the proposed approach of jointly learning to align and translate achieves
signiﬁcantly improved translation performance over the basic encoder–decoder approach. The improvement is more apparent with longer sentences, but can be observed with sentences of any
length. On the task of English-to-French translation, the proposed approach achieves, with a single
model, a translation performance comparable, or close, to the conventional phrase-based system.
Furthermore, qualitative analysis reveals that the proposed model ﬁnds a linguistically plausible
(soft-)alignment between a source sentence and the corresponding target sentence.


key: chroma:document
string_value:
Furthermore, qualitative analysis reveals that the proposed model ﬁnds a linguistically plausible
(soft-)alignment between a source sentence and the corresponding target sentence.
2 B ACKGROUND : NEURAL MACHINE TRANSLATION
From a probabilistic perspective, translation is equivalent to ﬁnding a target sentence ythat maximizes the conditional probability of ygiven a source sentence x, i.e., arg maxyp(yjx). In
neural machine translation, we ﬁt a parameterized model to maximize the conditional probability
of sentence pairs using a parallel training corpus. Once the conditional distribution is learned by a
translation model, given a source sentence a corresponding translation can be generated by searching
for the sentence that maximizes the conditional probability.
Recently, a number of papers have proposed the use of neural networks to directly learn this conditional distribution (see, e.g., Kalchbrenner and Blunsom, 2013; Cho et al. , 2014a; Sutskever et al. ,
2014; Cho et al. , 2014b; Forcada and ˜Neco, 1997). This neural machine translation approach typically consists of two components, the ﬁrst of which encodes a source sentence xand the second
decodes to a target sentence y. For instance, two recurrent neural networks (RNN) were used by


key: chroma:document
string_value:
decodes to a target sentence y. For instance, two recurrent neural networks (RNN) were used by
(Cho et al. , 2014a) and (Sutskever et al. , 2014) to encode a variable-length source sentence into a
ﬁxed-length vector and to decode the vector into a variable-length target sentence.
Despite being a quite new approach, neural machine translation has already shown promising results.
Sutskever et al. (2014) reported that the neural machine translation based on RNNs with long shortterm memory (LSTM) units achieves close to the state-of-the-art performance of the conventional
phrase-based machine translation system on an English-to-French translation task.1Adding neural
components to existing translation systems, for instance, to score the phrase pairs in the phrase
table (Cho et al. , 2014a) or to re-rank candidate translations (Sutskever et al. , 2014), has allowed to
surpass the previous state-of-the-art performance level.
2.1 RNN E NCODER –DECODER
Here, we describe brieﬂy the underlying framework, called RNN Encoder–Decoder , proposed by
Cho et al. (2014a) and Sutskever et al. (2014) upon which we build a novel architecture that learns


key: chroma:document
string_value:
Cho et al. (2014a) and Sutskever et al. (2014) upon which we build a novel architecture that learns
to align and translate simultaneously.
In the Encoder–Decoder framework, an encoder reads the input sentence, a sequence of vectors
x= (x1;;xTx), into a vector c.2The most common approach is to use an RNN such that
ht=f(xt;ht1) (1)
and
c=q(fh1;;hTxg);
whereht2Rnis a hidden state at time t, andcis a vector generated from the sequence of the
hidden states. fandqare some nonlinear functions. Sutskever et al. (2014) used an LSTM as fand
q(fh1;;hTg) =hT, for instance.
1We mean by the state-of-the-art performance, the performance of the conventional phrase-based system
without using any neural network-based component.
2Although most of the previous works (see, e.g., Cho et al. , 2014a; Sutskever et al. , 2014; Kalchbrenner and


key: chroma:document
string_value:
2Although most of the previous works (see, e.g., Cho et al. , 2014a; Sutskever et al. , 2014; Kalchbrenner and
Blunsom, 2013) used to encode a variable-length input sentence into a ﬁxed-length vector, it is not necessary,
and even it may be beneﬁcial to have a variable-length vector, as we will show later.
2
Published as a conference paper at ICLR 2015
The decoder is often trained to predict the next word yt0given the context vector cand all the
previously predicted words fy1;;yt01g. In other words, the decoder deﬁnes a probability over
the translation yby decomposing the joint probability into the ordered conditionals:
p(y) =TY
t=1p(ytjfy1;;yt1g;c); (2)
where y=
y1;;yTy
. With an RNN, each conditional probability is modeled as
p(ytjfy1;;yt1g;c) =g(yt1;st;c); (3)
wheregis a nonlinear, potentially multi-layered, function that outputs the probability of yt, andstis


key: chroma:document
string_value:
wheregis a nonlinear, potentially multi-layered, function that outputs the probability of yt, andstis
the hidden state of the RNN. It should be noted that other architectures such as a hybrid of an RNN
and a de-convolutional neural network can be used (Kalchbrenner and Blunsom, 2013).
3 L EARNING TO ALIGN AND TRANSLATE
In this section, we propose a novel architecture for neural machine translation. The new architecture
consists of a bidirectional RNN as an encoder (Sec. 3.2) and a decoder that emulates searching
through a source sentence during decoding a translation (Sec. 3.1).
3.1 D ECODER : GENERAL DESCRIPTION
x1x2x3xT+
αt,1
αt,2 αt,3αt,Tyt-1yt
h1h2h3 hTh1h2h3 hTst-1st
Figure 1: The graphical illustration of the proposed model
trying to generate the t-th target wordytgiven a source
sentence (x1;x2;:::;x T).In a new model architecture, we deﬁne each conditional probability
in Eq. (2) as:


key: chroma:document
string_value:
sentence (x1;x2;:::;x T).In a new model architecture, we deﬁne each conditional probability
in Eq. (2) as:
p(yijy1;:::;y i1;x) =g(yi1;si;ci); (4)
wheresiis an RNN hidden state for time i, computed by
si=f(si1;yi1;ci):
It should be noted that unlike the existing encoder–decoder approach (see Eq. (2)), here the probability is conditioned on a distinct
context vector cifor each target word yi.
The context vector cidepends on a sequence of annotations
(h1;;hTx)to which an encoder maps the input sentence. Each
annotationhicontains information about the whole input sequence
with a strong focus on the parts surrounding the i-th word of the
input sequence. We explain in detail how the annotations are computed in the next section.
The context vector ciis, then, computed as a weighted sum of these
annotationshi:
ci=TxX
j=1ijhj: (5)
The weightijof each annotation hjis computed by
ij=exp (eij)PTx


key: chroma:document
string_value:
ci=TxX
j=1ijhj: (5)
The weightijof each annotation hjis computed by
ij=exp (eij)PTx
k=1exp (eik); (6)
where
eij=a(si1;hj)
is an alignment model which scores how well the inputs around position jand the output at position
imatch. The score is based on the RNN hidden state si1(just before emitting yi, Eq. (4)) and the
j-th annotation hjof the input sentence.
We parametrize the alignment model aas a feedforward neural network which is jointly trained with
all the other components of the proposed system. Note that unlike in traditional machine translation,
3
Published as a conference paper at ICLR 2015
the alignment is not considered to be a latent variable. Instead, the alignment model directly computes a soft alignment, which allows the gradient of the cost function to be backpropagated through.
This gradient can be used to train the alignment model as well as the whole translation model jointly.
We can understand the approach of taking a weighted sum of all the annotations as computing an
expected annotation , where the expectation is over possible alignments. Let ijbe a probability that


key: chroma:document
string_value:
We can understand the approach of taking a weighted sum of all the annotations as computing an
expected annotation , where the expectation is over possible alignments. Let ijbe a probability that
the target word yiis aligned to, or translated from, a source word xj. Then, thei-th context vector
ciis the expected annotation over all the annotations with probabilities ij.
The probability ij, or its associated energy eij, reﬂects the importance of the annotation hjwith
respect to the previous hidden state si1in deciding the next state siand generating yi. Intuitively,
this implements a mechanism of attention in the decoder. The decoder decides parts of the source
sentence to pay attention to. By letting the decoder have an attention mechanism, we relieve the
encoder from the burden of having to encode all information in the source sentence into a ﬁxedlength vector. With this new approach the information can be spread throughout the sequence of
annotations, which can be selectively retrieved by the decoder accordingly.
3.2 E NCODER : BIDIRECTIONAL RNN FOR ANNOTATING SEQUENCES
The usual RNN, described in Eq. (1), reads an input sequence xin order starting from the ﬁrst


key: chroma:document
string_value:
The usual RNN, described in Eq. (1), reads an input sequence xin order starting from the ﬁrst
symbolx1to the last one xTx. However, in the proposed scheme, we would like the annotation
of each word to summarize not only the preceding words, but also the following words. Hence,
we propose to use a bidirectional RNN (BiRNN, Schuster and Paliwal, 1997), which has been
successfully used recently in speech recognition (see, e.g., Graves et al. , 2013).
A BiRNN consists of forward and backward RNN’s. The forward RNN !freads the input sequence
as it is ordered (from x1toxTx) and calculates a sequence of forward hidden states ( !h1;; !hTx).
The backward RNN freads the sequence in the reverse order (from xTxtox1), resulting in a
sequence of backward hidden states ( h1;; hTx).
We obtain an annotation for each word xjby concatenating the forward hidden state !hjand the
backward one hj, i.e.,hj=h !h>
j; h>
ji>


key: chroma:document
string_value:
backward one hj, i.e.,hj=h !h>
j; h>
ji>
. In this way, the annotation hjcontains the summaries
of both the preceding words and the following words. Due to the tendency of RNNs to better
represent recent inputs, the annotation hjwill be focused on the words around xj. This sequence
of annotations is used by the decoder and the alignment model later to compute the context vector
(Eqs. (5)–(6)).
See Fig. 1 for the graphical illustration of the proposed model.
4 E XPERIMENT SETTINGS
We evaluate the proposed approach on the task of English-to-French translation. We use the bilingual, parallel corpora provided by ACL WMT ’14.3As a comparison, we also report the performance of an RNN Encoder–Decoder which was proposed recently by Cho et al. (2014a). We use
the same training procedures and the same dataset for both models.4
4.1 D ATASET
WMT ’14 contains the following English-French parallel corpora: Europarl (61M words), news
commentary (5.5M), UN (421M) and two crawled corpora of 90M and 272.5M words respectively,


key: chroma:document
string_value:
commentary (5.5M), UN (421M) and two crawled corpora of 90M and 272.5M words respectively,
totaling 850M words. Following the procedure described in Cho et al. (2014a), we reduce the size of
the combined corpus to have 348M words using the data selection method by Axelrod et al. (2011).5
We do not use any monolingual data other than the mentioned parallel corpora, although it may be
possible to use a much larger monolingual corpus to pretrain an encoder. We concatenate news-test3http://www.statmt.org/wmt14/translation-task.html
4Implementations are available at https://github.com/lisa-groundhog/GroundHog .
5Available online at http://www-lium.univ-lemans.fr/ ˜schwenk/cslm_joint_paper/ .
4
Published as a conference paper at ICLR 2015
0 10 20 30 40 50 60
Sentence length051015202530BLEU score RNNsearch-50
RNNsearch-30
RNNenc-50
RNNenc-30
Figure 2: The BLEU scores
of the generated translations
on the test set with respect
to the lengths of the sentences. The results are on


key: chroma:document
string_value:
RNNenc-30
Figure 2: The BLEU scores
of the generated translations
on the test set with respect
to the lengths of the sentences. The results are on
the full test set which includes sentences having unknown words to the models.
2012 and news-test-2013 to make a development (validation) set, and evaluate the models on the test
set (news-test-2014) from WMT ’14, which consists of 3003 sentences not present in the training
data.
After a usual tokenization6, we use a shortlist of 30,000 most frequent words in each language to
train our models. Any word not included in the shortlist is mapped to a special token ( [UNK ]). We
do not apply any other special preprocessing, such as lowercasing or stemming, to the data.
4.2 M ODELS
We train two types of models. The ﬁrst one is an RNN Encoder–Decoder (RNNencdec, Cho et al. ,
2014a), and the other is the proposed model, to which we refer as RNNsearch. We train each model
twice: ﬁrst with the sentences of length up to 30 words (RNNencdec-30, RNNsearch-30) and then


key: chroma:document
string_value:
twice: ﬁrst with the sentences of length up to 30 words (RNNencdec-30, RNNsearch-30) and then
with the sentences of length up to 50 word (RNNencdec-50, RNNsearch-50).
The encoder and decoder of the RNNencdec have 1000 hidden units each.7The encoder of the
RNNsearch consists of forward and backward recurrent neural networks (RNN) each having 1000
hidden units. Its decoder has 1000 hidden units. In both cases, we use a multilayer network with a
single maxout (Goodfellow et al. , 2013) hidden layer to compute the conditional probability of each
target word (Pascanu et al. , 2014).
We use a minibatch stochastic gradient descent (SGD) algorithm together with Adadelta (Zeiler,
2012) to train each model. Each SGD update direction is computed using a minibatch of 80 sentences. We trained each model for approximately 5 days.
Once a model is trained, we use a beam search to ﬁnd a translation that approximately maximizes the
conditional probability (see, e.g., Graves, 2012; Boulanger-Lewandowski et al. , 2013). Sutskever
et al. (2014) used this approach to generate translations from their neural machine translation model.


key: chroma:document
string_value:
et al. (2014) used this approach to generate translations from their neural machine translation model.
For more details on the architectures of the models and training procedure used in the experiments,
see Appendices A and B.
5 R ESULTS
5.1 Q UANTITATIVE RESULTS
In Table 1, we list the translation performances measured in BLEU score. It is clear from the table
that in all the cases, the proposed RNNsearch outperforms the conventional RNNencdec. More
importantly, the performance of the RNNsearch is as high as that of the conventional phrase-based
translation system (Moses), when only the sentences consisting of known words are considered.
This is a signiﬁcant achievement, considering that Moses uses a separate monolingual corpus (418M
words) in addition to the parallel corpora we used to train the RNNsearch and RNNencdec.
6We used the tokenization script from the open-source machine translation package, Moses.
7In this paper, by a ’hidden unit’, we always mean the gated hidden unit (see Appendix A.1.1).
5
Published as a conference paper at ICLR 2015
The
agreement
on
the
European
Economic
Area
was
signed
in
August
1992
.
<end>
L'
accord


key: chroma:document
string_value:
The
agreement
on
the
European
Economic
Area
was
signed
in
August
1992
.
<end>
L'
accord
sur
la
zone
économique
européenne
a
été
signé
en
août
1992
.
<end>
It
should
be
noted
that
the
marine
environment
is
the
least
known
of
environments
.
<end>
Il
convient
de
noter
que
l'
environnement
marin
est
le
moins
connu
de
l'
environnement
.
<end>
(a) (b)
Destruction
of
the
equipment
means
that
Syria
can
no
longer
produce
new
chemical
weapons
.
<end>
La
destruction
de
l'
équipement
signifie
que
la
Syrie
ne
peut
plus
produire
de
nouvelles
armes
chimiques
.
<end>
"
This
will
change
my
future
with
my
family
,
"
the
man
said
.
<end>
"
Cela


key: chroma:document
string_value:
"
This
will
change
my
future
with
my
family
,
"
the
man
said
.
<end>
"
Cela
va
changer
mon
avenir
avec
ma
famille
"
,
a
dit
l'
homme
.
<end>
(c) (d)
Figure 3: Four sample alignments found by RNNsearch-50. The x-axis and y-axis of each plot
correspond to the words in the source sentence (English) and the generated translation (French),
respectively. Each pixel shows the weight ijof the annotation of the j-th source word for the i-th
target word (see Eq. (6)), in grayscale ( 0: black, 1: white). (a) an arbitrary sentence. (b–d) three
randomly selected samples among the sentences without any unknown words and of length between
10 and 20 words from the test set.
One of the motivations behind the proposed approach was the use of a ﬁxed-length context vector
in the basic encoder–decoder approach. We conjectured that this limitation may make the basic


key: chroma:document
string_value:
in the basic encoder–decoder approach. We conjectured that this limitation may make the basic
encoder–decoder approach to underperform with long sentences. In Fig. 2, we see that the performance of RNNencdec dramatically drops as the length of the sentences increases. On the other hand,
both RNNsearch-30 and RNNsearch-50 are more robust to the length of the sentences. RNNsearch50, especially, shows no performance deterioration even with sentences of length 50 or more. This
superiority of the proposed model over the basic encoder–decoder is further conﬁrmed by the fact
that the RNNsearch-30 even outperforms RNNencdec-50 (see Table 1).
6
Published as a conference paper at ICLR 2015
Model All No UNK
RNNencdec-30 13.93 24.19
RNNsearch-30 21.50 31.44
RNNencdec-50 17.82 26.71
RNNsearch-50 26.75 34.16
RNNsearch-50?28.45 36.15
Moses 33.30 35.63Table 1: BLEU scores of the trained models computed on the test set. The second and third columns
show respectively the scores on all the sentences and,


key: chroma:document
string_value:
show respectively the scores on all the sentences and,
on the sentences without any unknown word in themselves and in the reference translations. Note that
RNNsearch-50?was trained much longer until the
performance on the development set stopped improving. () We disallowed the models to generate [UNK]
tokens when only the sentences having no unknown
words were evaluated (last column).
5.2 Q UALITATIVE ANALYSIS
5.2.1 A LIGNMENT
The proposed approach provides an intuitive way to inspect the (soft-)alignment between the words
in a generated translation and those in a source sentence. This is done by visualizing the annotation
weightsijfrom Eq. (6), as in Fig. 3. Each row of a matrix in each plot indicates the weights
associated with the annotations. From this we see which positions in the source sentence were
considered more important when generating the target word.
We can see from the alignments in Fig. 3 that the alignment of words between English and French
is largely monotonic. We see strong weights along the diagonal of each matrix. However, we also
observe a number of non-trivial, non-monotonic alignments. Adjectives and nouns are typically
ordered differently between French and English, and we see an example in Fig. 3 (a). From this


key: chroma:document
string_value:
ordered differently between French and English, and we see an example in Fig. 3 (a). From this
ﬁgure, we see that the model correctly translates a phrase [European Economic Area] into [zone
´economique europ ´een]. The RNNsearch was able to correctly align [zone] with [Area], jumping
over the two words ([European] and [Economic]), and then looked one word back at a time to
complete the whole phrase [zone ´economique europ ´eenne].
The strength of the soft-alignment, opposed to a hard-alignment, is evident, for instance, from
Fig. 3 (d). Consider the source phrase [the man] which was translated into [l’ homme]. Any hard
alignment will map [the] to [l’] and [man] to [homme]. This is not helpful for translation, as one
must consider the word following [the] to determine whether it should be translated into [le], [la],
[les] or [l’]. Our soft-alignment solves this issue naturally by letting the model look at both [the] and
[man], and in this example, we see that the model was able to correctly translate [the] into [l’]. We


key: chroma:document
string_value:
[man], and in this example, we see that the model was able to correctly translate [the] into [l’]. We
observe similar behaviors in all the presented cases in Fig. 3. An additional beneﬁt of the soft alignment is that it naturally deals with source and target phrases of different lengths, without requiring a
counter-intuitive way of mapping some words to or from nowhere ([NULL]) (see, e.g., Chapters 4
and 5 of Koehn, 2010).
5.2.2 L ONG SENTENCES
As clearly visible from Fig. 2 the proposed model (RNNsearch) is much better than the conventional
model (RNNencdec) at translating long sentences. This is likely due to the fact that the RNNsearch
does not require encoding a long sentence into a ﬁxed-length vector perfectly, but only accurately
encoding the parts of the input sentence that surround a particular word.
As an example, consider this source sentence from the test set:
An admitting privilege is the right of a doctor to admit a patient to a hospital or
a medical centre tocarry outadiagnosis oraprocedure, based onhisstatus asa
health care worker atahospital.
The RNNencdec-50 translated this sentence into:


key: chroma:document
string_value:
health care worker atahospital.
The RNNencdec-50 translated this sentence into:
Un privil `ege d’admission est le droit d’un m ´edecin de reconna ˆıtre un patient `a
l’hˆopital ou un centre m ´edical d’un diagnostic oudeprendre undiagnostic en
fonction deson´etatdesant´e.
7
Published as a conference paper at ICLR 2015
The RNNencdec-50 correctly translated the source sentence until [a medical center]. However, from
there on (underlined), it deviated from the original meaning of the source sentence. For instance, it
replaced [based on his status as a health care worker at a hospital] in the source sentence with [en
fonction de son ´etat de sant ´e] (“based on his state of health”).
On the other hand, the RNNsearch-50 generated the following correct translation, preserving the
whole meaning of the input sentence without omitting any details:
Un privil `ege d’admission est le droit d’un m ´edecin d’admettre un patient `a un


key: chroma:document
string_value:
Un privil `ege d’admission est le droit d’un m ´edecin d’admettre un patient `a un
hˆopital ou un centre m ´edical pour effectuer undiagnostic ouuneproc´edure, selon
sonstatut detravailleur dessoins desant´e`al’hˆopital.
Let us consider another sentence from the test set:
This kind of experience is part of Disney’s efforts to ”extend the lifetime of its
series and build new relationships with audiences viadigital platforms that are
becoming ever more important, ” headded.
The translation by the RNNencdec-50 is
Ce type d’exp ´erience fait partie des initiatives du Disney pour ”prolonger la dur ´ee
de vie de ses nouvelles et de d ´evelopper des liens avec les lecteurs num´eriques qui
deviennent plus complexes.
As with the previous example, the RNNencdec began deviating from the actual meaning of the
source sentence after generating approximately 30 words (see the underlined phrase). After that


key: chroma:document
string_value:
As with the previous example, the RNNencdec began deviating from the actual meaning of the
source sentence after generating approximately 30 words (see the underlined phrase). After that
point, the quality of the translation deteriorates, with basic mistakes such as the lack of a closing
quotation mark.
Again, the RNNsearch-50 was able to translate this long sentence correctly:
Ce genre d’exp ´erience fait partie des efforts de Disney pour ”prolonger la dur ´ee
de vie de ses s ´eries et cr ´eer de nouvelles relations avec des publics viades
plateformes num´eriques deplus enplus importantes”, a-t-il ajout ´e.
In conjunction with the quantitative results presented already, these qualitative observations conﬁrm our hypotheses that the RNNsearch architecture enables far more reliable translation of long
sentences than the standard RNNencdec model.
In Appendix C, we provide a few more sample translations of long source sentences generated by
the RNNencdec-50, RNNsearch-50 and Google Translate along with the reference translations.
6 R ELATED WORK
6.1 L EARNING TO ALIGN
A similar approach of aligning an output symbol with an input symbol was proposed recently by


key: chroma:document
string_value:
6 R ELATED WORK
6.1 L EARNING TO ALIGN
A similar approach of aligning an output symbol with an input symbol was proposed recently by
Graves (2013) in the context of handwriting synthesis. Handwriting synthesis is a task where the
model is asked to generate handwriting of a given sequence of characters. In his work, he used a
mixture of Gaussian kernels to compute the weights of the annotations, where the location, width
and mixture coefﬁcient of each kernel was predicted from an alignment model. More speciﬁcally,
his alignment was restricted to predict the location such that the location increases monotonically.
The main difference from our approach is that, in (Graves, 2013), the modes of the weights of the
annotations only move in one direction. In the context of machine translation, this is a severe limitation, as (long-distance) reordering is often needed to generate a grammatically correct translation
(for instance, English-to-German).
Our approach, on the other hand, requires computing the annotation weight of every word in the
source sentence for each word in the translation. This drawback is not severe with the task of
translation in which most of input and output sentences are only 15–40 words. However, this may
limit the applicability of the proposed scheme to other tasks.
8


key: chroma:document
string_value:
translation in which most of input and output sentences are only 15–40 words. However, this may
limit the applicability of the proposed scheme to other tasks.
8
Published as a conference paper at ICLR 2015
6.2 N EURAL NETWORKS FOR MACHINE TRANSLATION
Since Bengio et al. (2003) introduced a neural probabilistic language model which uses a neural network to model the conditional probability of a word given a ﬁxed number of the preceding words,
neural networks have widely been used in machine translation. However, the role of neural networks has been largely limited to simply providing a single feature to an existing statistical machine
translation system or to re-rank a list of candidate translations provided by an existing system.
For instance, Schwenk (2012) proposed using a feedforward neural network to compute the score of
a pair of source and target phrases and to use the score as an additional feature in the phrase-based
statistical machine translation system. More recently, Kalchbrenner and Blunsom (2013) and Devlin
et al. (2014) reported the successful use of the neural networks as a sub-component of the existing
translation system. Traditionally, a neural network trained as a target-side language model has been
used to rescore or rerank a list of candidate translations (see, e.g., Schwenk et al. , 2006).


key: chroma:document
string_value:
used to rescore or rerank a list of candidate translations (see, e.g., Schwenk et al. , 2006).
Although the above approaches were shown to improve the translation performance over the stateof-the-art machine translation systems, we are more interested in a more ambitious objective of
designing a completely new translation system based on neural networks. The neural machine translation approach we consider in this paper is therefore a radical departure from these earlier works.
Rather than using a neural network as a part of the existing system, our model works on its own and
generates a translation from a source sentence directly.
7 C ONCLUSION
The conventional approach to neural machine translation, called an encoder–decoder approach, encodes a whole input sentence into a ﬁxed-length vector from which a translation will be decoded.
We conjectured that the use of a ﬁxed-length context vector is problematic for translating long sentences, based on a recent empirical study reported by Cho et al. (2014b) and Pouget-Abadie et al.
(2014).
In this paper, we proposed a novel architecture that addresses this issue. We extended the basic
encoder–decoder by letting a model (soft-)search for a set of input words, or their annotations computed by an encoder, when generating each target word. This frees the model from having to encode


key: chroma:document
string_value:
a whole source sentence into a ﬁxed-length vector, and also lets the model focus only on information
relevant to the generation of the next target word. This has a major positive impact on the ability
of the neural machine translation system to yield good results on longer sentences. Unlike with
the traditional machine translation systems, all of the pieces of the translation system, including
the alignment mechanism, are jointly trained towards a better log-probability of producing correct
translations.
We tested the proposed model, called RNNsearch, on the task of English-to-French translation. The
experiment revealed that the proposed RNNsearch outperforms the conventional encoder–decoder
model (RNNencdec) signiﬁcantly, regardless of the sentence length and that it is much more robust to the length of a source sentence. From the qualitative analysis where we investigated the
(soft-)alignment generated by the RNNsearch, we were able to conclude that the model can correctly align each target word with the relevant words, or their annotations, in the source sentence as
it generated a correct translation.
Perhaps more importantly, the proposed approach achieved a translation performance comparable to
the existing phrase-based statistical machine translation. It is a striking result, considering that the
proposed architecture, or the whole family of neural machine translation, has only been proposed


key: chroma:document
string_value:
the existing phrase-based statistical machine translation. It is a striking result, considering that the
proposed architecture, or the whole family of neural machine translation, has only been proposed
as recently as this year. We believe the architecture proposed here is a promising step toward better
machine translation and a better understanding of natural languages in general.
One of challenges left for the future is to better handle unknown, or rare words. This will be required
for the model to be more widely used and to match the performance of current state-of-the-art
machine translation systems in all contexts.
9
Published as a conference paper at ICLR 2015
ACKNOWLEDGMENTS
The authors would like to thank the developers of Theano (Bergstra et al. , 2010; Bastien et al. ,
2012). We acknowledge the support of the following agencies for research funding and computing
support: NSERC, Calcul Qu ´ebec, Compute Canada, the Canada Research Chairs and CIFAR. Bahdanau thanks the support from Planet Intelligent Systems GmbH. We also thank Felix Hill, Bart van
Merri ´enboer, Jean Pouget-Abadie, Coline Devin and Tae-Ho Kim.
REFERENCES
Axelrod, A., He, X., and Gao, J. (2011). Domain adaptation via pseudo in-domain data selection.


key: chroma:document
string_value:
REFERENCES
Axelrod, A., He, X., and Gao, J. (2011). Domain adaptation via pseudo in-domain data selection.
InProceedings of the ACL Conference on Empirical Methods in Natural Language Processing
(EMNLP) , pages 355–362. Association for Computational Linguistics.
Bastien, F., Lamblin, P., Pascanu, R., Bergstra, J., Goodfellow, I. J., Bergeron, A., Bouchard, N.,
and Bengio, Y . (2012). Theano: new features and speed improvements. Deep Learning and
Unsupervised Feature Learning NIPS 2012 Workshop.
Bengio, Y ., Simard, P., and Frasconi, P. (1994). Learning long-term dependencies with gradient
descent is difﬁcult. IEEE Transactions on Neural Networks ,5(2), 157–166.
Bengio, Y ., Ducharme, R., Vincent, P., and Janvin, C. (2003). A neural probabilistic language model.
J. Mach. Learn. Res. ,3, 1137–1155.


key: chroma:document
string_value:
J. Mach. Learn. Res. ,3, 1137–1155.
Bergstra, J., Breuleux, O., Bastien, F., Lamblin, P., Pascanu, R., Desjardins, G., Turian, J., WardeFarley, D., and Bengio, Y . (2010). Theano: a CPU and GPU math expression compiler. In
Proceedings of the Python for Scientiﬁc Computing Conference (SciPy) . Oral Presentation.
Boulanger-Lewandowski, N., Bengio, Y ., and Vincent, P. (2013). Audio chord recognition with
recurrent neural networks. In ISMIR .
Cho, K., van Merrienboer, B., Gulcehre, C., Bougares, F., Schwenk, H., and Bengio, Y . (2014a).
Learning phrase representations using RNN encoder-decoder for statistical machine translation.
InProceedings of the Empiricial Methods in Natural Language Processing (EMNLP 2014) . to
appear.
Cho, K., van Merri ¨enboer, B., Bahdanau, D., and Bengio, Y . (2014b). On the properties of neural
machine translation: Encoder–Decoder approaches. In Eighth Workshop on Syntax, Semantics


key: chroma:document
string_value:
machine translation: Encoder–Decoder approaches. In Eighth Workshop on Syntax, Semantics
and Structure in Statistical Translation . to appear.
Devlin, J., Zbib, R., Huang, Z., Lamar, T., Schwartz, R., and Makhoul, J. (2014). Fast and robust
neural network joint models for statistical machine translation. In Association for Computational
Linguistics .
Forcada, M. L. and ˜Neco, R. P. (1997). Recursive hetero-associative memories for translation. In
J. Mira, R. Moreno-D ´ıaz, and J. Cabestany, editors, Biological and Artiﬁcial Computation: From
Neuroscience to Technology , volume 1240 of Lecture Notes in Computer Science , pages 453–462.
Springer Berlin Heidelberg.
Goodfellow, I., Warde-Farley, D., Mirza, M., Courville, A., and Bengio, Y . (2013). Maxout networks. In Proceedings of The 30th International Conference on Machine Learning , pages 1319–
1327.
Graves, A. (2012). Sequence transduction with recurrent neural networks. In Proceedings of the
29th International Conference on Machine Learning (ICML 2012) .


key: chroma:document
string_value:
1327.
Graves, A. (2012). Sequence transduction with recurrent neural networks. In Proceedings of the
29th International Conference on Machine Learning (ICML 2012) .
Graves, A. (2013). Generating sequences with recurrent neural networks. arXiv: 1308.0850
[cs.NE] .
Graves, A., Jaitly, N., and Mohamed, A.-R. (2013). Hybrid speech recognition with deep bidirectional LSTM. In Automatic Speech Recognition and Understanding (ASRU), 2013 IEEE Workshop on , pages 273–278.
10
Published as a conference paper at ICLR 2015
Hermann, K. and Blunsom, P. (2014). Multilingual distributed representations without word alignment. In Proceedings of the Second International Conference on Learning Representations (ICLR
2014) .
Hochreiter, S. (1991). Untersuchungen zu dynamischen neuronalen Netzen. Diploma thesis, Institut
f¨ur Informatik, Lehrstuhl Prof. Brauer, Technische Universit ¨at M ¨unchen.
Hochreiter, S. and Schmidhuber, J. (1997). Long short-term memory. Neural Computation ,9(8),
1735–1780.


key: chroma:document
string_value:
Hochreiter, S. and Schmidhuber, J. (1997). Long short-term memory. Neural Computation ,9(8),
1735–1780.
Kalchbrenner, N. and Blunsom, P. (2013). Recurrent continuous translation models. In Proceedings
of the ACL Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages
1700–1709. Association for Computational Linguistics.
Koehn, P. (2010). Statistical Machine Translation . Cambridge University Press, New York, NY ,
USA.
Koehn, P., Och, F. J., and Marcu, D. (2003). Statistical phrase-based translation. In Proceedings
of the 2003 Conference of the North American Chapter of the Association for Computational
Linguistics on Human Language Technology - Volume 1 , NAACL ’03, pages 48–54, Stroudsburg,
PA, USA. Association for Computational Linguistics.
Pascanu, R., Mikolov, T., and Bengio, Y . (2013a). On the difﬁculty of training recurrent neural
networks. In ICML’2013 .
Pascanu, R., Mikolov, T., and Bengio, Y . (2013b). On the difﬁculty of training recurrent neural


key: chroma:document
string_value:
Pascanu, R., Mikolov, T., and Bengio, Y . (2013b). On the difﬁculty of training recurrent neural
networks. In Proceedings of the 30th International Conference on Machine Learning (ICML
2013) .
Pascanu, R., Gulcehre, C., Cho, K., and Bengio, Y . (2014). How to construct deep recurrent neural
networks. In Proceedings of the Second International Conference on Learning Representations
(ICLR 2014) .
Pouget-Abadie, J., Bahdanau, D., van Merri ¨enboer, B., Cho, K., and Bengio, Y . (2014). Overcoming
the curse of sentence length for neural machine translation using automatic segmentation. In
Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation . to appear.
Schuster, M. and Paliwal, K. K. (1997). Bidirectional recurrent neural networks. Signal Processing,
IEEE Transactions on ,45(11), 2673–2681.
Schwenk, H. (2012). Continuous space translation models for phrase-based statistical machine
translation. In M. Kay and C. Boitet, editors, Proceedings of the 24th International Conference on


key: chroma:document
string_value:
translation. In M. Kay and C. Boitet, editors, Proceedings of the 24th International Conference on
Computational Linguistics (COLIN) , pages 1071–1080. Indian Institute of Technology Bombay.
Schwenk, H., Dchelotte, D., and Gauvain, J.-L. (2006). Continuous space language models for
statistical machine translation. In Proceedings of the COLING/ACL on Main conference poster
sessions , pages 723–730. Association for Computational Linguistics.
Sutskever, I., Vinyals, O., and Le, Q. (2014). Sequence to sequence learning with neural networks.
InAdvances in Neural Information Processing Systems (NIPS 2014) .
Zeiler, M. D. (2012). ADADELTA: An adaptive learning rate method. arXiv: 1212.5701
[cs.LG] .
11
Published as a conference paper at ICLR 2015
A M ODEL ARCHITECTURE
A.1 A RCHITECTURAL CHOICES
The proposed scheme in Section 3 is a general framework where one can freely deﬁne, for instance,
the activation functions fof recurrent neural networks (RNN) and the alignment model a. Here, we
describe the choices we made for the experiments in this paper.


key: chroma:document
string_value:
the activation functions fof recurrent neural networks (RNN) and the alignment model a. Here, we
describe the choices we made for the experiments in this paper.
A.1.1 R ECURRENT NEURAL NETWORK
For the activation function fof an RNN, we use the gated hidden unit recently proposed by Cho
et al. (2014a). The gated hidden unit is an alternative to the conventional simple units such as an
element-wise tanh . This gated unit is similar to a long short-term memory (LSTM) unit proposed
earlier by Hochreiter and Schmidhuber (1997), sharing with it the ability to better model and learn
long-term dependencies. This is made possible by having computation paths in the unfolded RNN
for which the product of derivatives is close to 1. These paths allow gradients to ﬂow backward
easily without suffering too much from the vanishing effect (Hochreiter, 1991; Bengio et al. , 1994;
Pascanu et al. , 2013a). It is therefore possible to use LSTM units instead of the gated hidden unit
described here, as was done in a similar context by Sutskever et al. (2014).
The new state siof the RNN employing ngated hidden units8is computed by


key: chroma:document
string_value:
described here, as was done in a similar context by Sutskever et al. (2014).
The new state siof the RNN employing ngated hidden units8is computed by
si=f(si1;yi1;ci) = (1zi)si1+zi~si;
whereis an element-wise multiplication, and ziis the output of the update gates (see below). The
proposed updated state ~siis computed by
~si= tanh (We(yi1) +U[risi1] +Cci);
wheree(yi1)2Rmis anm-dimensional embedding of a word yi1, andriis the output of the
reset gates (see below). When yiis represented as a 1-of-Kvector,e(yi)is simply a column of an
embedding matrix E2RmK. Whenever possible, we omit bias terms to make the equations less
cluttered.
The update gates ziallow each hidden unit to maintain its previous activation, and the reset gates ri
control how much and what information from the previous state should be reset. We compute them
by
zi=(Wze(yi1) +Uzsi1+Czci);


key: chroma:document
string_value:
by
zi=(Wze(yi1) +Uzsi1+Czci);
ri=(Wre(yi1) +Ursi1+Crci);
where()is a logistic sigmoid function.
At each step of the decoder, we compute the output probability (Eq. (4)) as a multi-layered function (Pascanu et al. , 2014). We use a single hidden layer of maxout units (Goodfellow et al. , 2013)
and normalize the output probabilities (one for each word) with a softmax function (see Eq. (6)).
A.1.2 A LIGNMENT MODEL
The alignment model should be designed considering that the model needs to be evaluated TxTy
times for each sentence pair of lengths TxandTy. In order to reduce computation, we use a singlelayer multilayer perceptron such that
a(si1;hj) =v>
atanh (Wasi1+Uahj);
whereWa2Rnn;Ua2Rn2nandva2Rnare the weight matrices. Since Uahjdoes not
depend oni, we can pre-compute it in advance to minimize the computational cost.


key: chroma:document
string_value:
depend oni, we can pre-compute it in advance to minimize the computational cost.
8Here, we show the formula of the decoder. The same formula can be used in the encoder by simply
ignoring the context vector ciand the related terms.
12
Published as a conference paper at ICLR 2015
A.2 D ETAILED DESCRIPTION OF THE MODEL
A.2.1 E NCODER
In this section, we describe in detail the architecture of the proposed model (RNNsearch) used in the
experiments (see Sec. 4–5). From here on, we omit all bias terms in order to increase readability.
The model takes a source sentence of 1-of-K coded word vectors as input
x= (x1;:::;x Tx); xi2RKx
and outputs a translated sentence of 1-of-K coded word vectors
y= (y1;:::;y Ty); yi2RKy;
whereKxandKyare the vocabulary sizes of source and target languages, respectively. TxandTy
respectively denote the lengths of source and target sentences.
First, the forward states of the bidirectional recurrent neural network (BiRNN) are computed:
 !hi=(


key: chroma:document
string_value:
respectively denote the lengths of source and target sentences.
First, the forward states of the bidirectional recurrent neural network (BiRNN) are computed:
 !hi=(
(1 !zi) !hi1+ !zi !hi, ifi>0
0 , ifi= 0
where
 !hi= tanh !WExi+ !Uh !ri !hi1i
 !zi= !WzExi+ !Uz !hi1
 !ri= !WrExi+ !Ur !hi1
:
E2RmKxis the word embedding matrix. !W; !Wz; !Wr2Rnm, !U; !Uz; !Ur2Rnnare
weight matrices. mandnare the word embedding dimensionality and the number of hidden units,
respectively. ()is as usual a logistic sigmoid function.
The backward states ( h1;; hTx)are computed similarly. We share the word embedding matrix
Ebetween the forward and backward RNNs, unlike the weight matrices.


key: chroma:document
string_value:
Ebetween the forward and backward RNNs, unlike the weight matrices.
We concatenate the forward and backward states to to obtain the annotations (h1;h2;;hTx),
where
hi=" !hi hi#
(7)
A.2.2 D ECODER
The hidden state siof the decoder given the annotations from the encoder is computed by
si=(1zi)si1+zi~si;
where
~si= tanh (WEy i1+U[risi1] +Cci)
zi=(WzEyi1+Uzsi1+Czci)
ri=(WrEyi1+Ursi1+Crci)
Eis the word embedding matrix for the target language. W;W z;Wr2Rnm,U;Uz;Ur2Rnn,
andC;C z;Cr2Rn2nare weights. Again, mandnare the word embedding dimensionality
and the number of hidden units, respectively. The initial hidden state s0is computed by s0=
tanh
Ws h1
;whereWs2Rnn.


key: chroma:document
string_value:
tanh
Ws h1
;whereWs2Rnn.
The context vector ciare recomputed at each step by the alignment model:
ci=TxX
j=1ijhj;
13
Published as a conference paper at ICLR 2015
Model Updates (105)Epochs Hours GPU Train NLL Dev. NLL
RNNenc-30 8.46 6.4 109 TITAN BLACK 28.1 53.0
RNNenc-50 6.00 4.5 108 Quadro K-6000 44.0 43.6
RNNsearch-30 4.71 3.6 113 TITAN BLACK 26.7 47.2
RNNsearch-50 2.88 2.2 111 Quadro K-6000 40.7 38.1
RNNsearch-50?6.67 5.0 252 Quadro K-6000 36.7 35.2
Table 2: Learning statistics and relevant information. Each update corresponds to updating the
parameters once using a single minibatch. One epoch is one pass through the training set. NLL is
the average conditional log-probabilities of the sentences in either the training set or the development
set. Note that the lengths of the sentences differ.
where
ij=exp (eij)PTx
k=1exp (eik)
eij=v>


key: chroma:document
string_value:
set. Note that the lengths of the sentences differ.
where
ij=exp (eij)PTx
k=1exp (eik)
eij=v>
atanh (Wasi1+Uahj);
andhjis thej-th annotation in the source sentence (see Eq. (7)). va2Rn0;Wa2Rn0nand
Ua2Rn02nare weight matrices. Note that the model becomes RNN Encoder–Decoder (Cho
et al. , 2014a), if we ﬁx cito !hTx.
With the decoder state si1, the context ciand the last generated word yi1, we deﬁne the probability
of a target word yias
p(yijsi;yi1;ci)/exp
y>
iWoti
;
where
ti=
max~ti;2j1;~ti;2j	>
j=1;:::;l
and~ti;kis thek-th element of a vector ~tiwhich is computed by
~ti=Uosi1+VoEyi1+Coci:


key: chroma:document
string_value:
and~ti;kis thek-th element of a vector ~tiwhich is computed by
~ti=Uosi1+VoEyi1+Coci:
Wo2RKyl,Uo2R2ln,Vo2R2lmandCo2R2l2nare weight matrices. This can be understood as having a deep output (Pascanu et al. , 2014) with a single maxout hidden layer (Goodfellow
et al. , 2013).
A.2.3 M ODEL SIZE
For all the models used in this paper, the size of a hidden layer nis 1000, the word embedding
dimensionality mis 620 and the size of the maxout hidden layer in the deep output lis 500. The
number of hidden units in the alignment model n0is 1000.
B T RAINING PROCEDURE
B.1 P ARAMETER INITIALIZATION
We initialized the recurrent weight matrices U;Uz;Ur; U; Uz; Ur; !U; !Uzand !Uras random orthogonal matrices. For WaandUa, we initialized them by sampling each element from the Gaussian


key: chroma:document
string_value:
distribution of mean 0and variance 0:0012. All the elements of Vaand all the bias vectors were initialized to zero. Any other weight matrix was initialized by sampling from the Gaussian distribution
of mean 0and variance 0:012.
B.2 T RAINING
We used the stochastic gradient descent (SGD) algorithm. Adadelta (Zeiler, 2012) was used to
automatically adapt the learning rate of each parameter ( = 106and= 0:95). We explicitly
14
Published as a conference paper at ICLR 2015
normalized the L2-norm of the gradient of the cost function each time to be at most a predeﬁned
threshold of 1, when the norm was larger than the threshold (Pascanu et al. , 2013b). Each SGD
update direction was computed with a minibatch of 80 sentences.
At each update our implementation requires time proportional to the length of the longest sentence in
a minibatch. Hence, to minimize the waste of computation, before every 20-th update, we retrieved
1600 sentence pairs, sorted them according to the lengths and split them into 20 minibatches. The
training data was shufﬂed once before training and was traversed sequentially in this manner.
In Tables 2 we present the statistics related to training all the models used in the experiments.


key: chroma:document
string_value:
In Tables 2 we present the statistics related to training all the models used in the experiments.
C T RANSLATIONS OF LONG SENTENCES
Source An admitting privilege is the right of a doctor to admit a patient to a hospital or a medical centre
to carry out a diagnosis or a procedure, based on his status as a health care worker at a hospital.
Reference Le privil `ege d’admission est le droit d’un m ´edecin, en vertu de son statut de membre soignant
d’un h ˆopital, d’admettre un patient dans un h ˆopital ou un centre m ´edical aﬁn d’y d ´elivrer un
diagnostic ou un traitement.
RNNenc-50 Un privil `ege d’admission est le droit d’un m ´edecin de reconna ˆıtre un patient `a l’h ˆopital ou un
centre m ´edical d’un diagnostic ou de prendre un diagnostic en fonction de son ´etat de sant ´e.


key: chroma:document
string_value:
centre m ´edical d’un diagnostic ou de prendre un diagnostic en fonction de son ´etat de sant ´e.
RNNsearch-50 Un privil `ege d’admission est le droit d’un m ´edecin d’admettre un patient `a un h ˆopital ou un
centre m ´edical pour effectuer un diagnostic ou une proc ´edure, selon son statut de travailleur des
soins de sant ´e`a l’h ˆopital.
Google
TranslateUn privil `ege admettre est le droit d’un m ´edecin d’admettre un patient dans un h ˆopital ou un
centre m ´edical pour effectuer un diagnostic ou une proc ´edure, fond ´ee sur sa situation en tant
que travailleur de soins de sant ´e dans un h ˆopital.
Source This kind of experience is part of Disney’s efforts to ”extend the lifetime of its series and build
new relationships with audiences via digital platforms that are becoming ever more important,”
he added.


key: chroma:document
string_value:
new relationships with audiences via digital platforms that are becoming ever more important,”
he added.
Reference Ce type d’exp ´erience entre dans le cadre des efforts de Disney pour ” ´etendre la dur ´ee de
vie de ses s ´eries et construire de nouvelles relations avec son public gr ˆace`a des plateformes
num´eriques qui sont de plus en plus importantes”, a-t-il ajout ´e.
RNNenc-50 Ce type d’exp ´erience fait partie des initiatives du Disney pour ”prolonger la dur ´ee de vie de
ses nouvelles et de d ´evelopper des liens avec les lecteurs num ´eriques qui deviennent plus complexes.
RNNsearch-50 Ce genre d’exp ´erience fait partie des efforts de Disney pour ”prolonger la dur ´ee de vie de ses
s´eries et cr ´eer de nouvelles relations avec des publics via des plateformes num ´eriques de plus
en plus importantes”, a-t-il ajout ´e.
Google


key: chroma:document
string_value:
en plus importantes”, a-t-il ajout ´e.
Google
TranslateCe genre d’exp ´erience fait partie des efforts de Disney `a “´etendre la dur ´ee de vie de sa s ´erie et
construire de nouvelles relations avec le public par le biais des plates-formes num ´eriques qui
deviennent de plus en plus important”, at-il ajout ´e.
Source In a press conference on Thursday, Mr Blair stated that there was nothing in this video that might
constitute a ”reasonable motive” that could lead to criminal charges being brought against the
mayor.
Reference En conf ´erence de presse, jeudi, M. Blair a afﬁrm ´e qu’il n’y avait rien dans cette vid ´eo qui puisse
constituer des ”motifs raisonnables” pouvant mener au d ´epˆot d’une accusation criminelle contre
le maire.


key: chroma:document
string_value:
le maire.
RNNenc-50 Lors de la conf ´erence de presse de jeudi, M. Blair a dit qu’il n’y avait rien dans cette vid ´eo qui
pourrait constituer une ”motivation raisonnable” pouvant entra ˆıner des accusations criminelles
port´ees contre le maire.
RNNsearch-50 Lors d’une conf ´erence de presse jeudi, M. Blair a d ´eclar ´e qu’il n’y avait rien dans cette vid ´eo qui
pourrait constituer un ”motif raisonnable” qui pourrait conduire `a des accusations criminelles
contre le maire.
Google
TranslateLors d’une conf ´erence de presse jeudi, M. Blair a d ´eclar ´e qu’il n’y avait rien dans cette vido
qui pourrait constituer un ”motif raisonnable” qui pourrait mener `a des accusations criminelles
portes contre le maire.


key: chroma:document
string_value:
qui pourrait constituer un ”motif raisonnable” qui pourrait mener `a des accusations criminelles
portes contre le maire.
Table 3: The translations generated by RNNenc-50 and RNNsearch-50 from long source sentences
(30 words or more) selected from the test set. For each source sentence, we also show the goldstandard translation. The translations by Google Translate were made on 27 August 2014.
15


key: chroma:document
string_value:
arXiv:1409.1556v6  [cs.CV]  10 Apr 2015Publishedasa conferencepaperat ICLR2015
VERYDEEPCONVOLUTIONAL NETWORKS
FORLARGE-SCALEIMAGERECOGNITION
KarenSimonyan∗& AndrewZisserman+
VisualGeometryGroup,DepartmentofEngineeringScience, UniversityofOxford
{karen,az }@robots.ox.ac.uk
ABSTRACT
In this work we investigate the effect of the convolutional n etwork depth on its
accuracy in the large-scale image recognition setting. Our main contribution is
a thorough evaluation of networks of increasing depth using an architecture with
verysmall( 3×3)convolutionﬁlters,whichshowsthatasigniﬁcantimprove ment
on the prior-art conﬁgurations can be achieved by pushing th e depth to 16–19
weight layers. These ﬁndings were the basis of our ImageNet C hallenge 2014
submission,whereourteamsecuredtheﬁrstandthesecondpl acesinthelocalisation and classiﬁcation tracks respectively. We also show th at our representations
generalise well to other datasets, where they achieve state -of-the-art results. We


key: chroma:document
string_value:
generalise well to other datasets, where they achieve state -of-the-art results. We
have made our two best-performingConvNet models publicly a vailable to facilitate furtherresearchontheuse ofdeepvisualrepresentati onsincomputervision.
1 INTRODUCTION
Convolutional networks (ConvNets) have recently enjoyed a great success in large-scale image and video recognition (Krizhevskyetal., 2012; Zeiler& Fergus, 2013; Sermanetet al., 2014;
Simonyan& Zisserman, 2014) which has become possible due to the large public image repositories,suchasImageNet(Denget al.,2009),andhigh-perform ancecomputingsystems,suchasGPUs
orlarge-scaledistributedclusters(Deanet al., 2012). In particular,animportantroleintheadvance
ofdeepvisualrecognitionarchitectureshasbeenplayedby theImageNetLarge-ScaleVisualRecognition Challenge (ILSVRC) (Russakovskyet al., 2014), whic h has served as a testbed for a few


key: chroma:document
string_value:
generationsof large-scale image classiﬁcation systems, f rom high-dimensionalshallow feature encodings(Perronninetal.,2010)(thewinnerofILSVRC-2011 )todeepConvNets(Krizhevskyet al.,
2012)(thewinnerofILSVRC-2012).
With ConvNets becoming more of a commodity in the computer vi sion ﬁeld, a number of attempts have been made to improve the original architecture o f Krizhevskyet al. (2012) in a
bid to achieve better accuracy. For instance, the best-perf orming submissions to the ILSVRC2013 (Zeiler&Fergus, 2013; Sermanetetal., 2014) utilised smaller receptive window size and
smaller stride of the ﬁrst convolutional layer. Another lin e of improvements dealt with training
and testing the networks densely over the whole image and ove r multiple scales (Sermanetet al.,
2014; Howard, 2014). In this paper, we address another impor tant aspect of ConvNet architecture
design–itsdepth. Tothisend,we ﬁxotherparametersofthea rchitecture,andsteadilyincreasethe
depth of the network by adding more convolutionallayers, wh ich is feasible due to the use of very


key: chroma:document
string_value:
depth of the network by adding more convolutionallayers, wh ich is feasible due to the use of very
small (3×3)convolutionﬁltersinall layers.
As a result, we come up with signiﬁcantly more accurate ConvN et architectures, which not only
achieve the state-of-the-art accuracy on ILSVRC classiﬁca tion and localisation tasks, but are also
applicabletootherimagerecognitiondatasets,wherethey achieveexcellentperformanceevenwhen
usedasa partofa relativelysimple pipelines(e.g.deepfea turesclassiﬁed byalinearSVM without
ﬁne-tuning). We havereleasedourtwobest-performingmode ls1tofacilitatefurtherresearch.
The rest of the paper is organised as follows. In Sect. 2, we de scribe our ConvNet conﬁgurations.
The details of the image classiﬁcation trainingand evaluat ionare then presented in Sect. 3, and the
∗current afﬁliation: Google DeepMind+current afﬁliation: Universityof Oxfordand Google DeepMi nd
1http://www.robots.ox.ac.uk/ ˜vgg/research/very_deep/
1


key: chroma:document
string_value:
1http://www.robots.ox.ac.uk/ ˜vgg/research/very_deep/
1
Publishedasa conferencepaperat ICLR2015
conﬁgurations are compared on the ILSVRC classiﬁcation tas k in Sect. 4. Sect. 5 concludes the
paper. For completeness,we also describeand assess our ILS VRC-2014object localisationsystem
inAppendixA,anddiscussthegeneralisationofverydeepfe aturestootherdatasetsinAppendixB.
Finally,AppendixCcontainsthelist ofmajorpaperrevisio ns.
2 CONVNETCONFIGURATIONS
To measure the improvement brought by the increased ConvNet depth in a fair setting, all our
ConvNet layer conﬁgurations are designed using the same pri nciples, inspired by Ciresan etal.
(2011); Krizhevskyet al. (2012). In this section, we ﬁrst de scribe a generic layout of our ConvNet
conﬁgurations(Sect.2.1)andthendetailthespeciﬁcconﬁg urationsusedintheevaluation(Sect.2.2).


key: chroma:document
string_value:
Ourdesignchoicesarethendiscussedandcomparedtothepri orart inSect. 2.3.
2.1 A RCHITECTURE
During training, the input to our ConvNets is a ﬁxed-size 224×224RGB image. The only preprocessingwedoissubtractingthemeanRGBvalue,computed onthetrainingset,fromeachpixel.
Theimageispassedthroughastackofconvolutional(conv.) layers,whereweuseﬁlterswithavery
small receptive ﬁeld: 3×3(which is the smallest size to capture the notion of left/rig ht, up/down,
center). In one of the conﬁgurationswe also utilise 1×1convolutionﬁlters, which can be seen as
a linear transformationof the input channels (followed by n on-linearity). The convolutionstride is
ﬁxedto1pixel;thespatialpaddingofconv.layerinputissuchthatt hespatialresolutionispreserved
afterconvolution,i.e. the paddingis 1pixel for3×3conv.layers. Spatial poolingis carriedoutby


key: chroma:document
string_value:
afterconvolution,i.e. the paddingis 1pixel for3×3conv.layers. Spatial poolingis carriedoutby
ﬁvemax-poolinglayers,whichfollowsomeoftheconv.layer s(notalltheconv.layersarefollowed
bymax-pooling). Max-poolingisperformedovera 2×2pixelwindow,withstride 2.
Astackofconvolutionallayers(whichhasadifferentdepth indifferentarchitectures)isfollowedby
three Fully-Connected(FC) layers: the ﬁrst two have4096ch annelseach,the thirdperforms1000way ILSVRC classiﬁcation and thus contains1000channels(o ne foreach class). The ﬁnal layer is
thesoft-maxlayer. Theconﬁgurationofthefullyconnected layersis thesameinall networks.
Allhiddenlayersareequippedwiththerectiﬁcation(ReLU( Krizhevskyetal.,2012))non-linearity.
We note that none of our networks (except for one) contain Loc al Response Normalisation
(LRN) normalisation (Krizhevskyet al., 2012): as will be sh own in Sect. 4, such normalisation


key: chroma:document
string_value:
(LRN) normalisation (Krizhevskyet al., 2012): as will be sh own in Sect. 4, such normalisation
does not improve the performance on the ILSVRC dataset, but l eads to increased memory consumption and computation time. Where applicable, the param eters for the LRN layer are those
of(Krizhevskyetal., 2012).
2.2 C ONFIGURATIONS
The ConvNet conﬁgurations, evaluated in this paper, are out lined in Table 1, one per column. In
the following we will refer to the nets by their names (A–E). A ll conﬁgurationsfollow the generic
design presented in Sect. 2.1, and differ only in the depth: f rom 11 weight layers in the network A
(8conv.and3FClayers)to19weightlayersinthenetworkE(1 6conv.and3FClayers). Thewidth
of conv.layers (the number of channels) is rather small, sta rting from 64in the ﬁrst layer and then
increasingbyafactorof 2aftereachmax-poolinglayer,untilit reaches 512.
In Table 2 we reportthe numberof parametersfor each conﬁgur ation. In spite of a large depth, the


key: chroma:document
string_value:
In Table 2 we reportthe numberof parametersfor each conﬁgur ation. In spite of a large depth, the
numberof weights in our netsis not greater thanthe numberof weightsin a moreshallow net with
largerconv.layerwidthsandreceptiveﬁelds(144Mweights in(Sermanetet al., 2014)).
2.3 D ISCUSSION
Our ConvNet conﬁgurations are quite different from the ones used in the top-performing entries
of the ILSVRC-2012 (Krizhevskyetal., 2012) and ILSVRC-201 3 competitions (Zeiler& Fergus,
2013;Sermanetet al.,2014). Ratherthanusingrelativelyl argereceptiveﬁeldsintheﬁrstconv.layers(e.g.11×11withstride 4in(Krizhevskyet al.,2012),or 7×7withstride 2in(Zeiler& Fergus,
2013; Sermanetet al., 2014)), we use very small 3×3receptive ﬁelds throughout the whole net,
whichareconvolvedwiththeinputateverypixel(withstrid e1). Itiseasytoseethatastackoftwo


key: chroma:document
string_value:
whichareconvolvedwiththeinputateverypixel(withstrid e1). Itiseasytoseethatastackoftwo
3×3conv.layers(withoutspatialpoolinginbetween)hasaneff ectivereceptiveﬁeldof 5×5;three
2
Publishedasa conferencepaperat ICLR2015
Table 1:ConvNet conﬁgurations (shown in columns). The depth of the conﬁgurations increase s
fromtheleft(A)totheright(E),asmorelayersareadded(th eaddedlayersareshowninbold). The
convolutional layer parameters are denoted as “conv /an}bracketle{treceptive ﬁeld size /an}bracketri}ht-/an}bracketle{tnumber of channels /an}bracketri}ht”.
TheReLU activationfunctionisnotshownforbrevity.
ConvNet Conﬁguration
A A-LRN B C D E
11weight 11weight 13 weight 16weight 16weight 19 weight
layers layers layers layers layers layers
input (224×224RGBimage)
conv3-64 conv3-64 conv3-64 conv3-64 conv3-64 conv3-64


key: chroma:document
string_value:
input (224×224RGBimage)
conv3-64 conv3-64 conv3-64 conv3-64 conv3-64 conv3-64
LRN conv3-64 conv3-64 conv3-64 conv3-64
maxpool
conv3-128 conv3-128 conv3-128 conv3-128 conv3-128 conv3-128
conv3-128 conv3-128 conv3-128 conv3-128
maxpool
conv3-256 conv3-256 conv3-256 conv3-256 conv3-256 conv3-256
conv3-256 conv3-256 conv3-256 conv3-256 conv3-256 conv3-256
conv1-256 conv3-256 conv3-256
conv3-256
maxpool
conv3-512 conv3-512 conv3-512 conv3-512 conv3-512 conv3-512
conv3-512 conv3-512 conv3-512 conv3-512 conv3-512 conv3-512
conv1-512 conv3-512 conv3-512
conv3-512
maxpool
conv3-512 conv3-512 conv3-512 conv3-512 conv3-512 conv3-512
conv3-512 conv3-512 conv3-512 conv3-512 conv3-512 conv3-512


key: chroma:document
string_value:
conv3-512 conv3-512 conv3-512 conv3-512 conv3-512 conv3-512
conv1-512 conv3-512 conv3-512
conv3-512
maxpool
FC-4096
FC-4096
FC-1000
soft-max
Table2:Number ofparameters (inmillions).
Network A,A-LRN BCDE
Number of parameters 133 133134138144
such layers have a 7×7effectivereceptive ﬁeld. So what have we gainedby using, fo r instance, a
stackofthree 3×3conv.layersinsteadofasingle 7×7layer? First,weincorporatethreenon-linear
rectiﬁcation layers instead of a single one, which makes the decision functionmore discriminative.
Second, we decrease the number of parameters: assuming that both the input and the output of a
three-layer 3×3convolutionstack has Cchannels,the stack is parametrisedby 3/parenleftbig
32C2/parenrightbig
= 27C2
weights; at the same time, a single 7×7conv. layer would require 72C2= 49C2parameters, i.e.


key: chroma:document
string_value:
= 27C2
weights; at the same time, a single 7×7conv. layer would require 72C2= 49C2parameters, i.e.
81%more. Thiscan be seen as imposinga regularisationon the 7×7conv.ﬁlters, forcingthemto
haveadecompositionthroughthe 3×3ﬁlters(withnon-linearityinjectedin between).
The incorporation of 1×1conv. layers (conﬁguration C, Table 1) is a way to increase th e nonlinearity of the decision function without affecting the re ceptive ﬁelds of the conv. layers. Even
thoughinourcasethe 1×1convolutionisessentiallyalinearprojectionontothespa ceofthesame
dimensionality(thenumberofinputandoutputchannelsist hesame),anadditionalnon-linearityis
introducedbytherectiﬁcationfunction. Itshouldbenoted that1×1conv.layershaverecentlybeen
utilisedin the“NetworkinNetwork”architectureofLinet a l.(2014).
Small-size convolution ﬁlters have been previously used by Ciresan etal. (2011), but their nets


key: chroma:document
string_value:
Small-size convolution ﬁlters have been previously used by Ciresan etal. (2011), but their nets
are signiﬁcantly less deep than ours, and they did not evalua te on the large-scale ILSVRC
dataset. Goodfellowet al. (2014) applied deep ConvNets ( 11weight layers) to the task of
street number recognition, and showed that the increased de pth led to better performance.
GoogLeNet(Szegedyet al., 2014), a top-performingentryof the ILSVRC-2014classiﬁcation task,
was developed independentlyof our work, but is similar in th at it is based on very deep ConvNets
3
Publishedasa conferencepaperat ICLR2015
(22 weight layers) and small convolution ﬁlters (apart from 3×3, they also use 1×1and5×5
convolutions). Their network topology is, however, more co mplex than ours, and the spatial resolution of the feature maps is reduced more aggressively in th e ﬁrst layers to decrease the amount
of computation. As will be shown in Sect. 4.5, our model is out performing that of Szegedyetal.


key: chroma:document
string_value:
of computation. As will be shown in Sect. 4.5, our model is out performing that of Szegedyetal.
(2014)intermsofthesingle-networkclassiﬁcationaccura cy.
3 CLASSIFICATION FRAMEWORK
In the previous section we presented the details of our netwo rk conﬁgurations. In this section, we
describethe detailsofclassiﬁcationConvNettrainingand evaluation.
3.1 T RAINING
The ConvNet training procedure generally follows Krizhevs kyetal. (2012) (except for sampling
theinputcropsfrommulti-scaletrainingimages,asexplai nedlater). Namely,thetrainingiscarried
out by optimising the multinomial logistic regression obje ctive using mini-batch gradient descent
(based on back-propagation(LeCunet al., 1989)) with momen tum. The batch size was set to 256,
momentum to 0.9. The training was regularised by weight decay (the L2penalty multiplier set to
5·10−4)anddropoutregularisationfortheﬁrsttwofully-connect edlayers(dropoutratiosetto 0.5).


key: chroma:document
string_value:
5·10−4)anddropoutregularisationfortheﬁrsttwofully-connect edlayers(dropoutratiosetto 0.5).
Thelearningrate wasinitially setto 10−2,andthendecreasedbyafactorof 10whenthevalidation
set accuracy stopped improving. In total, the learning rate was decreased 3 times, and the learning
was stopped after 370K iterations (74 epochs). We conjecture that in spite of the l arger number of
parametersandthegreaterdepthofournetscomparedto(Kri zhevskyetal.,2012),thenetsrequired
lessepochstoconvergedueto(a)implicitregularisationi mposedbygreaterdepthandsmallerconv.
ﬁlter sizes; (b)pre-initialisationofcertainlayers.
The initialisation of the networkweightsis important,sin ce bad initialisation can stall learningdue
to the instability of gradient in deep nets. To circumvent th is problem, we began with training
the conﬁgurationA (Table 1), shallow enoughto be trained wi th randominitialisation. Then,when


key: chroma:document
string_value:
the conﬁgurationA (Table 1), shallow enoughto be trained wi th randominitialisation. Then,when
trainingdeeperarchitectures,weinitialisedtheﬁrstfou rconvolutionallayersandthelastthreefullyconnectedlayerswiththelayersofnetA(theintermediatel ayerswereinitialisedrandomly). Wedid
notdecreasethelearningrateforthepre-initialisedlaye rs,allowingthemtochangeduringlearning.
For random initialisation (where applicable), we sampled t he weights from a normal distribution
with thezeromeanand 10−2variance. The biaseswere initialisedwith zero. It isworth notingthat
after the paper submission we found that it is possible to ini tialise the weights without pre-training
byusingthe randominitialisationprocedureofGlorot&Ben gio(2010).
Toobtaintheﬁxed-size 224×224ConvNetinputimages,theywererandomlycroppedfromresca led
training images (one crop per image per SGD iteration). To fu rther augment the training set, the
cropsunderwentrandomhorizontalﬂippingandrandomRGBco lourshift(Krizhevskyet al.,2012).
Trainingimagerescalingisexplainedbelow.


key: chroma:document
string_value:
Trainingimagerescalingisexplainedbelow.
Training image size. LetSbe the smallest side of an isotropically-rescaledtraining image, from
which the ConvNet input is cropped (we also refer to Sas the training scale). While the crop size
is ﬁxed to 224×224, in principle Scan take on any value not less than 224: forS= 224the crop
will capture whole-image statistics, completely spanning the smallest side of a training image; for
S≫224thecropwillcorrespondtoasmallpartoftheimage,contain ingasmallobjectoranobject
part.
We considertwoapproachesforsettingthetrainingscale S. Theﬁrst istoﬁx S,whichcorresponds
to single-scale training (note that image content within th e sampled crops can still represent multiscale image statistics). In our experiments, we evaluated m odels trained at two ﬁxed scales: S=
256(which has been widely used in the prior art (Krizhevskyet al ., 2012; Zeiler&Fergus, 2013;
Sermanetet al., 2014)) and S= 384. Given a ConvNet conﬁguration,we ﬁrst trained the network


key: chroma:document
string_value:
Sermanetet al., 2014)) and S= 384. Given a ConvNet conﬁguration,we ﬁrst trained the network
usingS= 256. To speed-up training of the S= 384network, it was initialised with the weights
pre-trainedwith S= 256,andwe useda smallerinitiallearningrateof 10−3.
The second approachto setting Sis multi-scale training, where each training image is indiv idually
rescaled by randomly sampling Sfrom a certain range [Smin,Smax](we used Smin= 256and
Smax= 512). Sinceobjectsinimagescanbeofdifferentsize,itisbene ﬁcialtotakethisintoaccount
duringtraining. Thiscanalso beseen astrainingset augmen tationbyscale jittering,wherea single
4
Publishedasa conferencepaperat ICLR2015
model is trained to recognise objects over a wide range of sca les. For speed reasons, we trained
multi-scale models by ﬁne-tuning all layers of a single-sca le model with the same conﬁguration,
pre-trainedwithﬁxed S= 384.
3.2 T ESTING


key: chroma:document
string_value:
pre-trainedwithﬁxed S= 384.
3.2 T ESTING
Attest time,givena trainedConvNetandaninputimage,itis classiﬁed inthefollowingway. First,
it is isotropically rescaled to a pre-deﬁned smallest image side, denoted as Q(we also refer to it
as the test scale). We note that Qis not necessarily equal to the training scale S(as we will show
in Sect. 4, usingseveralvaluesof QforeachSleadsto improvedperformance). Then,the network
is applied densely overthe rescaled test image in a way simil ar to (Sermanetet al., 2014). Namely,
the fully-connected layers are ﬁrst converted to convoluti onal layers (the ﬁrst FC layer to a 7×7
conv. layer, the last two FC layers to 1×1conv. layers). The resulting fully-convolutional net is
then applied to the whole (uncropped) image. The result is a c lass score map with the number of
channels equal to the number of classes, and a variable spati al resolution, dependent on the input
imagesize. Finally,toobtainaﬁxed-sizevectorofclasssc oresfortheimage,theclassscoremapis


key: chroma:document
string_value:
imagesize. Finally,toobtainaﬁxed-sizevectorofclasssc oresfortheimage,theclassscoremapis
spatially averaged(sum-pooled). We also augmentthe test s et by horizontalﬂippingof the images;
thesoft-maxclassposteriorsoftheoriginalandﬂippedima gesareaveragedtoobtaintheﬁnalscores
fortheimage.
Since the fully-convolutionalnetwork is applied over the w hole image, there is no need to sample
multiple crops at test time (Krizhevskyetal., 2012), which is less efﬁcient as it requires network
re-computationforeachcrop. Atthesametime,usingalarge setofcrops,asdonebySzegedyetal.
(2014),canleadtoimprovedaccuracy,asit resultsin aﬁner samplingoftheinputimagecompared
tothefully-convolutionalnet. Also,multi-cropevaluati oniscomplementarytodenseevaluationdue
to different convolution boundary conditions: when applyi ng a ConvNet to a crop, the convolved
feature mapsare paddedwith zeros, while in the case of dense evaluationthe paddingfor the same


key: chroma:document
string_value:
feature mapsare paddedwith zeros, while in the case of dense evaluationthe paddingfor the same
crop naturally comes from the neighbouring parts of an image (due to both the convolutions and
spatial pooling), which substantially increases the overa ll network receptive ﬁeld, so more context
iscaptured. Whilewebelievethatinpracticetheincreased computationtimeofmultiplecropsdoes
notjustifythepotentialgainsinaccuracy,forreferencew ealsoevaluateournetworksusing 50crops
perscale( 5×5regulargridwith 2ﬂips),foratotalof 150cropsover 3scales,whichiscomparable
to144cropsover 4scalesusedbySzegedyetal. (2014).
3.3 IMPLEMENTATION DETAILS
OurimplementationisderivedfromthepubliclyavailableC ++ Caffetoolbox(Jia,2013)(branched
out in December 2013), but contains a number of signiﬁcant mo diﬁcations, allowing us to perform
trainingandevaluationonmultipleGPUsinstalledinasing lesystem,aswellastrainandevaluateon
full-size (uncropped) images at multiple scales (as descri bed above). Multi-GPU training exploits


key: chroma:document
string_value:
full-size (uncropped) images at multiple scales (as descri bed above). Multi-GPU training exploits
data parallelism, and is carried out by splitting each batch of training images into several GPU
batches, processed in parallel on each GPU. After the GPU bat ch gradientsare computed, they are
averaged to obtain the gradient of the full batch. Gradient c omputation is synchronous across the
GPUs, sothe resultisexactlythesame aswhentrainingona si ngleGPU.
While more sophisticated methods of speeding up ConvNet tra ining have been recently proposed (Krizhevsky, 2014), which employmodeland data paral lelism for differentlayersof the net,
wehavefoundthatourconceptuallymuchsimplerschemealre adyprovidesaspeedupof 3.75times
on an off-the-shelf4-GPU system, as comparedto using a sing le GPU. On a system equippedwith
fourNVIDIATitanBlackGPUs,trainingasinglenettook2–3w eeksdependingonthearchitecture.
4 CLASSIFICATION EXPERIMENTS
Dataset. In this section, we present the image classiﬁcation results achieved by the described


key: chroma:document
string_value:
4 CLASSIFICATION EXPERIMENTS
Dataset. In this section, we present the image classiﬁcation results achieved by the described
ConvNetarchitecturesontheILSVRC-2012dataset(whichwa susedforILSVRC2012–2014challenges). The dataset includes images of 1000 classes, and is split into three sets: training ( 1.3M
images), validation ( 50K images), and testing ( 100K images with held-out class labels). The classiﬁcation performanceis evaluated using two measures: the top-1 and top-5 error. The former is a
multi-class classiﬁcation error, i.e. the proportion of in correctly classiﬁed images; the latter is the
5
Publishedasa conferencepaperat ICLR2015
main evaluation criterion used in ILSVRC, and is computed as the proportion of images such that
theground-truthcategoryisoutsidethetop-5predictedca tegories.
Forthemajorityofexperiments,weusedthevalidationseta sthetestset. Certainexperimentswere
also carried out on the test set and submitted to the ofﬁcial I LSVRC server as a “VGG” team entry
tothe ILSVRC-2014competition(Russakovskyet al., 2014).


key: chroma:document
string_value:
tothe ILSVRC-2014competition(Russakovskyet al., 2014).
4.1 SINGLESCALEEVALUATION
We begin with evaluating the performanceof individual Conv Net models at a single scale with the
layerconﬁgurationsdescribedin Sect. 2.2. The test images ize was set as follows: Q=Sforﬁxed
S,andQ= 0.5(Smin+Smax)forjittered S∈[Smin,Smax]. Theresultsofareshownin Table3.
First, we note that using local response normalisation (A-L RN network) does not improve on the
model A without any normalisation layers. We thus do not empl oy normalisation in the deeper
architectures(B–E).
Second, we observe that the classiﬁcation error decreases w ith the increased ConvNet depth: from
11 layers in A to 19 layers in E. Notably, in spite of the same de pth, the conﬁguration C (which
containsthree 1×1conv.layers),performsworsethantheconﬁgurationD,whic huses3×3conv.
layersthroughoutthenetwork. Thisindicatesthatwhileth e additionalnon-linearitydoeshelp(Cis


key: chroma:document
string_value:
layersthroughoutthenetwork. Thisindicatesthatwhileth e additionalnon-linearitydoeshelp(Cis
better than B), it is also important to capture spatial conte xt by using conv. ﬁlters with non-trivial
receptive ﬁelds (D is better than C). The error rate of our arc hitecture saturates when the depth
reaches19layers,butevendeepermodelsmightbebeneﬁcialforlarger datasets. Wealsocompared
the net B with a shallow net with ﬁve 5×5conv. layers, which was derived from B by replacing
eachpairof 3×3conv. layerswithasingle 5×5conv. layer(whichhasthesamereceptiveﬁeldas
explained in Sect. 2.3). The top-1 error of the shallow net wa s measured to be 7%higher than that
of B (on a center crop),which conﬁrmsthat a deepnet with smal l ﬁlters outperformsa shallow net
withlargerﬁlters.
Finally, scale jittering at training time ( S∈[256;512] ) leads to signiﬁcantly better results than


key: chroma:document
string_value:
Finally, scale jittering at training time ( S∈[256;512] ) leads to signiﬁcantly better results than
training on images with ﬁxed smallest side ( S= 256orS= 384), even though a single scale is
usedattesttime. Thisconﬁrmsthattrainingsetaugmentati onbyscalejitteringisindeedhelpfulfor
capturingmulti-scaleimagestatistics.
Table3:ConvNetperformanceatasingle testscale.
ConvNet conﬁg. (Table 1) smallest image side top-1 val.error (%) top-5 val.error (%)
train(S)test (Q)
A 256 256 29.6 10.4
A-LRN 256 256 29.7 10.5
B 256 256 28.7 9.9
C256 256 28.1 9.4
384 384 28.1 9.3
[256;512] 384 27.3 8.8
D256 256 27.0 8.8
384 384 26.8 8.7
[256;512] 384 25.6 8.1
E256 256 27.3 9.0
384 384 26.9 8.7
[256;512] 384 25.5 8.0
4.2 M ULTI-SCALEEVALUATION


key: chroma:document
string_value:
384 384 26.9 8.7
[256;512] 384 25.5 8.0
4.2 M ULTI-SCALEEVALUATION
HavingevaluatedtheConvNetmodelsatasinglescale,wenow assesstheeffectofscalejitteringat
testtime. Itconsistsofrunningamodeloverseveralrescal edversionsofatestimage(corresponding
to different values of Q), followed by averaging the resulting class posteriors. Co nsidering that a
large discrepancy between training and testing scales lead s to a drop in performance, the models
trained with ﬁxed Swere evaluated over three test image sizes, close to the trai ning one: Q=
{S−32,S,S+ 32}. At the same time, scale jittering at training time allows th e network to be
appliedto a widerrangeofscales at test time,so the modeltr ainedwithvariable S∈[Smin;Smax]
wasevaluatedoveralargerrangeofsizes Q={Smin,0.5(Smin+Smax),Smax}.
6
Publishedasa conferencepaperat ICLR2015
Theresults,presentedinTable4,indicatethatscalejitte ringattest timeleadstobetterperformance


key: chroma:document
string_value:
6
Publishedasa conferencepaperat ICLR2015
Theresults,presentedinTable4,indicatethatscalejitte ringattest timeleadstobetterperformance
(as compared to evaluating the same model at a single scale, s hown in Table 3). As before, the
deepest conﬁgurations(D and E) perform the best, and scale j ittering is better than training with a
ﬁxed smallest side S. Our best single-network performance on the validation set is24.8%/7.5%
top-1/top-5error(highlightedinboldinTable4). Onthete stset,theconﬁgurationEachieves 7.3%
top-5error.
Table4:ConvNetperformanceatmultiple test scales.
ConvNet conﬁg. (Table 1) smallest image side top-1val. error (%) top-5val. error (%)
train(S)test(Q)
B 256 224,256,288 28.2 9.6
C256 224,256,288 27.7 9.2
384 352,384,416 27.8 9.2
[256;512] 256,384,512 26.3 8.2
D256 224,256,288 26.6 8.6


key: chroma:document
string_value:
[256;512] 256,384,512 26.3 8.2
D256 224,256,288 26.6 8.6
384 352,384,416 26.5 8.6
[256;512] 256,384,512 24.8 7.5
E256 224,256,288 26.9 8.7
384 352,384,416 26.7 8.6
[256;512] 256,384,512 24.8 7.5
4.3 M ULTI-CROP EVALUATION
In Table 5 we compare dense ConvNet evaluation with mult-cro p evaluation (see Sect. 3.2 for details). We also assess the complementarityof thetwo evalua tiontechniquesbyaveragingtheirsoftmax outputs. As can be seen, using multiple crops performs sl ightly better than dense evaluation,
andthe two approachesareindeedcomplementary,astheir co mbinationoutperformseach ofthem.
As noted above, we hypothesize that this is due to a different treatment of convolution boundary
conditions.
Table 5:ConvNetevaluationtechniques comparison. Inall experimentsthe trainingscale Swas
sampledfrom [256;512] ,andthreetest scales Qwereconsidered: {256,384,512}.


key: chroma:document
string_value:
sampledfrom [256;512] ,andthreetest scales Qwereconsidered: {256,384,512}.
ConvNet conﬁg. (Table 1) Evaluationmethod top-1 val. error(%) top-5 val. error (%)
Ddense 24.8 7.5
multi-crop 24.6 7.5
multi-crop &dense 24.4 7.2
Edense 24.8 7.5
multi-crop 24.6 7.4
multi-crop &dense 24.4 7.1
4.4 C ONVNETFUSION
Upuntilnow,weevaluatedtheperformanceofindividualCon vNetmodels. Inthispartoftheexperiments,wecombinetheoutputsofseveralmodelsbyaveragin gtheirsoft-maxclassposteriors. This
improvesthe performancedueto complementarityof the mode ls, andwas used in the top ILSVRC
submissions in 2012 (Krizhevskyet al., 2012) and 2013 (Zeil er&Fergus, 2013; Sermanetet al.,
2014).
The results are shown in Table 6. By the time of ILSVRC submiss ion we had only trained the
single-scale networks, as well as a multi-scale model D (by ﬁ ne-tuning only the fully-connected


key: chroma:document
string_value:
single-scale networks, as well as a multi-scale model D (by ﬁ ne-tuning only the fully-connected
layers rather than all layers). The resulting ensemble of 7 n etworks has 7.3%ILSVRC test error.
After the submission, we considered an ensemble of only two b est-performing multi-scale models
(conﬁgurations D and E), which reduced the test error to 7.0%using dense evaluation and 6.8%
using combined dense and multi-crop evaluation. For refere nce, our best-performingsingle model
achieves7.1%error(modelE, Table5).
4.5 C OMPARISON WITH THE STATE OF THE ART
Finally, we compare our results with the state of the art in Ta ble 7. In the classiﬁcation task of
ILSVRC-2014 challenge (Russakovskyet al., 2014), our “VGG ” team secured the 2nd place with
7
Publishedasa conferencepaperat ICLR2015
Table6:Multiple ConvNetfusion results.
Combined ConvNet modelsError
top-1 val top-5val top-5test
ILSVRCsubmission


key: chroma:document
string_value:
Table6:Multiple ConvNetfusion results.
Combined ConvNet modelsError
top-1 val top-5val top-5test
ILSVRCsubmission
(D/256/224,256,288), (D/384/352,384,416), (D/[256;512 ]/256,384,512)
(C/256/224,256,288), (C/384/352,384,416)
(E/256/224,256,288), (E/384/352,384,416)24.7 7.5 7.3
post-submission
(D/[256;512]/256,384,512), (E/[256;512]/256,384,512) ,dense eval. 24.0 7.1 7.0
(D/[256;512]/256,384,512), (E/[256;512]/256,384,512) ,multi-crop 23.9 7.2 (D/[256;512]/256,384,512), (E/[256;512]/256,384,512) ,multi-crop &dense eval. 23.7 6.8 6.8
7.3%test errorusinganensembleof7 models. Afterthesubmissio n,we decreasedtheerrorrateto


key: chroma:document
string_value:
7.3%test errorusinganensembleof7 models. Afterthesubmissio n,we decreasedtheerrorrateto
6.8%usinganensembleof2models.
As can be seen from Table 7, our very deep ConvNetssigniﬁcant ly outperformthe previousgeneration of models, which achieved the best results in the ILSVR C-2012 and ILSVRC-2013 competitions. Our result is also competitivewith respect to the cla ssiﬁcation task winner(GoogLeNetwith
6.7%error) and substantially outperforms the ILSVRC-2013 winn ing submission Clarifai, which
achieved 11.2%with outside training data and 11.7%without it. This is remarkable, considering
that our best result is achievedby combiningjust two models – signiﬁcantly less than used in most
ILSVRC submissions. In terms of the single-net performance , our architecture achieves the best
result (7.0%test error), outperforming a single GoogLeNet by 0.9%. Notably, we did not depart
from the classical ConvNet architecture of LeCunetal. (198 9), but improved it by substantially
increasingthedepth.


key: chroma:document
string_value:
from the classical ConvNet architecture of LeCunetal. (198 9), but improved it by substantially
increasingthedepth.
Table 7:Comparison with the state of the art in ILSVRC classiﬁcation . Our methodis denoted
as“VGG”.Onlytheresultsobtainedwithoutoutsidetrainin gdataarereported.
Method top-1 val. error(%) top-5val. error (%) top-5testerror (%)
VGG(2nets, multi-crop& dense eval.) 23.7 6.8 6.8
VGG(1net, multi-crop& dense eval.) 24.4 7.1 7.0
VGG(ILSVRCsubmission, 7nets, dense eval.) 24.7 7.5 7.3
GoogLeNet (Szegedy et al., 2014) (1net) - 7.9
GoogLeNet (Szegedy et al., 2014) (7nets) - 6.7
MSRA(He et al., 2014) (11nets) - - 8.1
MSRA(He et al., 2014) (1net) 27.9 9.1 9.1
Clarifai(Russakovsky et al., 2014) (multiplenets) - - 11.7


key: chroma:document
string_value:
Clarifai(Russakovsky et al., 2014) (multiplenets) - - 11.7
Clarifai(Russakovsky et al., 2014) (1net) - - 12.5
Zeiler& Fergus (Zeiler&Fergus, 2013) (6nets) 36.0 14.7 14.8
Zeiler& Fergus (Zeiler&Fergus, 2013) (1net) 37.5 16.0 16.1
OverFeat (Sermanetet al.,2014) (7nets) 34.0 13.2 13.6
OverFeat (Sermanetet al.,2014) (1net) 35.7 14.2 Krizhevsky et al.(Krizhevsky et al., 2012) (5nets) 38.1 16.4 16.4
Krizhevsky et al.(Krizhevsky et al., 2012) (1net) 40.7 18.2 5 CONCLUSION
In this work we evaluated very deep convolutional networks ( up to 19 weight layers) for largescale image classiﬁcation. It was demonstrated that the rep resentation depth is beneﬁcial for the
classiﬁcationaccuracy,andthatstate-of-the-artperfor manceontheImageNetchallengedatasetcan


key: chroma:document
string_value:
classiﬁcationaccuracy,andthatstate-of-the-artperfor manceontheImageNetchallengedatasetcan
beachievedusingaconventionalConvNetarchitecture(LeC unet al.,1989;Krizhevskyet al.,2012)
withsubstantiallyincreaseddepth. Intheappendix,weals oshowthatourmodelsgeneralisewellto
a wide range of tasks and datasets, matchingor outperformin gmore complexrecognitionpipelines
builtaroundlessdeepimagerepresentations. Ourresultsy etagainconﬁrmtheimportanceof depth
invisualrepresentations.
ACKNOWLEDGEMENTS
ThisworkwassupportedbyERCgrantVisRecno.228180. Wegra tefullyacknowledgethesupport
ofNVIDIACorporationwiththedonationoftheGPUsusedfort hisresearch.
8
Publishedasa conferencepaperat ICLR2015
REFERENCES
Bell, S., Upchurch, P.,Snavely, N., and Bala, K. Material re cognition inthe wild withthe materials in context
database. CoRR,abs/1412.0623, 2014.


key: chroma:document
string_value:
database. CoRR,abs/1412.0623, 2014.
Chatﬁeld, K., Simonyan, K., Vedaldi, A., and Zisserman, A. R eturn of the devil in the details: Delving deep
intoconvolutional nets. In Proc.BMVC. ,2014.
Cimpoi,M.,Maji,S.,andVedaldi,A. Deepconvolutionalﬁlt erbanksfortexturerecognitionandsegmentation.
CoRR,abs/1411.6836, 2014.
Ciresan, D. C., Meier, U., Masci, J., Gambardella, L. M., and Schmidhuber, J. Flexible, high performance
convolutional neural networks for image classiﬁcation. In IJCAI,pp. 1237–1242, 2011.
Dean, J., Corrado, G., Monga, R., Chen, K., Devin, M., Mao, M. , Ranzato, M., Senior, A., Tucker, P., Yang,
K.,Le,Q. V.,andNg, A.Y. Large scale distributeddeepnetwo rks. InNIPS,pp. 1232–1240, 2012.


key: chroma:document
string_value:
K.,Le,Q. V.,andNg, A.Y. Large scale distributeddeepnetwo rks. InNIPS,pp. 1232–1240, 2012.
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei , L. Imagenet: A large-scale hierarchical image
database. In Proc.CVPR ,2009.
Donahue,J.,Jia,Y.,Vinyals,O.,Hoffman,J.,Zhang,N.,Tz eng,E.,andDarrell,T.Decaf: Adeepconvolutional
activation feature for generic visual recognition. CoRR,abs/1310.1531, 2013.
Everingham, M., Eslami, S.M. A., Van Gool, L., Williams,C., Winn, J., and Zisserman, A. The Pascal visual
object classes challenge: Aretrospective. IJCV,111(1):98–136, 2015.
Fei-Fei, L., Fergus, R., and Perona, P. Learning generative visual models from few training examples: An
incremental bayesian approach tested on 101 object categor ies. InIEEE CVPR Workshop of Generative
Model BasedVision , 2004.


key: chroma:document
string_value:
incremental bayesian approach tested on 101 object categor ies. InIEEE CVPR Workshop of Generative
Model BasedVision , 2004.
Girshick, R. B., Donahue, J., Darrell, T., and Malik, J. Rich feature hierarchies for accurate object detection
and semantic segmentation. CoRR,abs/1311.2524v5, 2014. PublishedinProc.CVPR,2014.
Gkioxari, G.,Girshick, R.,and Malik, J. Actions and attrib utes from wholes and parts. CoRR,abs/1412.2604,
2014.
Glorot, X. andBengio, Y. Understanding the difﬁcultyof tra iningdeep feedforward neural networks. In Proc.
AISTATS,volume 9, pp. 249–256, 2010.
Goodfellow, I. J., Bulatov, Y., Ibarz, J., Arnoud, S., and Sh et, V. Multi-digit number recognition from street
view imagery usingdeep convolutional neural networks. In Proc.ICLR ,2014.
Grifﬁn, G., Holub, A., and Perona, P. Caltech-256 object cat egory dataset. Technical Report 7694, California
Institute of Technology, 2007.


key: chroma:document
string_value:
Institute of Technology, 2007.
He, K., Zhang, X., Ren, S., and Sun, J. Spatial pyramid poolin g in deep convolutional networks for visual
recognition. CoRR,abs/1406.4729v2, 2014.
Hoai, M. Regularizedmax pooling forimage categorization. InProc. BMVC. ,2014.
Howard, A.G. Someimprovements ondeepconvolutional neura l networkbasedimageclassiﬁcation. In Proc.
ICLR,2014.
Jia, Y. Caffe: An open source convolutional architecture fo r fast feature embedding.
http://caffe.berkeleyvision.org/ ,2013.
Karpathy, A. and Fei-Fei, L. Deep visual-semantic alignmen ts for generating image descriptions. CoRR,
abs/1412.2306, 2014.
Kiros, R., Salakhutdinov, R., and Zemel, R. S. Unifying visu al-semantic embeddings with multimodal neural
language models. CoRR,abs/1411.2539, 2014.
Krizhevsky, A. One weirdtrickfor parallelizingconvoluti onal neural networks. CoRR,abs/1404.5997, 2014.


key: chroma:document
string_value:
Krizhevsky, A. One weirdtrickfor parallelizingconvoluti onal neural networks. CoRR,abs/1404.5997, 2014.
Krizhevsky, A., Sutskever, I., and Hinton, G. E. ImageNet cl assiﬁcation with deep convolutional neural networks. In NIPS,pp. 1106–1114, 2012.
LeCun,Y.,Boser, B.,Denker, J.S.,Henderson, D.,Howard, R .E.,Hubbard, W.,andJackel, L.D. Backpropagationapplied tohandwrittenzipcode recognition. Neural Computation , 1(4):541–551, 1989.
Lin,M., Chen, Q.,andYan, S. Networkinnetwork. In Proc.ICLR ,2014.
Long, J., Shelhamer, E., and Darrell, T. Fully convolutiona l networks for semantic segmentation. CoRR,
abs/1411.4038, 2014.
Oquab, M., Bottou, L., Laptev, I., and Sivic, J. Learning and Transferring Mid-Level Image Representations
using Convolutional Neural Networks. In Proc.CVPR ,2014.


key: chroma:document
string_value:
using Convolutional Neural Networks. In Proc.CVPR ,2014.
Perronnin, F.,S´ anchez, J.,andMensink, T. Improving theF isherkernel forlarge-scale image classiﬁcation. In
Proc.ECCV ,2010.
Razavian, A.,Azizpour, H.,Sullivan, J.,andCarlsson,S. C NNFeaturesoff-the-shelf: anAstounding Baseline
for Recognition. CoRR,abs/1403.6382, 2014.
9
Publishedasa conferencepaperat ICLR2015
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A.,
Bernstein, M., Berg, A. C., and Fei-Fei, L. ImageNet large sc ale visual recognition challenge. CoRR,
abs/1409.0575, 2014.
Sermanet,P.,Eigen, D.,Zhang, X.,Mathieu, M.,Fergus,R., andLeCun,Y. OverFeat: IntegratedRecognition,
Localizationand Detectionusing Convolutional Networks. InProc.ICLR ,2014.


key: chroma:document
string_value:
Localizationand Detectionusing Convolutional Networks. InProc.ICLR ,2014.
Simonyan, K. and Zisserman, A. Two-stream convolutional ne tworks for action recognition in videos. CoRR,
abs/1406.2199, 2014. Published inProc.NIPS,2014.
Szegedy, C., Liu, W.,Jia, Y., Sermanet, P.,Reed, S.,Anguel ov, D.,Erhan, D., Vanhoucke, V., and Rabinovich,
A. Goingdeeper withconvolutions. CoRR,abs/1409.4842, 2014.
Wei, Y., Xia, W., Huang, J., Ni, B., Dong, J., Zhao, Y., and Yan , S. CNN: Single-label to multi-label. CoRR,
abs/1406.5726, 2014.
Zeiler, M. D. and Fergus, R. Visualizing and understanding c onvolutional networks. CoRR, abs/1311.2901,
2013. PublishedinProc. ECCV,2014.
A LOCALISATION
In the main bodyof the paper we have consideredthe classiﬁca tion task of the ILSVRC challenge,


key: chroma:document
string_value:
A LOCALISATION
In the main bodyof the paper we have consideredthe classiﬁca tion task of the ILSVRC challenge,
and performed a thorough evaluation of ConvNet architectur es of different depth. In this section,
we turn to the localisation task of the challenge, which we ha ve won in 2014 with 25.3%error. It
can be seen as a special case of object detection, where a sing le object bounding box should be
predictedforeach ofthe top-5classes, irrespectiveof the actual numberofobjectsof the class. For
thiswe adoptthe approachof Sermanetet al. (2014), the winn ersof the ILSVRC-2013localisation
challenge,withafewmodiﬁcations. Ourmethodisdescribed inSect.A.1andevaluatedinSect.A.2.
A.1 L OCALISATION CONVNET
To perform object localisation, we use a very deep ConvNet, w here the last fully connected layer
predicts the bounding box location instead of the class scor es. A bounding box is represented by
a 4-D vector storing its center coordinates, width, and heig ht. There is a choice of whether the


key: chroma:document
string_value:
a 4-D vector storing its center coordinates, width, and heig ht. There is a choice of whether the
boundingbox prediction is shared across all classes (singl e-class regression, SCR (Sermanetet al.,
2014))orisclass-speciﬁc(per-classregression,PCR).In theformercase,thelastlayeris4-D,while
in the latter it is 4000-D (since there are 1000 classes in the dataset). Apart from the last bounding
boxpredictionlayer,weuse theConvNetarchitectureD (Tab le1),whichcontains16weightlayers
andwasfoundtobe thebest-performingin theclassiﬁcation task (Sect.4).
Training. Training of localisation ConvNets is similar to that of the c lassiﬁcation ConvNets
(Sect.3.1). Themaindifferenceisthatwereplacethelogis ticregressionobjectivewithaEuclidean
loss,whichpenalisesthedeviationofthepredictedboundi ngboxparametersfromtheground-truth.
We trainedtwo localisation models, each on a single scale: S= 256andS= 384(due to the time


key: chroma:document
string_value:
We trainedtwo localisation models, each on a single scale: S= 256andS= 384(due to the time
constraints,we didnot use trainingscale jitteringforour ILSVRC-2014submission). Trainingwas
initialised with the correspondingclassiﬁcation models ( trained on the same scales), and the initial
learning rate was set to 10−3. We exploredboth ﬁne-tuningall layers and ﬁne-tuningonly the ﬁrst
two fully-connected layers, as done in (Sermanetetal., 201 4). The last fully-connected layer was
initialisedrandomlyandtrainedfromscratch.
Testing. We consider two testing protocols. The ﬁrst is used for compa ring different network
modiﬁcations on the validation set, and considers only the b oundingbox prediction for the ground
truth class (to factor out the classiﬁcation errors). The bo unding box is obtained by applying the
networkonlyto thecentralcropoftheimage.
The second, fully-ﬂedged, testing procedure is based on the dense application of the localisation
ConvNet to the whole image, similarly to the classiﬁcation t ask (Sect. 3.2). The difference is that


key: chroma:document
string_value:
ConvNet to the whole image, similarly to the classiﬁcation t ask (Sect. 3.2). The difference is that
instead of the class score map, the output of the last fully-c onnected layer is a set of bounding
box predictions. To come up with the ﬁnal prediction, we util ise the greedy merging procedure
of Sermanetetal. (2014), which ﬁrst merges spatially close predictions (by averaging their coordinates), and then rates them based on the class scores, obta ined from the classiﬁcation ConvNet.
When several localisation ConvNets are used, we ﬁrst take th e union of their sets of boundingbox
predictions, and then run the mergingprocedureon the union . We did not use the multiple pooling
10
Publishedasa conferencepaperat ICLR2015
offsets technique of Sermanetetal. (2014), which increase s the spatial resolution of the bounding
boxpredictionsandcanfurtherimprovetheresults.
A.2 L OCALISATION EXPERIMENTS


key: chroma:document
string_value:
boxpredictionsandcanfurtherimprovetheresults.
A.2 L OCALISATION EXPERIMENTS
In this section we ﬁrst determine the best-performinglocal isation setting (using the ﬁrst test protocol), and then evaluate it in a fully-ﬂedged scenario (the se cond protocol). The localisation error
is measured according to the ILSVRC criterion (Russakovsky et al., 2014), i.e. the bounding box
predictionis deemed correctif its intersectionoverunion ratio with the ground-truthboundingbox
isabove0.5.
Settings comparison. As can be seen from Table 8, per-class regression (PCR) outpe rforms the
class-agnostic single-class regression (SCR), which diff ers from the ﬁndings of Sermanetetal.
(2014), where PCR was outperformed by SCR. We also note that ﬁ ne-tuning all layers for the localisation task leads to noticeablybetter results than ﬁne -tuningonly the fully-connectedlayers(as
donein(Sermanetet al.,2014)). Intheseexperiments,thes mallestimagessidewassetto S= 384;
theresultswith S= 256exhibitthesamebehaviourandarenotshownforbrevity.


key: chroma:document
string_value:
theresultswith S= 256exhibitthesamebehaviourandarenotshownforbrevity.
Table 8:Localisation error for different modiﬁcations with the simpliﬁed testing protocol: the
boundingbox is predictedfrom a single central image crop, a nd the ground-truthclass is used. All
ConvNet layers (except for the last one) have the conﬁgurati on D (Table 1), while the last layer
performseithersingle-classregression(SCR) orper-clas sregression(PCR).
Fine-tunedlayers regression type GTclass localisationerror
1st and2nd FCSCR 36.4
PCR 34.3
all PCR 33.1
Fully-ﬂedgedevaluation. Havingdeterminedthebestlocalisationsetting(PCR,ﬁne- tuningofall
layers),we nowapply it in the fully-ﬂedgedscenario,where the top-5class labelsare predictedusing our best-performingclassiﬁcation system (Sect. 4.5), and multiple densely-computedbounding


key: chroma:document
string_value:
box predictions are merged using the method of Sermanetetal . (2014). As can be seen from Table 9, applicationof the localisationConvNetto the whole i magesubstantiallyimprovesthe results
compared to using a center crop (Table 8), despite using the t op-5 predicted class labels instead of
thegroundtruth. Similarlytotheclassiﬁcationtask(Sect .4),testingatseveralscalesandcombining
thepredictionsofmultiplenetworksfurtherimprovesthep erformance.
Table9:Localisationerror
smallestimage side top-5localisationerror (%)
train(S) test(Q) val. test.
256 256 29.5 384 384 28.2 26.7
384 352,384 27.5 fusion: 256/256 and 384/352,384 26.9 25.3
Comparison with the state of the art. We compare our best localisation result with the state
of the art in Table 10. With 25.3%test error, our “VGG” team won the localisation challenge of
ILSVRC-2014 (Russakovskyet al., 2014). Notably, our resul ts are considerably better than those
of the ILSVRC-2013winnerOverfeat(Sermanetet al., 2014), even thoughwe used less scales and


key: chroma:document
string_value:
of the ILSVRC-2013winnerOverfeat(Sermanetet al., 2014), even thoughwe used less scales and
did not employ their resolution enhancement technique. We e nvisage that better localisation performance can be achieved if this technique is incorporated i nto our method. This indicates the
performanceadvancementbroughtbyourverydeepConvNets– wegotbetterresultswithasimpler
localisationmethod,buta morepowerfulrepresentation.
B GENERALISATION OF VERYDEEPFEATURES
In the previous sections we have discussed training and eval uation of very deep ConvNets on the
ILSVRC dataset. In this section, we evaluate our ConvNets, p re-trained on ILSVRC, as feature
11
Publishedasa conferencepaperat ICLR2015
Table 10: Comparison with the state of the art in ILSVRC localisation . Our methodis denoted
as“VGG”.
Method top-5val. error (%) top-5 testerror (%)
VGG 26.9 25.3
GoogLeNet (Szegedyet al., 2014) - 26.7
OverFeat (Sermanet etal.,2014) 30.0 29.9
Krizhevsky et al.(Krizhevsky et al.,2012) - 34.2


key: chroma:document
string_value:
Krizhevsky et al.(Krizhevsky et al.,2012) - 34.2
extractors on other, smaller, datasets, where training lar ge models from scratch is not feasible due
to over-ﬁtting. Recently, there has been a lot of interest in such a use case (Zeiler&Fergus, 2013;
Donahueet al., 2013; Razavianet al., 2014; Chatﬁeldet al., 2014), as it turns out that deep image
representations,learntonILSVRC,generalisewelltoothe rdatasets,wheretheyhaveoutperformed
hand-crafted representations by a large margin. Following that line of work, we investigate if our
modelsleadtobetterperformancethanmoreshallowmodelsu tilisedinthestate-of-the-artmethods.
In this evaluation, we consider two models with the best clas siﬁcation performance on ILSVRC
(Sect.4)–conﬁgurations“Net-D”and“Net-E”(whichwemade publiclyavailable).
To utilise the ConvNets, pre-trained on ILSVRC, for image cl assiﬁcation on other datasets, we


key: chroma:document
string_value:
To utilise the ConvNets, pre-trained on ILSVRC, for image cl assiﬁcation on other datasets, we
remove the last fully-connected layer (which performs 1000 -way ILSVRC classiﬁcation), and use
4096-Dactivationsofthepenultimatelayerasimagefeatur es,whichareaggregatedacrossmultiple
locations and scales. The resulting image descriptor is L2-normalised and combined with a linear
SVM classiﬁer, trained on the target dataset. For simplicit y, pre-trained ConvNet weights are kept
ﬁxed(noﬁne-tuningisperformed).
Aggregation of features is carried out in a similar manner to our ILSVRC evaluation procedure
(Sect. 3.2). Namely, an image is ﬁrst rescaled so that its sma llest side equals Q, and then the network is densely applied over the image plane (which is possib le when all weight layers are treated
as convolutional). We then perform global average pooling o n the resulting feature map, which
producesa 4096-Dimage descriptor. The descriptor is then a veraged with the descriptor of a horizontally ﬂipped image. As was shown in Sect. 4.2, evaluation over multiple scales is beneﬁcial, so


key: chroma:document
string_value:
we extract features over several scales Q. The resulting multi-scale features can be either stacked
or pooled across scales. Stacking allows a subsequent class iﬁer to learn how to optimally combine
image statistics over a range of scales; this, however, come s at the cost of the increased descriptor
dimensionality. We returntothediscussionofthisdesignc hoicein theexperimentsbelow. We also
assess late fusion of features, computed using two networks , which is performed by stacking their
respectiveimagedescriptors.
Table11: Comparisonwiththestateoftheartinimageclassiﬁcationo nVOC-2007,VOC-2012,
Caltech-101, and Caltech-256 . Our models are denoted as “VGG”. Results marked with * were
achievedusingConvNetspre-trainedonthe extended ILSVRCdataset(2000classes).
MethodVOC-2007 VOC-2012 Caltech-101 Caltech-256
(meanAP) (mean AP) (meanclass recall) (mean class recall)
Zeiler& Fergus (Zeiler&Fergus, 2013) - 79.0 86.5±0.5 74.2±0.3


key: chroma:document
string_value:
Zeiler& Fergus (Zeiler&Fergus, 2013) - 79.0 86.5±0.5 74.2±0.3
Chatﬁeldetal. (Chatﬁeldet al., 2014) 82.4 83.2 88.4±0.6 77.6±0.1
He etal. (Heet al.,2014) 82.4 - 93.4±0.5 Weiet al.(Weiet al., 2014) 81.5(85.2∗)81.7 (90.3∗) - VGGNet-D (16layers) 89.3 89.0 91.8±1.0 85.0±0.2
VGGNet-E(19 layers) 89.3 89.0 92.3±0.5 85.1±0.3
VGGNet-D & Net-E 89.7 89.3 92.7±0.5 86.2±0.3
Image Classiﬁcation on VOC-2007and VOC-2012. We beginwith the evaluationon the image
classiﬁcation task of PASCAL VOC-2007 and VOC-2012 benchma rks (Everinghametal., 2015).


key: chroma:document
string_value:
classiﬁcation task of PASCAL VOC-2007 and VOC-2012 benchma rks (Everinghametal., 2015).
These datasets contain 10K and 22.5K images respectively, a nd each image is annotated with one
or several labels, correspondingto 20 object categories. T he VOC organisersprovidea pre-deﬁned
split into training, validation, and test data (the test dat a for VOC-2012 is not publicly available;
instead,anofﬁcialevaluationserverisprovided). Recogn itionperformanceismeasuredusingmean
averageprecision(mAP)acrossclasses.
Notably, by examining the performance on the validation set s of VOC-2007 and VOC-2012, we
foundthat aggregatingimage descriptors,computedat mult iple scales, by averagingperformssim12
Publishedasa conferencepaperat ICLR2015
ilarly to the aggregation by stacking. We hypothesize that t his is due to the fact that in the VOC


key: chroma:document
string_value:
Publishedasa conferencepaperat ICLR2015
ilarly to the aggregation by stacking. We hypothesize that t his is due to the fact that in the VOC
dataset the objects appear over a variety of scales, so there is no particular scale-speciﬁc semantics which a classiﬁer could exploit. Since averaging has a b eneﬁt of not inﬂating the descriptor dimensionality, we were able to aggregated image descri ptors over a wide range of scales:
Q∈ {256,384,512,640,768}. It is worth noting though that the improvement over a smalle r
rangeof{256,384,512}wasrathermarginal( 0.3%).
Thetestsetperformanceisreportedandcomparedwithother approachesinTable11. Ournetworks
“Net-D”and“Net-E”exhibitidenticalperformanceonVOCda tasets,andtheircombinationslightly
improves the results. Our methods set the new state of the art across image representations, pretrained on the ILSVRC dataset, outperformingthe previousb est result of Chatﬁeldet al. (2014) by
more than 6%. It should be noted that the method of Wei et al. (2014), which achieves1%better


key: chroma:document
string_value:
more than 6%. It should be noted that the method of Wei et al. (2014), which achieves1%better
mAP on VOC-2012, is pre-trained on an extended 2000-class IL SVRC dataset, which includes
additional 1000 categories, semantically close to those in VOC datasets. It also beneﬁts from the
fusionwith anobjectdetection-assistedclassiﬁcation pi peline.
ImageClassiﬁcationonCaltech-101andCaltech-256. InthissectionweevaluateverydeepfeaturesonCaltech-101(Fei-Feiet al.,2004)andCaltech-256 (Grifﬁnet al.,2007)imageclassiﬁcation
benchmarks. Caltech-101contains9Kimageslabelledinto1 02classes(101objectcategoriesanda
backgroundclass), while Caltech-256 is larger with 31K ima ges and 257 classes. A standard evaluation protocolon these datasets is to generateseveral ran domsplits into training and test data and
report the average recognition performance across the spli ts, which is measured by the mean class
recall(whichcompensatesforadifferentnumberoftestima gesperclass). FollowingChatﬁeld etal.


key: chroma:document
string_value:
recall(whichcompensatesforadifferentnumberoftestima gesperclass). FollowingChatﬁeld etal.
(2014); Zeiler&Fergus(2013); He etal. (2014),onCaltech- 101we generated3 randomsplits into
training and test data, so that each split contains 30 traini ng images per class, and up to 50 test
images per class. On Caltech-256 we also generated 3 splits, each of which contains 60 training
images per class (and the rest is used for testing). In each sp lit, 20% of training images were used
asa validationset forhyper-parameterselection.
We found that unlike VOC, on Caltech datasets the stacking of descriptors, computed over multiple scales, performs better than averaging or max-pooling. This can be explained by the fact that
in Caltech images objects typically occupy the whole image, so multi-scale image features are semanticallydifferent(capturingthe wholeobject vs. object parts), andstacking allows a classiﬁer to
exploitsuchscale-speciﬁcrepresentations. We usedthree scalesQ∈ {256,384,512}.
Ourmodelsarecomparedtoeachotherandthestateofthearti nTable11. Ascanbeseen,thedeeper


key: chroma:document
string_value:
Ourmodelsarecomparedtoeachotherandthestateofthearti nTable11. Ascanbeseen,thedeeper
19-layerNet-Eperformsbetterthanthe16-layerNet-D,and theircombinationfurtherimprovesthe
performance. On Caltech-101, our representations are comp etitive with the approach of He etal.
(2014),which,however,performssigniﬁcantlyworsethano urnetsonVOC-2007. OnCaltech-256,
ourfeaturesoutperformthestate oftheart (Chatﬁeldetal. , 2014)byalargemargin( 8.6%).
Action Classiﬁcation on VOC-2012. We also evaluated our best-performing image representation (the stacking of Net-D and Net-E features) on the PASCAL VOC-2012 action classiﬁcation
task (Everinghamet al., 2015), which consists in predictin g an action class from a single image,
given a bounding box of the person performing the action. The dataset contains 4.6K training images,labelledinto11classes. SimilarlytotheVOC-2012ob jectclassiﬁcationtask,theperformance


key: chroma:document
string_value:
is measured using the mAP. We considered two training settin gs: (i) computing the ConvNet featuresonthewholeimageandignoringtheprovidedboundingb ox;(ii)computingthefeaturesonthe
wholeimageandontheprovidedboundingbox,andstackingth emtoobtaintheﬁnalrepresentation.
TheresultsarecomparedtootherapproachesinTable12.
OurrepresentationachievesthestateofartontheVOCactio nclassiﬁcationtaskevenwithoutusing
the provided bounding boxes, and the results are further imp roved when using both images and
bounding boxes. Unlike other approaches, we did not incorpo rate any task-speciﬁc heuristics, but
reliedontherepresentationpowerofverydeepconvolution alfeatures.
Other Recognition Tasks. Since the public release of our models, they have been active ly used
by the research community for a wide range of image recogniti on tasks, consistently outperforming more shallow representations. For instance, Girshicke t al. (2014) achieve the state of the
object detection results by replacing the ConvNet of Krizhe vskyet al. (2012) with our 16-layer


key: chroma:document
string_value:
object detection results by replacing the ConvNet of Krizhe vskyet al. (2012) with our 16-layer
model. Similar gains over a more shallow architecture of Kri zhevskyet al. (2012) have been ob13
Publishedasa conferencepaperat ICLR2015
Table 12: Comparison with the state of the art in single-image action c lassiﬁcation on VOC2012. Our models are denoted as “VGG”. Results marked with * were a chieved using ConvNets
pre-trainedonthe extended ILSVRCdataset (1512classes).
Method VOC-2012 (meanAP)
(Oquab et al., 2014) 70.2∗
(Gkioxari etal.,2014) 73.6
(Hoai,2014) 76.3
VGG Net-D& Net-E,image-only 79.2
VGG Net-D& Net-E,image and bounding box 84.0
served in semantic segmentation (Longet al., 2014), image c aption generation (Kirosetal., 2014;
Karpathy& Fei-Fei, 2014),textureandmaterialrecognitio n(Cimpoiet al., 2014; Bell etal., 2014).
C PAPERREVISIONS


key: chroma:document
string_value:
C PAPERREVISIONS
Here we present the list of major paper revisions, outlining the substantial changes for the convenienceofthe reader.
v1Initialversion. Presentstheexperimentscarriedoutbefo rethe ILSVRCsubmission.
v2Addspost-submissionILSVRCexperimentswithtrainingset augmentationusingscalejittering,
whichimprovestheperformance.
v3Addsgeneralisationexperiments(AppendixB) on PASCAL VOC andCaltech image classiﬁcationdatasets. Themodelsusedforthese experimentsarepub liclyavailable.
v4The paper is converted to ICLR-2015 submission format. Also adds experiments with multiple
cropsforclassiﬁcation.
v6Camera-readyICLR-2015conferencepaper. Addsa compariso nof the net B with a shallow net
andtheresultsonPASCAL VOCactionclassiﬁcationbenchmar k.
14


key: chroma:document
string_value:
Fully Convolutional Networks for Semantic Segmentation
Jonathan LongEvan ShelhamerTrevor Darrell
UC Berkeley
fjonlong,shelhamer,trevor g@cs.berkeley.edu
Abstract
Convolutional networks are powerful visual models that
yield hierarchies of features. We show that convolutional networks by themselves, trained end-to-end, pixelsto-pixels, exceed the state-of-the-art in semantic segmentation. Our key insight is to build “fully convolutional”
networks that take input of arbitrary size and produce
correspondingly-sized output with efﬁcient inference and
learning. We deﬁne and detail the space of fully convolutional networks, explain their application to spatially dense
prediction tasks, and draw connections to prior models. We
adapt contemporary classiﬁcation networks (AlexNet [19],
the VGG net [31], and GoogLeNet [32]) into fully convolutional networks and transfer their learned representations
by ﬁne-tuning [4] to the segmentation task. We then deﬁne a novel architecture that combines semantic information from a deep, coarse layer with appearance information
from a shallow, ﬁne layer to produce accurate and detailed


key: chroma:document
string_value:
from a shallow, ﬁne layer to produce accurate and detailed
segmentations. Our fully convolutional network achieves
state-of-the-art segmentation of PASCAL VOC (20% relative improvement to 62.2% mean IU on 2012), NYUDv2,
and SIFT Flow, while inference takes less than one ﬁfth of a
second for a typical image.
1. Introduction
Convolutional networks are driving advances in recognition. Convnets are not only improving for whole-image
classiﬁcation [19, 31, 32], but also making progress on local tasks with structured output. These include advances in
bounding box object detection [29, 12, 17], part and keypoint prediction [39, 24], and local correspondence [24, 9].
The natural next step in the progression from coarse to
ﬁne inference is to make a prediction at every pixel. Prior
approaches have used convnets for semantic segmentation
[27, 2, 8, 28, 16, 14, 11], in which each pixel is labeled with
the class of its enclosing object or region, but with shortcomings that this work addresses.
Authors contributed equally
963842564096409621
21backward/learningforward/inference
pixelwise predictionsegmentation g.t.


key: chroma:document
string_value:
Authors contributed equally
963842564096409621
21backward/learningforward/inference
pixelwise predictionsegmentation g.t.
256384Figure 1. Fully convolutional networks can efﬁciently learn to
make dense predictions for per-pixel tasks like semantic segmentation.
We show that a fully convolutional network (FCN),
trained end-to-end, pixels-to-pixels on semantic segmentation exceeds the state-of-the-art without further machinery. To our knowledge, this is the ﬁrst work to train FCNs
end-to-end (1) for pixelwise prediction and (2) from supervised pre-training. Fully convolutional versions of existing
networks predict dense outputs from arbitrary-sized inputs.
Both learning and inference are performed whole-image-ata-time by dense feedforward computation and backpropagation. In-network upsampling layers enable pixelwise prediction and learning in nets with subsampled pooling.
This method is efﬁcient, both asymptotically and absolutely, and precludes the need for the complications in other
works. Patchwise training is common [27, 2, 8, 28, 11], but


key: chroma:document
string_value:
works. Patchwise training is common [27, 2, 8, 28, 11], but
lacks the efﬁciency of fully convolutional training. Our approach does not make use of pre- and post-processing complications, including superpixels [8, 16], proposals [16, 14],
or post-hoc reﬁnement by random ﬁelds or local classiﬁers
[8, 16]. Our model transfers recent success in classiﬁcation [19, 31, 32] to dense prediction by reinterpreting classiﬁcation nets as fully convolutional and ﬁne-tuning from
their learned representations. In contrast, previous works
have applied small convnets without supervised pre-training
[8, 28, 27].
Semantic segmentation faces an inherent tension between semantics and location: global information resolves
what while local information resolves where. Deep feature
1arXiv:1411.4038v2  [cs.CV]  8 Mar 2015
hierarchies jointly encode location and semantics in a localto-global pyramid. We deﬁne a novel “skip” architecture
to combine deep, coarse, semantic information and shallow,
ﬁne, appearance information in Section 4.2 (see Figure 3).


key: chroma:document
string_value:
to combine deep, coarse, semantic information and shallow,
ﬁne, appearance information in Section 4.2 (see Figure 3).
In the next section, we review related work on deep classiﬁcation nets, FCNs, and recent approaches to semantic
segmentation using convnets. The following sections explain FCN design and dense prediction tradeoffs, introduce
our architecture with in-network upsampling and multilayer combinations, and describe our experimental framework. Finally, we demonstrate state-of-the-art results on
PASCAL VOC 2011-2, NYUDv2, and SIFT Flow.
2. Related work
Our approach draws on recent successes of deep nets
for image classiﬁcation [19, 31, 32] and transfer learning
[4, 38]. Transfer was ﬁrst demonstrated on various visual
recognition tasks [4, 38], then on detection, and on both
instance and semantic segmentation in hybrid proposalclassiﬁer models [12, 16, 14]. We now re-architect and ﬁnetune classiﬁcation nets to direct, dense prediction of semantic segmentation. We chart the space of FCNs and situate
prior models, both historical and recent, in this framework.
Fully convolutional networks To our knowledge, the


key: chroma:document
string_value:
prior models, both historical and recent, in this framework.
Fully convolutional networks To our knowledge, the
idea of extending a convnet to arbitrary-sized inputs ﬁrst
appeared in Matan et al. [25], which extended the classic
LeNet [21] to recognize strings of digits. Because their net
was limited to one-dimensional input strings, Matan et al.
used Viterbi decoding to obtain their outputs. Wolf and Platt
[37] expand convnet outputs to 2-dimensional maps of detection scores for the four corners of postal address blocks.
Both of these historical works do inference and learning
fully convolutionally for detection. Ning et al. [27] deﬁne
a convnet for coarse multiclass segmentation of C. elegans
tissues with fully convolutional inference.
Fully convolutional computation has also been exploited
in the present era of many-layered nets. Sliding window
detection by Sermanet et al . [29], semantic segmentation
by Pinheiro and Collobert [28], and image restoration by
Eigen et al. [5] do fully convolutional inference. Fully convolutional training is rare, but used effectively by Tompson
et al. [35] to learn an end-to-end part detector and spatial


key: chroma:document
string_value:
et al. [35] to learn an end-to-end part detector and spatial
model for pose estimation, although they do not exposit on
or analyze this method.
Alternatively, He et al . [17] discard the nonconvolutional portion of classiﬁcation nets to make a
feature extractor. They combine proposals and spatial
pyramid pooling to yield a localized, ﬁxed-length feature
for classiﬁcation. While fast and effective, this hybrid
model cannot be learned end-to-end.
Dense prediction with convnets Several recent works
have applied convnets to dense prediction problems, including semantic segmentation by Ning et al. [27], Farabet et al.[8], and Pinheiro and Collobert [28]; boundary prediction
for electron microscopy by Ciresan et al. [2] and for natural
images by a hybrid neural net/nearest neighbor model by
Ganin and Lempitsky [11]; and image restoration and depth
estimation by Eigen et al. [5, 6]. Common elements of these
approaches include
small models restricting capacity and receptive ﬁelds;
patchwise training [27, 2, 8, 28, 11];
post-processing by superpixel projection, random ﬁeld


key: chroma:document
string_value:
patchwise training [27, 2, 8, 28, 11];
post-processing by superpixel projection, random ﬁeld
regularization, ﬁltering, or local classiﬁcation [8, 2,
11];
input shifting and output interlacing for dense output
[28, 11] as introduced by OverFeat [29];
multi-scale pyramid processing [8, 28, 11];
saturating tanh nonlinearities [8, 5, 28]; and
ensembles [2, 11],
whereas our method does without this machinery. However,
we do study patchwise training 3.4 and “shift-and-stitch”
dense output 3.2 from the perspective of FCNs. We also
discuss in-network upsampling 3.3, of which the fully connected prediction by Eigen et al. [6] is a special case.
Unlike these existing methods, we adapt and extend deep
classiﬁcation architectures, using image classiﬁcation as supervised pre-training, and ﬁne-tune fully convolutionally to
learn simply and efﬁciently from whole image inputs and
whole image ground thruths.


key: chroma:document
string_value:
learn simply and efﬁciently from whole image inputs and
whole image ground thruths.
Hariharan et al. [16] and Gupta et al. [14] likewise adapt
deep classiﬁcation nets to semantic segmentation, but do
so in hybrid proposal-classiﬁer models. These approaches
ﬁne-tune an R-CNN system [12] by sampling bounding
boxes and/or region proposals for detection, semantic segmentation, and instance segmentation. Neither method is
learned end-to-end.
They achieve state-of-the-art results on PASCAL VOC
segmentation and NYUDv2 segmentation respectively, so
we directly compare our standalone, end-to-end FCN to
their semantic segmentation results in Section 5.
3. Fully convolutional networks
Each layer of data in a convnet is a three-dimensional
array of size hwd, wherehandware spatial dimensions, anddis the feature or channel dimension. The ﬁrst
layer is the image, with pixel size hw, anddcolor channels. Locations in higher layers correspond to the locations
in the image they are path-connected to, which are called
their receptive ﬁelds .


key: chroma:document
string_value:
in the image they are path-connected to, which are called
their receptive ﬁelds .
Convnets are built on translation invariance. Their basic components (convolution, pooling, and activation functions) operate on local input regions, and depend only on
relative spatial coordinates. Writing xijfor the data vector
at location (i;j)in a particular layer, and yijfor the following layer, these functions compute outputs yijby
yij=fks(fxsi+i;sj +jg0i;jk)
wherekis called the kernel size, sis the stride or subsampling factor, and fksdetermines the layer type: a matrix
multiplication for convolution or average pooling, a spatial
max for max pooling, or an elementwise nonlinearity for an
activation function, and so on for other types of layers.
This functional form is maintained under composition,
with kernel size and stride obeying the transformation rule
fksgk0s0= (fg)k0+(k1)s0;ss0:
While a general deep net computes a general nonlinear
function, a net with only layers of this form computes a


key: chroma:document
string_value:
While a general deep net computes a general nonlinear
function, a net with only layers of this form computes a
nonlinear ﬁlter , which we call a deep ﬁlter orfully convolutional network . An FCN naturally operates on an input of
any size, and produces an output of corresponding (possibly
resampled) spatial dimensions.
A real-valued loss function composed with an FCN deﬁnes a task. If the loss function is a sum over the spatial
dimensions of the ﬁnal layer, `(x;) =P
ij`0(xij;), its
gradient will be a sum over the gradients of each of its spatial components. Thus stochastic gradient descent on `computed on whole images will be the same as stochastic gradient descent on `0, taking all of the ﬁnal layer receptive ﬁelds
as a minibatch.
When these receptive ﬁelds overlap signiﬁcantly, both
feedforward computation and backpropagation are much
more efﬁcient when computed layer-by-layer over an entire
image instead of independently patch-by-patch.
We next explain how to convert classiﬁcation nets into
fully convolutional nets that produce coarse output maps.


key: chroma:document
string_value:
image instead of independently patch-by-patch.
We next explain how to convert classiﬁcation nets into
fully convolutional nets that produce coarse output maps.
For pixelwise prediction, we need to connect these coarse
outputs back to the pixels. Section 3.2 describes a trick that
OverFeat [29] introduced for this purpose. We gain insight
into this trick by reinterpreting it as an equivalent network
modiﬁcation. As an efﬁcient, effective alternative, we introduce deconvolution layers for upsampling in Section 3.3.
In Section 3.4 we consider training by patchwise sampling,
and give evidence in Section 4.3 that our whole image training is faster and equally effective.
3.1. Adapting classiﬁers for dense prediction
Typical recognition nets, including LeNet [21], AlexNet
[19], and its deeper successors [31, 32], ostensibly take
ﬁxed-sized inputs and produce nonspatial outputs. The fully
connected layers of these nets have ﬁxed dimensions and
throw away spatial coordinates. However, these fully connected layers can also be viewed as convolutions with kernels that cover their entire input regions. Doing so casts
them into fully convolutional networks that take input of
any size and output classiﬁcation maps. This transformation


key: chroma:document
string_value:
them into fully convolutional networks that take input of
any size and output classiﬁcation maps. This transformation
`tabby cat" `
96256384384256409640961000
96384256409640961000
256384tabby cat heatmapconvolutionalizationFigure 2. Transforming fully connected layers into convolution
layers enables a classiﬁcation net to output a heatmap. Adding
layers and a spatial loss (as in Figure 1) produces an efﬁcient machine for end-to-end dense learning.
is illustrated in Figure 2. (By contrast, nonconvolutional
nets, such as the one by Le et al. [20], lack this capability.)
Furthermore, while the resulting maps are equivalent to
the evaluation of the original net on particular input patches,
the computation is highly amortized over the overlapping
regions of those patches. For example, while AlexNet takes
1:2ms (on a typical GPU) to produce the classiﬁcation
scores of a 227227image, the fully convolutional version takes 22ms to produce a 1010grid of outputs from
a500500image, which is more than 5times faster than
the na ¨ıve approach1.


key: chroma:document
string_value:
a500500image, which is more than 5times faster than
the na ¨ıve approach1.
The spatial output maps of these convolutionalized models make them a natural choice for dense problems like semantic segmentation. With ground truth available at every output cell, both the forward and backward passes are
straightforward, and both take advantage of the inherent
computational efﬁciency (and aggressive optimization) of
convolution.
The corresponding backward times for the AlexNet example are 2:4ms for a single image and 37ms for a fully
convolutional 1010output map, resulting in a speedup
similar to that of the forward pass. This dense backpropagation is illustrated in Figure 1.
While our reinterpretation of classiﬁcation nets as fully
convolutional yields output maps for inputs of any size, the
output dimensions are typically reduced by subsampling.
The classiﬁcation nets subsample to keep ﬁlters small and
computational requirements reasonable. This coarsens the
output of a fully convolutional version of these nets, reducing it from the size of the input by a factor equal to the pixel
stride of the receptive ﬁelds of the output units.
1Assuming efﬁcient batching of single image inputs. The classiﬁcation


key: chroma:document
string_value:
stride of the receptive ﬁelds of the output units.
1Assuming efﬁcient batching of single image inputs. The classiﬁcation
scores for a single image by itself take 5.4 ms to produce, which is nearly
25times slower than the fully convolutional version.
3.2. Shift-and-stitch is ﬁlter rarefaction
Input shifting and output interlacing is a trick that yields
dense predictions from coarse outputs without interpolation, introduced by OverFeat [29]. If the outputs are downsampled by a factor of f, the input is shifted (by left and top
padding)xpixels to the right and ypixels down, once for
every value of (x;y)2f0;:::;f1gf 0;:::;f1g.
Thesef2inputs are each run through the convnet, and the
outputs are interlaced so that the predictions correspond to
the pixels at the centers of their receptive ﬁelds.
Changing only the ﬁlters and layer strides of a convnet
can produce the same output as this shift-and-stitch trick.
Consider a layer (convolution or pooling) with input stride


key: chroma:document
string_value:
can produce the same output as this shift-and-stitch trick.
Consider a layer (convolution or pooling) with input stride
s, and a following convolution layer with ﬁlter weights fij
(eliding the feature dimensions, irrelevant here). Setting the
lower layer’s input stride to 1upsamples its output by a factor ofs, just like shift-and-stitch. However, convolving the
original ﬁlter with the upsampled output does not produce
the same result as the trick, because the original ﬁlter only
sees a reduced portion of its (now upsampled) input. To
reproduce the trick, rarefy the ﬁlter by enlarging it as
f0
ij=fi=s;j=s ifsdivides both iandj;
0 otherwise,
(withiandjzero-based). Reproducing the full net output
of the trick involves repeating this ﬁlter enlargement layerby-layer until all subsampling is removed.
Simply decreasing subsampling within a net is a tradeoff:
the ﬁlters see ﬁner information, but have smaller receptive
ﬁelds and take longer to compute. We have seen that the
shift-and-stitch trick is another kind of tradeoff: the output


key: chroma:document
string_value:
ﬁelds and take longer to compute. We have seen that the
shift-and-stitch trick is another kind of tradeoff: the output
is made denser without decreasing the receptive ﬁeld sizes
of the ﬁlters, but the ﬁlters are prohibited from accessing
information at a ﬁner scale than their original design.
Although we have done preliminary experiments with
shift-and-stitch, we do not use it in our model. We ﬁnd
learning through upsampling, as described in the next section, to be more effective and efﬁcient, especially when
combined with the skip layer fusion described later on.
3.3. Upsampling is backwards strided convolution
Another way to connect coarse outputs to dense pixels
is interpolation. For instance, simple bilinear interpolation
computes each output yijfrom the nearest four inputs by a
linear map that depends only on the relative positions of the
input and output cells.
In a sense, upsampling with factor fis convolution with
afractional input stride of 1=f. So long as fis integral, a
natural way to upsample is therefore backwards convolution
(sometimes called deconvolution ) with an output stride of
f. Such an operation is trivial to implement, since it simply


key: chroma:document
string_value:
natural way to upsample is therefore backwards convolution
(sometimes called deconvolution ) with an output stride of
f. Such an operation is trivial to implement, since it simply
reverses the forward and backward passes of convolution.Thus upsampling is performed in-network for end-to-end
learning by backpropagation from the pixelwise loss.
Note that the deconvolution ﬁlter in such a layer need not
be ﬁxed (e.g., to bilinear upsampling), but can be learned.
A stack of deconvolution layers and activation functions can
even learn a nonlinear upsampling.
In our experiments, we ﬁnd that in-network upsampling
is fast and effective for learning dense prediction. Our best
segmentation architecture uses these layers to learn to upsample for reﬁned prediction in Section 4.2.
3.4. Patchwise training is loss sampling
In stochastic optimization, gradient computation is
driven by the training distribution. Both patchwise training and fully-convolutional training can be made to produce any distribution, although their relative computational
efﬁciency depends on overlap and minibatch size. Whole
image fully convolutional training is identical to patchwise
training where each batch consists of all the receptive ﬁelds


key: chroma:document
string_value:
image fully convolutional training is identical to patchwise
training where each batch consists of all the receptive ﬁelds
of the units below the loss for an image (or collection of
images). While this is more efﬁcient than uniform sampling
of patches, it reduces the number of possible batches. However, random selection of patches within an image may be
recovered simply. Restricting the loss to a randomly sampled subset of its spatial terms (or, equivalently applying a
DropConnect mask [36] between the output and the loss)
excludes patches from the gradient computation.
If the kept patches still have signiﬁcant overlap, fully
convolutional computation will still speed up training. If
gradients are accumulated over multiple backward passes,
batches can include patches from several images.2
Sampling in patchwise training can correct class imbalance [27, 8, 2] and mitigate the spatial correlation of dense
patches [28, 16]. In fully convolutional training, class balance can also be achieved by weighting the loss, and loss
sampling can be used to address spatial correlation.
We explore training with sampling in Section 4.3, and do
not ﬁnd that it yields faster or better convergence for dense
prediction. Whole image training is effective and efﬁcient.
4. Segmentation Architecture


key: chroma:document
string_value:
not ﬁnd that it yields faster or better convergence for dense
prediction. Whole image training is effective and efﬁcient.
4. Segmentation Architecture
We cast ILSVRC classiﬁers into FCNs and augment
them for dense prediction with in-network upsampling and
a pixelwise loss. We train for segmentation by ﬁne-tuning.
Next, we build a novel skip architecture that combines
coarse, semantic and local, appearance information to reﬁne prediction.
For this investigation, we train and validate on the PASCAL VOC 2011 segmentation challenge [7]. We train with
2Note that not every possible patch is included this way, since the receptive ﬁelds of the ﬁnal layer units lie on a ﬁxed, strided grid. However,
by shifting the image left and down by a random value up to the stride,
random selection from all possible patches may be recovered.
a per-pixel multinomial logistic loss and validate with the
standard metric of mean pixel intersection over union, with
the mean taken over all classes, including background. The
training ignores pixels that are masked out (as ambiguous
or difﬁcult) in the ground truth.
4.1. From classiﬁer to dense FCN


key: chroma:document
string_value:
training ignores pixels that are masked out (as ambiguous
or difﬁcult) in the ground truth.
4.1. From classiﬁer to dense FCN
We begin by convolutionalizing proven classiﬁcation architectures as in Section 3. We consider the AlexNet3architecture [19] that won ILSVRC12, as well as the VGG
nets [31] and the GoogLeNet4[32] which did exceptionally well in ILSVRC14. We pick the VGG 16-layer net5,
which we found to be equivalent to the 19-layer net on this
task. For GoogLeNet, we use only the ﬁnal loss layer, and
improve performance by discarding the ﬁnal average pooling layer. We decapitate each net by discarding the ﬁnal
classiﬁer layer, and convert all fully connected layers to
convolutions. We append a 11convolution with channel dimension 21to predict scores for each of the PASCAL classes (including background) at each of the coarse
output locations, followed by a deconvolution layer to bilinearly upsample the coarse outputs to pixel-dense outputs
as described in Section 3.3. Table 1 compares the preliminary validation results along with the basic characteristics


key: chroma:document
string_value:
as described in Section 3.3. Table 1 compares the preliminary validation results along with the basic characteristics
of each net. We report the best results achieved after convergence at a ﬁxed learning rate (at least 175 epochs).
Fine-tuning from classiﬁcation to segmentation gave reasonable predictions for each net. Even the worst model
achieved75% of state-of-the-art performance. The
segmentation-equippped VGG net (FCN-VGG16) already
appears to be state-of-the-art at 56.0 mean IU on val, compared to 52.6 on test [16]. Training on extra data raises
performance to 59.4 mean IU on a subset of val7. Training
details are given in Section 4.3.
Despite similar classiﬁcation accuracy, our implementation of GoogLeNet did not match this segmentation result.
4.2. Combining what and where
We deﬁne a new fully convolutional net (FCN) for segmentation that combines layers of the feature hierarchy and
reﬁnes the spatial precision of the output. See Figure 3.
While fully convolutionalized classiﬁers can be ﬁnetuned to segmentation as shown in 4.1, and even score


key: chroma:document
string_value:
While fully convolutionalized classiﬁers can be ﬁnetuned to segmentation as shown in 4.1, and even score
highly on the standard metric, their output is dissatisfyingly
coarse (see Figure 4). The 32 pixel stride at the ﬁnal prediction layer limits the scale of detail in the upsampled output.
We address this by adding links that combine the ﬁnal
prediction layer with lower layers with ﬁner strides. This
3Using the publicly available CaffeNet reference model.
4Since there is no publicly available version of GoogLeNet, we use
our own reimplementation. Our version is trained with less extensive data
augmentation, and gets 68.5% top-1 and 88.4% top-5 ILSVRC accuracy.
5Using the publicly available version from the Caffe model zoo.Table 1. We adapt and extend three classiﬁcation convnets to segmentation. We compare performance by mean intersection over
union on the validation set of PASCAL VOC 2011 and by inference time (averaged over 20 trials for a 500500 input on an
NVIDIA Tesla K40c). We detail the architecture of the adapted
nets as regards dense prediction: number of parameter layers, receptive ﬁeld size of output units, and the coarsest stride within the


key: chroma:document
string_value:
nets as regards dense prediction: number of parameter layers, receptive ﬁeld size of output units, and the coarsest stride within the
net. (These numbers give the best performance obtained at a ﬁxed
learning rate, not best performance possible.)
FCNAlexNetFCNVGG16FCNGoogLeNet4
mean IU 39.8 56.0 42.5
forward time 50 ms 210 ms 59 ms
conv. layers 8 16 22
parameters 57M 134M 6M
rf size 355 404 907
max stride 32 32 32
turns a line topology into a DAG, with edges that skip ahead
from lower layers to higher ones (Figure 3). As they see
fewer pixels, the ﬁner scale predictions should need fewer
layers, so it makes sense to make them from shallower net
outputs. Combining ﬁne layers and coarse layers lets the
model make local predictions that respect global structure.
By analogy to the multiscale local jet of Florack et al. [10],
we call our nonlinear local feature hierarchy the deep jet .
We ﬁrst divide the output stride in half by predicting
from a 16 pixel stride layer. We add a 11convolution
layer on top of pool4 to produce additional class predictions. We fuse this output with the predictions computed


key: chroma:document
string_value:
from a 16 pixel stride layer. We add a 11convolution
layer on top of pool4 to produce additional class predictions. We fuse this output with the predictions computed
on top of conv7 (convolutionalized fc7) at stride 32 by
adding a 2upsampling layer and summing6both predictions. (See Figure 3). We initialize the 2upsampling to
bilinear interpolation, but allow the parameters to be learned
as described in Section 3.3. Finally, the stride 16 predictions
are upsampled back to the image. We call this net FCN-16s.
FCN-16s is learned end-to-end, initialized with the parameters of the last, coarser net, which we now call FCN-32s.
The new parameters acting on pool4 are zero-initialized so
that the net starts with unmodiﬁed predictions. The learning
rate is decreased by a factor of 100.
Learning this skip net improves performance on the validation set by 3.0 mean IU to 62.4. Figure 4 shows improvement in the ﬁne structure of the output. We compared
this fusion with learning only from the pool4 layer (which
resulted in poor performance), and simply decreasing the
learning rate without adding the extra link (which results


key: chroma:document
string_value:
this fusion with learning only from the pool4 layer (which
resulted in poor performance), and simply decreasing the
learning rate without adding the extra link (which results
in an insigniﬁcant performance improvement, without improving the quality of the output).
We continue in this fashion by fusing predictions from
pool3 with a 2upsampling of predictions fused from
pool4 andconv7 , building the net FCN-8s. We obtain
6Max fusion made learning difﬁcult due to gradient switching.
FCN-32s FCN-16s FCN-8s Ground truth
Figure 4. Reﬁning fully convolutional nets by fusing information
from layers with different strides improves segmentation detail.
The ﬁrst three images show the output from our 32, 16, and 8
pixel stride nets (see Figure 3).
Table 2. Comparison of skip FCNs on a subset of PASCAL
VOC2011 validation7. Learning is end-to-end, except for FCN32s-ﬁxed, where only the last layer is ﬁne-tuned. Note that FCN32s is FCN-VGG16, renamed to highlight stride.
pixel
acc.mean
acc.mean
IUf.w.
IU


key: chroma:document
string_value:
pixel
acc.mean
acc.mean
IUf.w.
IU
FCN-32s-ﬁxed 83.0 59.7 45.4 72.0
FCN-32s 89.1 73.3 59.4 81.4
FCN-16s 90.0 75.7 62.4 83.0
FCN-8s 90.3 75.9 62.7 83.2
a minor additional improvement to 62.7 mean IU, and ﬁnd
a slight improvement in the smoothness and detail of our
output. At this point our fusion improvements have met diminishing returns, both with respect to the IU metric which
emphasizes large-scale correctness, and also in terms of the
improvement visible e.g. in Figure 4, so we do not continue
fusing even lower layers.
Reﬁnement by other means Decreasing the stride of
pooling layers is the most straightforward way to obtain
ﬁner predictions. However, doing so is problematic for our
VGG16-based net. Setting the pool5 layer to have stride 1
requires our convolutionalized fc6 to have a kernel size of1414in order to maintain its receptive ﬁeld size. In addition to their computational cost, we had difﬁculty learning


key: chroma:document
string_value:
such large ﬁlters. We made an attempt to re-architect the
layers above pool5 with smaller ﬁlters, but were not successful in achieving comparable performance; one possible
explanation is that the initialization from ImageNet-trained
weights in the upper layers is important.
Another way to obtain ﬁner predictions is to use the shiftand-stitch trick described in Section 3.2. In limited experiments, we found the cost to improvement ratio from this
method to be worse than layer fusion.
4.3. Experimental framework
Optimization We train by SGD with momentum. We
use a minibatch size of 20 images and ﬁxed learning rates of
103,104, and 55for FCN-AlexNet, FCN-VGG16, and
FCN-GoogLeNet, respectively, chosen by line search. We
use momentum 0:9, weight decay of 54or24, and doubled the learning rate for biases, although we found training
to be insensitive to these parameters (but sensitive to the
learning rate). We zero-initialize the class scoring convolution layer, ﬁnding random initialization to yield neither
better performance nor faster convergence. Dropout was included where used in the original classiﬁer nets.


key: chroma:document
string_value:
better performance nor faster convergence. Dropout was included where used in the original classiﬁer nets.
Fine-tuning We ﬁne-tune all layers by backpropagation through the whole net. Fine-tuning the
output classiﬁer alone yields only 70% of the full ﬁnetuning performance as compared in Table 2. Training from
scratch is not feasible considering the time required to
learn the base classiﬁcation nets. (Note that the VGG net is
trained in stages, while we initialize from the full 16-layer
version.) Fine-tuning takes three days on a single GPU for
the coarse FCN-32s version, and about one day each to
upgrade to the FCN-16s and FCN-8s versions.
Patch Sampling As explained in Section 3.4, our full
image training effectively batches each image into a reguimage pool4 pool5 pool1 pool2 pool332x upsampled
prediction (FCN-32s)2x upsampled
prediction16x upsampled
prediction (FCN-16s)8x upsampled
prediction (FCN-8s)
pool4
prediction2x upsampled
prediction
pool3
predictionP P


key: chroma:document
string_value:
prediction (FCN-8s)
pool4
prediction2x upsampled
prediction
pool3
predictionP P
Figure 3. Our DAG nets learn to combine coarse, high layer information with ﬁne, low layer information. Layers are shown as grids that
reveal relative spatial coarseness. Only pooling and prediction layers are shown; intermediate convolution layers (including our converted
fully connected layers) are omitted. Solid line (FCN-32s): Our single-stream net, described in Section 4.1, upsamples stride 32 predictions
back to pixels in a single step. Dashed line (FCN-16s): Combining predictions from both the ﬁnal layer and the pool4 layer, at stride
16, lets our net predict ﬁner details, while retaining high-level semantic information. Dotted line (FCN-8s): Additional predictions from
pool3 , at stride 8, provide further precision.
500 1000 1500
iteration number0.40.60.81.01.2lossfull images
50% sampling
25% sampling
10000 20000 30000
relative time (num. images processed)0.40.60.81.01.2lossFigure 5. Training on whole images is just as effective as sampling
patches, but results in faster (wall time) convergence by making


key: chroma:document
string_value:
patches, but results in faster (wall time) convergence by making
more efﬁcient use of data. Left shows the effect of sampling on
convergence rate for a ﬁxed expected batch size, while right plots
the same by relative wall time.
lar grid of large, overlapping patches. By contrast, prior
work randomly samples patches over a full dataset [27, 2,
8, 28, 11], potentially resulting in higher variance batches
that may accelerate convergence [22]. We study this tradeoff by spatially sampling the loss in the manner described
earlier, making an independent choice to ignore each ﬁnal
layer cell with some probability 1p. To avoid changing the
effective batch size, we simultaneously increase the number
of images per batch by a factor 1=p. Note that due to the efﬁciency of convolution, this form of rejection sampling is
still faster than patchwise training for large enough values
ofp(e.g., at least for p > 0:2according to the numbers
in Section 3.1). Figure 5 shows the effect of this form of
sampling on convergence. We ﬁnd that sampling does not
have a signiﬁcant effect on convergence rate compared to
whole image training, but takes signiﬁcantly more time due


key: chroma:document
string_value:
have a signiﬁcant effect on convergence rate compared to
whole image training, but takes signiﬁcantly more time due
to the larger number of images that need to be considered
per batch. We therefore choose unsampled, whole image
training in our other experiments.
Class Balancing Fully convolutional training can balance classes by weighting or sampling the loss. Although
our labels are mildly unbalanced (about 3=4are background), we ﬁnd class balancing unnecessary.
Dense Prediction The scores are upsampled to the input dimensions by deconvolution layers within the net. Final layer deconvolutional ﬁlters are ﬁxed to bilinear interpolation, while intermediate upsampling layers are initialized to bilinear upsampling, and then learned. Shift-andstitch (Section 3.2), or the ﬁlter rarefaction equivalent, are
not used.
Augmentation We tried augmenting the training data
by randomly mirroring and “jittering” the images by translating them up to 32 pixels (the coarsest scale of prediction)
in each direction. This yielded no noticeable improvement.
More Training Data The PASCAL VOC 2011 segmentation challenge training set, which we used for Table 1,


key: chroma:document
string_value:
in each direction. This yielded no noticeable improvement.
More Training Data The PASCAL VOC 2011 segmentation challenge training set, which we used for Table 1,
labels 1112 images. Hariharan et al . [15] have collectedlabels for a much larger set of 8498 PASCAL training images, which was used to train the previous state-of-the-art
system, SDS [16]. This training data improves the FCNVGG16 validation score7by 3.4 points to 59.4 mean IU.
Implementation All models are trained and tested with
Caffe [18] on a single NVIDIA Tesla K40c. The models
and code will be released open-source on publication.
5. Results
We test our FCN on semantic segmentation and scene
parsing, exploring PASCAL VOC, NYUDv2, and SIFT
Flow. Although these tasks have historically distinguished
between objects and regions, we treat both uniformly as
pixel prediction. We evaluate our FCN skip architecture8
on each of these datasets, and then extend it to multi-modal
input for NYUDv2 and multi-task prediction for the semantic and geometric labels of SIFT Flow.
Metrics We report four metrics from common semantic
segmentation and scene parsing evaluations that are variations on pixel accuracy and region intersection over union


key: chroma:document
string_value:
Metrics We report four metrics from common semantic
segmentation and scene parsing evaluations that are variations on pixel accuracy and region intersection over union
(IU). Letnijbe the number of pixels of class ipredicted to
belong to class j, where there are ncldifferent classes, and
letti=P
jnijbe the total number of pixels of class i. We
compute:
pixel accuracy:P
inii=P
iti
mean accuraccy: (1=ncl)P
inii=ti
mean IU: (1=ncl)P
inii=
ti+P
jnjinii
frequency weighted IU:
(P
ktk)1P
itinii=
ti+P
jnjinii
PASCAL VOC Table 3 gives the performance of our
FCN-8s on the test sets of PASCAL VOC 2011 and 2012,
and compares it to the previous state-of-the-art, SDS [16],
and the well-known R-CNN [12]. We achieve the best results on mean IU9by a relative margin of 20%. Inference
time is reduced 114(convnet only, ignoring proposals and
reﬁnement) or 286(overall).


key: chroma:document
string_value:
time is reduced 114(convnet only, ignoring proposals and
reﬁnement) or 286(overall).
Table 3. Our fully convolutional net gives a 20% relative improvement over the state-of-the-art on the PASCAL VOC 2011 and 2012
test sets, and reduces inference time.
mean IU mean IU inference
VOC2011 test VOC2012 test time
R-CNN [12] 47.9 - SDS [16] 52.6 51.6 50 s
FCN-8s 62.7 62.2 175 ms
NYUDv2 [30] is an RGB-D dataset collected using the
7There are training images from [15] included in the PASCAL VOC
2011 val set, so we validate on the non-intersecting set of 736 images. An
earlier version of this paper mistakenly evaluated on the entire val set.
8Our models and code are publicly available at
https://github.com/BVLC/caffe/wiki/Model-Zoo#fcn .
9This is the only metric provided by the test server.
Table 4. Results on NYUDv2. RGBD is early-fusion of the
RGB and depth channels at the input. HHA is the depth embedding of [14] as horizontal disparity, height above ground, and


key: chroma:document
string_value:
RGB and depth channels at the input. HHA is the depth embedding of [14] as horizontal disparity, height above ground, and
the angle of the local surface normal with the inferred gravity
direction. RGB-HHA is the jointly trained late fusion model
that sums RGB and HHA predictions.
pixel
acc.mean
acc.mean
IUf.w.
IU
Gupta et al. [14] 60.3 - 28.6 47.0
FCN-32s RGB 60.0 42.2 29.2 43.9
FCN-32s RGBD 61.5 42.4 30.5 45.5
FCN-32s HHA 57.1 35.2 24.2 40.4
FCN-32s RGB-HHA 64.3 44.9 32.8 48.0
FCN-16s RGB-HHA 65.4 46.1 34.0 49.5
Microsoft Kinect. It has 1449 RGB-D images, with pixelwise labels that have been coalesced into a 40 class semantic segmentation task by Gupta et al. [13]. We report results
on the standard split of 795 training images and 654 testing
images. (Note: all model selection is performed on PASCAL 2011 val.) Table 4 gives the performance of our model


key: chroma:document
string_value:
on the standard split of 795 training images and 654 testing
images. (Note: all model selection is performed on PASCAL 2011 val.) Table 4 gives the performance of our model
in several variations. First we train our unmodiﬁed coarse
model (FCN-32s) on RGB images. To add depth information, we train on a model upgraded to take four-channel
RGB-D input (early fusion). This provides little beneﬁt,
perhaps due to the difﬁcultly of propagating meaningful
gradients all the way through the model. Following the success of Gupta et al. [14], we try the three-dimensional HHA
encoding of depth, training nets on just this information, as
well as a “late fusion” of RGB and HHA where the predictions from both nets are summed at the ﬁnal layer, and the
resulting two-stream net is learned end-to-end. Finally we
upgrade this late fusion net to a 16-stride version.
SIFT Flow is a dataset of 2,688 images with pixel labels
for 33 semantic categories (“bridge”, “mountain”, “sun”),


key: chroma:document
string_value:
SIFT Flow is a dataset of 2,688 images with pixel labels
for 33 semantic categories (“bridge”, “mountain”, “sun”),
as well as three geometric categories (“horizontal”, “vertical”, and “sky”). An FCN can naturally learn a joint representation that simultaneously predicts both types of labels.
We learn a two-headed version of FCN-16s with semantic and geometric prediction layers and losses. The learned
model performs as well on both tasks as two independently
trained models, while learning and inference are essentially
as fast as each independent model by itself. The results in
Table 5, computed on the standard split into 2,488 training
and 200 test images,10show state-of-the-art performance on
both tasks.
10Three of the SIFT Flow categories are not present in the test set. We
made predictions across all 33 categories, but only included categories actually present in the test set in our evaluation. (An earlier version of this paper reported a lower mean IU, which included all categories either present
or predicted in the evaluation.)Table 5. Results on SIFT Flow10with class segmentation
(center) and geometric segmentation (right). Tighe [33] is
a non-parametric transfer method. Tighe 1 is an exemplar


key: chroma:document
string_value:
(center) and geometric segmentation (right). Tighe [33] is
a non-parametric transfer method. Tighe 1 is an exemplar
SVM while 2 is SVM + MRF. Farabet is a multi-scale convnet trained on class-balanced samples (1) or natural frequency
samples (2). Pinheiro is a multi-scale, recurrent convnet, denoted RCNN 3(3). The metric for geometry is pixel accuracy.
pixel
acc.mean
acc.mean
IUf.w.
IUgeom.
acc.
Liuet al. [23] 76.7 - - - Tighe et al. [33] - - - - 90.8
Tighe et al. [34] 1 75.6 41.1 - - Tighe et al. [34] 2 78.6 39.2 - - Farabet et al. [8] 1 72.3 50.8 - - Farabet et al. [8] 2 78.5 29.6 - - Pinheiro et al. [28] 77.7 29.8 - - FCN-16s 85.2 51.7 39.5 76.1 94.3
FCN-8s SDS [16] Ground Truth Image


key: chroma:document
string_value:
FCN-8s SDS [16] Ground Truth Image
Figure 6. Fully convolutional segmentation nets produce stateof-the-art performance on PASCAL. The left column shows the
output of our highest performing net, FCN-8s. The second shows
the segmentations produced by the previous state-of-the-art system
by Hariharan et al. [16]. Notice the ﬁne structures recovered (ﬁrst
row), ability to separate closely interacting objects (second row),
and robustness to occluders (third row). The fourth row shows a
failure case: the net sees lifejackets in a boat as people.
6. Conclusion
Fully convolutional networks are a rich class of models, of which modern classiﬁcation convnets are a special case. Recognizing this, extending these classiﬁcation
nets to segmentation, and improving the architecture with
multi-resolution layer combinations dramatically improves
the state-of-the-art, while simultaneously simplifying and
speeding up learning and inference.
Acknowledgements This work was supported in part
by DARPA’s MSEE and SMISC programs, NSF awards IIS1427425, IIS-1212798, IIS-1116411, and the NSF GRFP,


key: chroma:document
string_value:
Toyota, and the Berkeley Vision and Learning Center. We
gratefully acknowledge NVIDIA for GPU donation. We
thank Bharath Hariharan and Saurabh Gupta for their advice and dataset tools. We thank Sergio Guadarrama for
reproducing GoogLeNet in Caffe. We thank Jitendra Malik
for his helpful comments. Thanks to Wei Liu for pointing
out an issue wth our SIFT Flow mean IU computation and
an error in our frequency weighted mean IU formula.
A. Upper Bounds on IU
In this paper, we have achieved good performance on
the mean IU segmentation metric even with coarse semantic
prediction. To better understand this metric and the limits
of this approach with respect to it, we compute approximate
upper bounds on performance with prediction at various
scales. We do this by downsampling ground truth images
and then upsampling them again to simulate the best results
obtainable with a particular downsampling factor. The following table gives the mean IU on a subset of PASCAL
2011 val for various downsampling factors.
factor mean IU
128 50.9
64 73.3
32 86.1
16 92.8
896.4
498.5
Pixel-perfect prediction is clearly not necessary to


key: chroma:document
string_value:
128 50.9
64 73.3
32 86.1
16 92.8
896.4
498.5
Pixel-perfect prediction is clearly not necessary to
achieve mean IU well above state-of-the-art, and, conversely, mean IU is a not a good measure of ﬁne-scale accuracy.
B. More Results
We further evaluate our FCN for semantic segmentation.
PASCAL-Context [26] provides whole scene annotations of PASCAL VOC 2010. While there are over 400 distinct classes, we follow the 59 class task deﬁned by [26] that
picks the most frequent classes. We train and evaluate on
the training and val sets respectively. In Table 6, we compare to the joint object + stuff variation of Convolutional
Feature Masking [3] which is the previous state-of-the-art
on this task. FCN-8s scores 35.1 mean IU for an 11% relative improvement.
Changelog
The arXiv version of this paper is kept up-to-date with
corrections and additional relevant material. The following
gives a brief history of changes.Table 6. Results on PASCAL-Context. CFM is the best result of
[3] by convolutional feature masking and segment pursuit with the


key: chroma:document
string_value:
[3] by convolutional feature masking and segment pursuit with the
VGG net. O2Pis the second order pooling method [1] as reported
in the errata of [26]. The 59 class task includes the 59 most frequent classes while the 33 class task consists of an easier subset
identiﬁed by [26].
59 classpixel
acc.mean
acc.mean
IUf.w.
IU
O2P - - 18.1 CFM - - 31.5 FCN-32s 63.8 42.7 31.8 48.3
FCN-16s 65.7 46.2 34.8 50.7
FCN-8s 65.9 46.5 35.1 51.0
33 class
O2P - - 29.2 CFM - - 46.1 FCN-32s 69.8 65.1 50.4 54.9
FCN-16s 71.8 68.0 53.4 57.5
FCN-8s 71.8 67.6 53.5 57.7
v2Add Appendix A giving upper bounds on mean IU and
Appendix B with PASCAL-Context results. Correct PASCAL validation numbers (previously, some val images were
included in train), SIFT Flow mean IU (which used an inappropriately strict metric), and an error in the frequency


key: chroma:document
string_value:
included in train), SIFT Flow mean IU (which used an inappropriately strict metric), and an error in the frequency
weighted mean IU formula. Add link to models and update
timing numbers to reﬂect improved implementation (which
is publicly available).
References
[1] J. Carreira, R. Caseiro, J. Batista, and C. Sminchisescu. Semantic segmentation with second-order pooling. In ECCV ,
2012. 9
[2] D. C. Ciresan, A. Giusti, L. M. Gambardella, and J. Schmidhuber. Deep neural networks segment neuronal membranes
in electron microscopy images. In NIPS , pages 2852–2860,
2012. 1, 2, 4, 7
[3] J. Dai, K. He, and J. Sun. Convolutional feature masking for joint object and stuff segmentation. arXiv preprint
arXiv:1412.1283 , 2014. 9
[4] J. Donahue, Y . Jia, O. Vinyals, J. Hoffman, N. Zhang,
E. Tzeng, and T. Darrell. DeCAF: A deep convolutional activation feature for generic visual recognition. In ICML , 2014.
1, 2


key: chroma:document
string_value:
E. Tzeng, and T. Darrell. DeCAF: A deep convolutional activation feature for generic visual recognition. In ICML , 2014.
1, 2
[5] D. Eigen, D. Krishnan, and R. Fergus. Restoring an image
taken through a window covered with dirt or rain. In Computer Vision (ICCV), 2013 IEEE International Conference
on, pages 633–640. IEEE, 2013. 2
[6] D. Eigen, C. Puhrsch, and R. Fergus. Depth map prediction
from a single image using a multi-scale deep network. arXiv
preprint arXiv:1406.2283 , 2014. 2
[7] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn,
and A. Zisserman. The PASCAL Visual Object Classes
Challenge 2011 (VOC2011) Results. http://www.pascalnetwork.org/challenges/VOC/voc2011/workshop/index.html.
4
[8] C. Farabet, C. Couprie, L. Najman, and Y . LeCun. Learning
hierarchical features for scene labeling. Pattern Analysis and
Machine Intelligence, IEEE Transactions on , 2013. 1, 2, 4,
7, 8


key: chroma:document
string_value:
hierarchical features for scene labeling. Pattern Analysis and
Machine Intelligence, IEEE Transactions on , 2013. 1, 2, 4,
7, 8
[9] P. Fischer, A. Dosovitskiy, and T. Brox. Descriptor matching
with convolutional neural networks: a comparison to SIFT.
CoRR , abs/1405.5769, 2014. 1
[10] L. Florack, B. T. H. Romeny, M. Viergever, and J. Koenderink. The gaussian scale-space paradigm and the multiscale local jet. International Journal of Computer Vision ,
18(1):61–75, 1996. 5
[11] Y . Ganin and V . Lempitsky. N4-ﬁelds: Neural network nearest neighbor ﬁelds for image transforms. In ACCV , 2014. 1,
2, 7
[12] R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature hierarchies for accurate object detection and semantic
segmentation. In Computer Vision and Pattern Recognition ,
2014. 1, 2, 7
[13] S. Gupta, P. Arbelaez, and J. Malik. Perceptual organization
and recognition of indoor scenes from RGB-D images. In


key: chroma:document
string_value:
[13] S. Gupta, P. Arbelaez, and J. Malik. Perceptual organization
and recognition of indoor scenes from RGB-D images. In
CVPR , 2013. 8
[14] S. Gupta, R. Girshick, P. Arbelaez, and J. Malik. Learning
rich features from RGB-D images for object detection and
segmentation. In ECCV . Springer, 2014. 1, 2, 8
[15] B. Hariharan, P. Arbelaez, L. Bourdev, S. Maji, and J. Malik.
Semantic contours from inverse detectors. In International
Conference on Computer Vision (ICCV) , 2011. 7
[16] B. Hariharan, P. Arbel ´aez, R. Girshick, and J. Malik. Simultaneous detection and segmentation. In European Conference on Computer Vision (ECCV) , 2014. 1, 2, 4, 5, 7, 8
[17] K. He, X. Zhang, S. Ren, and J. Sun. Spatial pyramid pooling
in deep convolutional networks for visual recognition. In
ECCV , 2014. 1, 2


key: chroma:document
string_value:
in deep convolutional networks for visual recognition. In
ECCV , 2014. 1, 2
[18] Y . Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell. Caffe: Convolutional architecture for fast feature embedding. arXiv preprint
arXiv:1408.5093 , 2014. 7
[19] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet
classiﬁcation with deep convolutional neural networks. In
NIPS , 2012. 1, 2, 3, 5
[20] Q. V . Le, R. Monga, M. Devin, K. Chen, G. S. Corrado,
J. Dean, and A. Y . Ng. Building high-level features using
large scale unsupervised learning. In ICML , 2012. 3
[21] Y . LeCun, B. Boser, J. Denker, D. Henderson, R. E. Howard,
W. Hubbard, and L. D. Jackel. Backpropagation applied to
hand-written zip code recognition. In Neural Computation ,
1989. 2, 3


key: chroma:document
string_value:
W. Hubbard, and L. D. Jackel. Backpropagation applied to
hand-written zip code recognition. In Neural Computation ,
1989. 2, 3
[22] Y . A. LeCun, L. Bottou, G. B. Orr, and K.-R. M ¨uller. Efﬁcient backprop. In Neural networks: Tricks of the trade ,
pages 9–48. Springer, 1998. 7
[23] C. Liu, J. Yuen, and A. Torralba. Sift ﬂow: Dense correspondence across scenes and its applications. Pattern Analysis
and Machine Intelligence, IEEE Transactions on , 33(5):978–
994, 2011. 8[24] J. Long, N. Zhang, and T. Darrell. Do convnets learn correspondence? In NIPS , 2014. 1
[25] O. Matan, C. J. Burges, Y . LeCun, and J. S. Denker. Multidigit recognition using a space displacement neural network.
InNIPS , pages 488–495. Citeseer, 1991. 2


key: chroma:document
string_value:
InNIPS , pages 488–495. Citeseer, 1991. 2
[26] R. Mottaghi, X. Chen, X. Liu, N.-G. Cho, S.-W. Lee, S. Fidler, R. Urtasun, and A. Yuille. The role of context for object
detection and semantic segmentation in the wild. In Computer Vision and Pattern Recognition (CVPR), 2014 IEEE
Conference on , pages 891–898. IEEE, 2014. 9
[27] F. Ning, D. Delhomme, Y . LeCun, F. Piano, L. Bottou, and
P. E. Barbano. Toward automatic phenotyping of developing
embryos from videos. Image Processing, IEEE Transactions
on, 14(9):1360–1371, 2005. 1, 2, 4, 7
[28] P. H. Pinheiro and R. Collobert. Recurrent convolutional
neural networks for scene labeling. In ICML , 2014. 1, 2,
4, 7, 8
[29] P. Sermanet, D. Eigen, X. Zhang, M. Mathieu, R. Fergus,
and Y . LeCun. Overfeat: Integrated recognition, localization


key: chroma:document
string_value:
and Y . LeCun. Overfeat: Integrated recognition, localization
and detection using convolutional networks. In ICLR , 2014.
1, 2, 3, 4
[30] N. Silberman, D. Hoiem, P. Kohli, and R. Fergus. Indoor
segmentation and support inference from rgbd images. In
ECCV , 2012. 7
[31] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. CoRR ,
abs/1409.1556, 2014. 1, 2, 3, 5
[32] C. Szegedy, W. Liu, Y . Jia, P. Sermanet, S. Reed,
D. Anguelov, D. Erhan, V . Vanhoucke, and A. Rabinovich.
Going deeper with convolutions. CoRR , abs/1409.4842,
2014. 1, 2, 3, 5
[33] J. Tighe and S. Lazebnik. Superparsing: scalable nonparametric image parsing with superpixels. In ECCV , pages 352–
365. Springer, 2010. 8
[34] J. Tighe and S. Lazebnik. Finding things: Image parsing with


key: chroma:document
string_value:
365. Springer, 2010. 8
[34] J. Tighe and S. Lazebnik. Finding things: Image parsing with
regions and per-exemplar detectors. In CVPR , 2013. 8
[35] J. Tompson, A. Jain, Y . LeCun, and C. Bregler. Joint training
of a convolutional network and a graphical model for human
pose estimation. CoRR , abs/1406.2984, 2014. 2
[36] L. Wan, M. Zeiler, S. Zhang, Y . L. Cun, and R. Fergus. Regularization of neural networks using dropconnect. In Proceedings of the 30th International Conference on Machine
Learning (ICML-13) , pages 1058–1066, 2013. 4
[37] R. Wolf and J. C. Platt. Postal address block location using
a convolutional locator network. Advances in Neural Information Processing Systems , pages 745–745, 1994. 2
[38] M. D. Zeiler and R. Fergus. Visualizing and understanding
convolutional networks. In Computer Vision–ECCV 2014 ,
pages 818–833. Springer, 2014. 2


key: chroma:document
string_value:
convolutional networks. In Computer Vision–ECCV 2014 ,
pages 818–833. Springer, 2014. 2
[39] N. Zhang, J. Donahue, R. Girshick, and T. Darrell. Partbased r-cnns for ﬁne-grained category detection. In Computer Vision–ECCV 2014 , pages 834–849. Springer, 2014.
1


key: chroma:document
string_value:
Deep Filter Banks for Texture Recognition and Segmentation
Mircea Cimpoi
University of Oxford
mircea@robots.ox.ac.ukSubhransu Maji
University of Massachusetts, Amherst
smaji@cs.umass.eduAndrea Vedaldi
University of Oxford
vedaldi@robots.ox.ac.uk
Abstract
Research in texture recognition often concentrates on the
problem of material recognition in uncluttered conditions,
an assumption rarely met by applications. In this work we
conduct a ﬁrst study of material and describable texture attributes recognition in clutter, using a new dataset derived
from the OpenSurface texture repository. Motivated by the
challenge posed by this problem, we propose a new texture
descriptor, FV-CNN, obtained by Fisher Vector pooling of a
Convolutional Neural Network (CNN) ﬁlter bank. FV-CNN
substantially improves the state-of-the-art in texture, material and scene recognition. Our approach achieves 79.8%
accuracy on Flickr material dataset and 81% accuracy on
MIT indoor scenes, providing absolute gains of more than
10% over existing approaches. FV-CNN easily transfers
across domains without requiring feature adaptation as for
methods that build on the fully-connected layers of CNNs.


key: chroma:document
string_value:
10% over existing approaches. FV-CNN easily transfers
across domains without requiring feature adaptation as for
methods that build on the fully-connected layers of CNNs.
Furthermore, FV-CNN can seamlessly incorporate multiscale information and describe regions of arbitrary shapes
and sizes. Our approach is particularly suited at localizing “stuff” categories and obtains state-of-the-art results
on MSRC segmentation dataset, as well as promising results
on recognizing materials and surface attributes in clutter on
the OpenSurfaces dataset.
1. Introduction
Texture is ubiquitous and provides useful cues of material properties of objects and their identity, especially when
shape is not useful. Hence, a signiﬁcant amount of effort in
the computer vision community has gone into recognizing
texture via tasks such as texture perception [1, 2, 12, 13]
and description [7, 11], material recognition [26, 36, 37],
segmentation [20, 29], and even synthesis [9, 43].
Perhaps the most studied task in texture understanding
is the one of material recognition, as captured in benchmarks such as CuRET [8], KTH-TIPS [5], and, more recently, FMD [38]. However, while at least the FMD dataset
contains images collected from the Internet, vividly dubbed


key: chroma:document
string_value:
contains images collected from the Internet, vividly dubbed
“images in the wild”, all these datasets make the simplifying
banded
 blotchy
 chequered
 grid
marbled
 paisley
 paisley
 wrinkled
brick
 ceramic
 carpet
 fabric
Figure 1. Texture recognition in clutter . Example of top retrieved texture segments by attributes (top two rows) and materials
(bottom) in the OpenSurfaces dataset.
assumption that textures ﬁll images. Thus, they are not necessarily representative of the signiﬁcantly harder problem
of recognising materials in natural images, where textures
appear in clutter. Building on a recent dataset collected by
the computer graphics community, the ﬁrst contribution of
this paper is a large-scale analysis of material and perceptual texture attribute recognition and segmentation in clutter(Fig. 1 and Sect. 2).
Motivated by the challenge posed by recognising texture
in clutter, we develop a new texture descriptor. In the simplest terms a texture is characterized by the arrangement of
local patterns, as captured in early works [26, 41] by the
distribution of local “ﬁlter bank” responses. These ﬁlter
banks were designed to capture edges, spots and bars at
different scales and orientations. Typical combinations of


key: chroma:document
string_value:
banks were designed to capture edges, spots and bars at
different scales and orientations. Typical combinations of
the ﬁlter responses, identiﬁed by vector quantisation, were
used as the computational basis of the “textons” proposed
by Julesz [22]. Texton distributions were the early versions
of “bag-of-words” representations, a dominant approach in
recognition in the early 2000s, since then improved by new
pooling schemes such as soft-assignment [27, 42, 48] and
Fisher Vectors (FVs) [32]. Until recently, FV with SIFT
features [28] as a local representation was the state-of-theart method for recognition, not only for textures, but for
objects and scenes too.
Later, however, Convolutional Neural Networks (CNNs)
1arXiv:1411.6836v2  [cs.CV]  9 Jul 2015
have emerged as the new state-of-the-art for recognition, exempliﬁed by remarkable results in image classiﬁcation [23],
detection [14] and segmentation [16] on a number of widely
used benchmarks. Key to their success is the ability to leverage large labelled datasets to learn increasingly complex


key: chroma:document
string_value:
detection [14] and segmentation [16] on a number of widely
used benchmarks. Key to their success is the ability to leverage large labelled datasets to learn increasingly complex
transformations of the input to capture invariances. Importantly, CNNs pre-trained on such large datasets have been
shown [6, 14, 30] to contain general-purpose feature extractors, transferrable to many other domains.
Domain transfer in CNNs is usually achieved by using
as features the output of a deep, fully-connected layer of
the network. From the perspective of textures, however,
this choice has three drawbacks. The ﬁrst one (I) is that,
while the convolutional layers are akin to non-linear ﬁlter
banks, the fully connected layers capture their spatial layout. While this may be useful for representing the shape of
an object, it may not be as useful for representing texture. A
second drawback (II) is that the input to the CNN has to be
of ﬁxed size to be compatible with the fully connected layers, which requires an expensive resizing of the input image,
particularly when features are computed for many different
regions [14, 15]. A third and more subtle drawback (III) is
that deeper layers may be more domain-speciﬁc and therefore potentially less transferrable than shallower layers.


key: chroma:document
string_value:
that deeper layers may be more domain-speciﬁc and therefore potentially less transferrable than shallower layers.
The second contribution of this paper is FV-CNN
(Sect. 3), a pooling method that overcomes these drawbacks . The idea is to regard the convolutional layers of a
CNN as a ﬁlter bank and build an orderless representation
using FV as a pooling mechanism, as is commonly done
in the bag-of-words approaches. Although the suggested
change is simple, the approach is remarkably ﬂexible and
effective. First, pooling is orderless and multi-scale, hence
suitable for textures. Second, any image size can be processed by convolutional layers, avoiding costly resizing operations. Third, convolutional ﬁlters, pooled by FV-CNN,
are shown to transfer more easily than fully-connected ones
even without ﬁne-tuning. While other authors [15, 18] have
recently proposed alternative pooling strategies for CNNs,
we show that our method is more natural, faster and often
signiﬁcantly more accurate.
Thethird contribution of the paper is a thorough evaluation of these descriptors on a variety of benchmarks, from
textures to objects (Sect. 4). In textures, we evaluate material and describable attributes recognition and segmentation


key: chroma:document
string_value:
textures to objects (Sect. 4). In textures, we evaluate material and describable attributes recognition and segmentation
on new datasets derived from OpenSurfaces (Sect. 2). When
used with linear SVMs, FV-CNN improves the state of the
art on texture recognition by a signiﬁcant margin. Like textures, scenes are also weakly structured and a bag-of-words
representation is effective. FV-CNN obtains 81.1% accuracy on the MIT indoor scenes dataset [34], signiﬁcantly
outperforming the current state-of-the-art of 70.8% [47].
What is remarkable is that, where [47] ﬁnds that CNNs
trained on scene recognition data perform better than CNNs
trained on an object domain (ImageNet), when used in FV-CNN not only is there an overall performance improvement, but the domain-speciﬁc advantage is entirely removed
(Tab. 3). This indicates that FV-CNN are in fact better at domain transfer. Our method also matches the previous best in
PASCAL VOC 2007 classiﬁcation dataset providing measurable boost over CNNs and closely approaches competitor methods on CUB 2010-2011 datasets when ground-truth
object bounding boxes are given.
FV-CNN can be used for describing regions by simply


key: chroma:document
string_value:
object bounding boxes are given.
FV-CNN can be used for describing regions by simply
pooling across pixels within the region. Combined with
a low-level segmentation algorithm this sufﬁces to localize textures within images. This approach is similar to a
recently proposed method called “R-CNN” for localizing
objects [14]. However, in contrast to it we do not need repeated evaluations of the CNN since the convolutional features can be computed just once for the entire image and
pooled differently. This makes FV-CNN not only faster,
but also as experiments suggest, much more accurate at
texture localization. We achieve state of the art results on
the MSRC segmentation dataset using a simple scheme of
classifying “superpixels” obtaining an accuracy of 87.0%
(previous best 86.5%). The corresponding R-CNN obtains
57.7% accuracy. Segmentation results are promising in the
OpenSurfaces dataset [4] as well.
Finally, we analyze the utility of different network layers
and architectures as ﬁlter banks, concluding that: SIFT is
competitive only with the ﬁrst few layers of a CNN (Fig. 4)


key: chroma:document
string_value:
and architectures as ﬁlter banks, concluding that: SIFT is
competitive only with the ﬁrst few layers of a CNN (Fig. 4)
and that signiﬁcant improvement to the underlying CNN architecture, such as the ones achieved by the very deep models of Simonyan and Zisserman [39], directly translate into
much better ﬁlter banks for texture recognition.
2. Texture recognition in clutter
A contribution of this work is the analysis of materials
and texture attributes in realistic imaging conditions. Earlier datasets such as KTH-TIPS were collected in controlled
conditions, which makes their applicability to natural images unclear. More recent datasets such as FMD and DTD
remove this limitation by building on images downloaded
from the Internet, dubbed images “in the wild”. However,
in these datasets texture always ﬁll the ﬁeld of view of the
camera. In this paper we remove this limitation by experimenting for the ﬁrst time with a large dataset of textures
collected in the wild and in cluttered conditions.
In particular, we build on the Open Surfaces (OS) dataset
that was recently introduced by Bell et al. [4] in computer
graphics. OS comprises 25,357 images, each containing a
number of high-quality texture/material segments. Many of


key: chroma:document
string_value:
graphics. OS comprises 25,357 images, each containing a
number of high-quality texture/material segments. Many of
these segments are annotated with additional attributes such
as the material name, the viewpoint, the BRDF, and the object class. Not all segments have a complete set of annotations; the experiments in this paper focus on the 58,928 that
contain material names. Since material classes are highly
unbalanced, only the materials that contain at least 400 examples are considered. This result in 53,915 annotated
material segments in 10,422 images spanning 22 different
classes.1Images are split evenly into training, validation,
and test subsets with 3,474 images each. Segment sizes are
highly variable, with half of them being relatively small,
with an area smaller than 6464pixels. While the lack
of exhaustive annotations makes it impossible to deﬁne a
complete background class, several less common materials
(including for example segments that annotators could not
assign to a material) are merged in an “other” class that acts
as pseudo-background.
In order to study perceptual properties as well as materials, we augment the OS dataset with some of the 47
attributes from the DTD [7]. Since the OS segments do
not trigger with sufﬁcient frequency all the 47 attributes,


key: chroma:document
string_value:
attributes from the DTD [7]. Since the OS segments do
not trigger with sufﬁcient frequency all the 47 attributes,
the evaluation is restricted to eleven of them for which it
was possible to identify at least 100 matching segments.2
The attributes were manually labelled in the 53,915 segments retained for materials. We refer to this data as
OSA. The complete list of images, segments, labels, and
splits are available at http://www.robots.ox.ac.
uk/˜vgg/data/wildtex/ .
3. Method
This section describes the methodological contributions
of this paper: region description and segmentation.
3.1. Region description
This section introduces a number of visual descriptors
suitable to model the appearance of image regions. Texture
is traditionally described by orderless pooling of ﬁlter bank
responses as, unlike in objects, the overall shape information is usually unimportant. However, small under-sampled
textures may beneﬁt if recognized in the context of an object. Thus, the primacy of orderless pooling may not always
hold in the recognition of textures in natural conditions.
In order to explore the interplay between shape and orderless pooling, we evaluate two corresponding region descriptors: FC-CNN for shape and FV-CNN for texture.


key: chroma:document
string_value:
In order to explore the interplay between shape and orderless pooling, we evaluate two corresponding region descriptors: FC-CNN for shape and FV-CNN for texture.
Both descriptors are based on the same CNN features [23]
obtained from an off-the-shelf CNN pre-trained on the ImageNet ILSVRC 2012 data as suggested in [6, 21, 35]. Since
the underlying CNN is the same, it is meaningful to compare FC- and FV-CNN directly.
1The classes and corresponding number of example segments are:
brick (610), cardboard (423), carpet/rug (1,975), ceramic (1,643), concrete (567), fabric/cloth (7,484), food (1,461), glass (4,571), granite/marble (1,596), hair (443), other (2,035), laminate (510), leather (957),
metal (4,941), painted (7,870), paper/tissue (1,226), plastic/clear (586),
plastic/opaque (1,800), stone (417), tile (3,085), wallpaper (483), wood
(9,232).
2These are: banded, blotchy, chequered, ﬂecked, gauzy, grid, marbled,


key: chroma:document
string_value:
(9,232).
2These are: banded, blotchy, chequered, ﬂecked, gauzy, grid, marbled,
paisley, pleated, stratiﬁed, wrinkled.Object descriptor: FC-CNN. The FC-CNN descriptor is
obtained by extracting as features the output of the penultimate Fully-Connected (FC) layer of a CNN, including
the non-linear gating function, applied to the input image.
This can be considered an object descriptor because the
fully connected layers allow FC-CNN to capture the overall
shape of the object contained in the region. FC-CNN is applied to an image region R(which may be the whole image)
by warping the bounding box enclosing R(plus a 10% border) to a square of a ﬁxed size matching the default CNN
input geometry, obtaining the same R-CNN descriptor introduced by Girshick et al. [14] as a state-of-the-art object
detector in the PASCAL VOC [10] data.
Texture descriptor: FV-CNN. The FV-CNN descriptor
is inspired by the state-of-the-art texture descriptors of [7]
based on the Fisher Vector (FV). Differently from FC-CNN,
FV pools local features densely within the described regions


key: chroma:document
string_value:
based on the Fisher Vector (FV). Differently from FC-CNN,
FV pools local features densely within the described regions
removing global spatial information , and is therefore more
apt at describing textures than objects. Here FV is computed
on the output of a single (last) convolutional layer of the
CNN, but we compared features from other layers as well
(Sect 4.4). By avoiding the computation of the fully connected layers, the input image does not need to be rescaled
to a speciﬁc size; in fact, the dense convolutional features
are extracted at multiple scales and pooled into a single FV
just like for SIFT. The pooled convolutional features are extracted immediately after the last linear ﬁltering operator
and are not otherwise normalised.
The FV-CNN descriptor is related to the one proposed
by Gong et al. [15]. There VLAD pooling, which is similar to FV , is applied to FC-CNN-like descriptors extracted
from densely sampled image windows. A key difference of
FV-CNN is that dense features are extracted from the convolutional rather than fully-connected layers. This is more
natural, signiﬁcantly more efﬁcient (as it does not require


key: chroma:document
string_value:
natural, signiﬁcantly more efﬁcient (as it does not require
recomputing the network for each extracted local descriptor) and, as shown in Sect. 4, more accurate.
3.2. Region segmentation
This section discusses our method to automatically partition an image into a number of recognisable regions. Inspired by Cimpoi et al. [7] that successfully ported object
description methods to texture descriptors, here we propose
a segmentation technique inspired by object detection. An
increasingly popular method for object detection, followed
for example by R-CNN [14], is to ﬁrst propose a number of
candidate object regions using low-level image cues, and
then verifying a shortlist of such regions using a powerful classiﬁer. Applied to textures, this requires a low-level
mechanism to generate textured region proposals, followed
by a region classiﬁer. A key advantage of this approach
is that it allows applying object- (FC-CNN) and texturelike (FV-CNN) descriptors alike. After proposal classiﬁcation, each pixel can be assigned more than one label; this is
solved with simple voting schemes, also inspired by object
detections methods.
The paper explores two such region generation methods:


key: chroma:document
string_value:
solved with simple voting schemes, also inspired by object
detections methods.
The paper explores two such region generation methods:
the crisp regions of [19] and the multi-scale combinatorial
grouping of [3]. In both cases, region proposals are generated using low-level image cues, such as colour or texture
consistency, as speciﬁed by the original methods. It would
of course be possible to incorporate FC-CNN and FV-CNN
among these energy terms to potentially strengthen the region generation mechanism itself. However, this contradicts partially the logic of the scheme, which breaks down
the problem into cheaply generating tentative segmentations and then verifying them using a more powerful (and
likely expensive) model. Furthermore, and more importantly, these cues focus on separating texture instances , as
presented in each particular image, whereas FC-CNN and
FV-CNN are meant to identify a texture class. It is reasonable to expect instance-speciﬁc cues (say the colour of a
painted wall) to be better for segmentation.
4. Results
This section evaluates the proposed region recognition
methods for classifying and segmenting materials, describable texture properties, and higher-level object categories.
Sect. 4.1 evaluates the classiﬁcation task by assessing how
well regions can be classiﬁed given that their true extent is


key: chroma:document
string_value:
Sect. 4.1 evaluates the classiﬁcation task by assessing how
well regions can be classiﬁed given that their true extent is
known and Sect. 4.3 evaluates both classiﬁcation and segmentation . The rest of the section introduces the evaluation
benchmarks and technical details of the representations.
Datasets. The evaluation considers three texture recognition benchmarks other than OS (Sect. 2). The ﬁrst one is the
Flickr Material Dataset (FMD) [37], a recent benchmark
containing 10 material classes. The second one is the Describable Texture Datasets (DTD) [7], which contains texture images jointly annotated with 47 describable attributes
drawn from the psychological literature. Both FMD and
DTD contain images “in the wild”, i.e. collected in uncontrolled conditions. However, differently from OS, these images are uncluttered. The third texture dataset is KTH-TIPS2b[5, 17], containing a number of example images for each
of four samples of 11 material categories. For each material, images of one sample are used for training and the
remaining for testing.
Object categorisation is evaluated in the PASCAL VOC
2007 [10] dataset, containing 20 object categories, any
combination of which may be associated to any of the


key: chroma:document
string_value:
Object categorisation is evaluated in the PASCAL VOC
2007 [10] dataset, containing 20 object categories, any
combination of which may be associated to any of the
benchmark images. Scene categorisation uses the MIT Indoor [34] dataset, containing 67 indoor scene classes. Finegrained categorisation uses the Caltech/UCSD Bird dataset
(CUB) [45], containing images of 200 bird species.
Note that some of these datasets come with ground truth
region/object localisation. The +R sufﬁx will be appendedto a dataset to indicate that this information is used both at
training and testing time. For example, OS means that segmentation is performed automatically at test time, whereas
OS+R means that ground-truth segmentations are used.
Evaluation measures. For each dataset the corresponding standard evaluator protocols and accuracy measures are
used. In particular, for FMD, DTD, MIT Indoor, CUB,
and OS+R, evaluation uses average classiﬁcation accuracy,
per-segment/image and normalized for each class. When
evaluating the quality of a segmentation algorithm, however, one must account for the fact that in most datasets,
and in OS and MSRC in particular, not all pixels are labelled. In this case, accuracy is measured per-pixel rather


key: chroma:document
string_value:
and in OS and MSRC in particular, not all pixels are labelled. In this case, accuracy is measured per-pixel rather
than per-segment, ignoring all pixels that are unlabelled in
the ground truth. For MSRC, furthermore, accuracy is normalised across all pixels regardless of their category. For
OSA, since some segments may have more than one label,
we are reporting mAP, following the standard procedure for
multi-label datasets. Finally, PASCAL VOC 2007 classiﬁcation uses mean average precision (mAP), computed using
the TRECVID 11-point interpolation [10].3
Descriptor details. FC-CNN and FV-CNN build on the
pre-trained VGG-M [6] model as this performs better than
other popular models such as [21] while having a similar computational cost. This network results in 4096dimensional FC-CNN features and 512-dimensional local
features for FV-CNN computation. The latter are pooled
into a FV representation with 64 Gaussian components,
resulting in 65K-dimensional descriptors. While the FVCNN dimensionality is much higher than the 4K dimensions of FC-CNN, the FV is known to be highly redundant and can be typically compressed by one order of magnitude without appreciable reduction in the classiﬁcation


key: chroma:document
string_value:
performance [31], so the effective dimensionality of FCand FV-CNN is likely comparable. We veriﬁed that by
PCA-reducing FV to 4096 dimensions and observing only
a marginal reduction in classiﬁcation performance in the
PASCAL VOC object recognition task described below. In
addition to VGG-M, the recent state-of-the art VGG-VD
(very deep with 19 layers) model of Simonyan and Zisserman [39] is also evaluated.
Due to the similarity between FV-CNN and the dense
SIFT FV descriptors used for texture recognition in [7],
the latter is evaluated as well. Since SIFT descriptors
are smaller (128-dimensional) than the convolutional ones
(512-dimensional), a larger number of Gaussian components (256) are used to obtain FV descriptors with a comparable dimensionality. The SIFT descriptors support is
3232pixels at the base scale.
In order to make results comparable to [7], we use
the same settings whenever possible. FV-CNN and DSIFT compute features after rescaling the image by factors
3The deﬁnition of AP was changed in later versions of the benchmark.


key: chroma:document
string_value:
3The deﬁnition of AP was changed in later versions of the benchmark.
dataset meas.IFVVGG-M VGG-VD FV-SIFTSoA(%) FC FV FC+FV FC FV FC+FV FC+FV-VD
(a)KTH-T2b acc 70.82.7712.373.34.773.94.975.41.581.82.581.12.4 81.52.0 76.02.9[40]
FMD acc 59.81.670.31.873.52.076.61.977.41.879.81.882.41.5 82.21.4 57.71.7[33, 37]
OS+R acc 39.8 41.3 52.5 54.9 43.4 59.5 60.9 58.7 –
(b)DTD acc 58.61.258.80.866.81.669.81.162.90.872.31.074.71.0 75.50.8 –


key: chroma:document
string_value:
OSA+R mAP 56.5 54.3 65.2 67.9 49.7 67.2 67.9 68.2 –
(d)MSRC+R acc 85.7 85.0 95.4 96.9 79.4 97.7 98.8 99.1 –
MSRC+R msrc-acc 92.0 84.0 97.6 98.1 82.0 99.2 99.6 99.5 –
VOC07 mAP11 59.9 76.8 76.4 79.5 81.7 84.9 85.1 84.5 85.2 [44]
MIT Indoor acc 54.9 62.5 74.2 74.4 67.6 81.0 80.3 80.0 70.8 [47]
CUB acc 17.5 46.1 49.9 54.9 54.6 66.7 67.3 65.4 73.9 (62.8) [46]
CUB+R acc 27.7 56.5 65.5 68.1 62.8 73.0 74.9 73.6 76.37 [46]
Table 1. Evaluation of texture descriptors. The table compares FC-CNN, FV-CNN on two networks trained on ImageNet – VGG-M and


key: chroma:document
string_value:
Table 1. Evaluation of texture descriptors. The table compares FC-CNN, FV-CNN on two networks trained on ImageNet – VGG-M and
VGG-VD, and IFV on dense SIFT. We evaluated these descriptors on (a) material datasets (FMD, KTH-T2b, OS+R), (b) texture attributes
(DTD, OSA+R) and (c) general categorisation datasets (MSRC+R,VOC07,MIT Indoor) and ﬁne grained categorisation (CUB, CUB+R).
For this experiment the region support is assumed to be known (and equal to the entire image for all the datasets except OS+R and MSRC+R
and for CUB+R, where it is set to the bounding box of a bird). () using a model without parts. Best results are marked in bold.
VGG-M VGG-VD
dataset measure (%) FC-CNN FV-CNN FV+FC-CNN FC-CNN FV-CNN FC+FV-CNN SoA
OS pp-acc 36.3 48.7 (46.9) 50.5 38.8 55.4 (55.7) 55.2 –


key: chroma:document
string_value:
OS pp-acc 36.3 48.7 (46.9) 50.5 38.8 55.4 (55.7) 55.2 –
MSRC msrc-acc 56.1 82.3 75.5 57.7 87.0 80.4 86.5 [24]
Table 2. Segmentation and recognition using crisp region proposals of materials (OS) and things & stuff (MSRC). Per-pixel accuracies are
reported, using the MSRC variant (see text) for the MSRC dataset. Results using MCG proposals [3] are reported in brackets for FV-CNN.
2s; s=3;2:5; : : :1:5(but, for efﬁciency, discarding
scales that would make the image larger than 10242pixels). Before pooling descriptors with a FV , these are usually
de-correlated by using PCA. Here PCA is applied to SIFT,
additionally reducing its dimension to 80, as this was empirically shown to improve the overall recognition performance. However, PCA is not applied to the convolutional
features in FV-CNN as in this case results were worse.
Learning details. The region descriptors (FC-CNN, FVCNN, and D-SIFT) are classiﬁed using 1-vs-rest Support


key: chroma:document
string_value:
Learning details. The region descriptors (FC-CNN, FVCNN, and D-SIFT) are classiﬁed using 1-vs-rest Support
Vector Machine (SVM) classiﬁers. Prior to learning, descriptors are L2normalised and the learning constant set
toC= 1. This is motivated by the fact that, after data
normalisation, the exact choice of Chas a negligible effect
on performance. Furthermore, the accuracy of the 1-vs-rest
classiﬁcation scheme is improved by recalibrating the SVM
scores after training, by scaling the SVM weight vector and
bias such that the median scores of the negative and positive
training samples for each class are mapped respectively to
the values1and1.
4.1. Region recognition: textures
This and the following section evaluate region recognition assuming that the ground-truth region Ris known (Table 1), for example because it ﬁlls the entire image. This
section focuses on textures (materials and perceptual attributes), while the next on objects and scenes.
Texture recognition without clutter. This experimentevaluates the performance of FC-CNN, FV-CNN, D-SIFT,
and their combinations in standard texture recognition


key: chroma:document
string_value:
Texture recognition without clutter. This experimentevaluates the performance of FC-CNN, FV-CNN, D-SIFT,
and their combinations in standard texture recognition
benchmarks such as FMD, KTH-TIPS-2, and DTD. FCCNN is roughly equivalent to the DeCAF method used
in [7] for this data as regions ﬁll images; however, while the
performance of our FC-CNN is similar in KTH ( 70%),
it is substantially better in FMD ( 60:7%!70:4%accuracy) and DTD ( 54:8%!58:7%). This likely is caused by
the improved underlying CNN, an advantage which is more
obvious in FMD and DTD that are closer to object recognition than KTH. FV-CNN performs within 2%in FMD and
KTH but substantially better for DTD ( 58:7%!66:6%).
D-SIFT is comparable in performance to FC-CNN in DTD
and KTH, but substantially worse ( 70:4%!59:2%) in
FMD. Our conclusion is that, even when textures ﬁll the
input image as in these benchmarks, orderless pooling in
FV-CNN and D-SIFT can be either the same or substantially better than the pooling operated in the fully-connected


key: chroma:document
string_value:
input image as in these benchmarks, orderless pooling in
FV-CNN and D-SIFT can be either the same or substantially better than the pooling operated in the fully-connected
layers by FC-CNN.
Combining FC- and FV-CNN improves performance in
all datasets by 13%. While this combination is already signiﬁcantly above the state-of-the-art in DTD and
FMD ( +2:6%=11:2%), the method of [7] still outperforms
these descriptors in KTH. However, replacing VGG-M with
VGG-VD signiﬁcantly improves the performance in all
cases – a testament to the power of deep features. In particular, the best method FC+FV-CNN-VD, improves the state
of the art by at least 6%in all datasets. Interestingly, this is
obtained by using a single low-level feature type as FC- and
FV-CNN build on the same convolutional features. Adding
D-SIFT results in at most 1%improvement, and in some
cases it slightly degrades performance.
Texture recognition in clutter. The advantage of FV-CNN
over FC-CNN is much larger when textures do not ﬁll the


key: chroma:document
string_value:
cases it slightly degrades performance.
Texture recognition in clutter. The advantage of FV-CNN
over FC-CNN is much larger when textures do not ﬁll the
image but are extracted from clutter. In OS+R (Sect. 2),
material recognition accuracy starts at about 46% for both
FC-CNN and D-SIFT; however, FV-CNN improves this by
more than 11% (46:5%!58:1%). The combination of FCand FV-CNN improves results further by 2%, but adding
SIFT deteriorates performance. With the very deep CNN
conclusions are similar; however, switching to VGG-VD
barely affects the FC-CNN performance ( 46:5!48:0%),
but strongly affects the one of FV-CNN ( 58:1%!65:1%).
This conﬁrms that FC-CNN, while excellent in object detection, is not a very good descriptor for classifying textured
regions. Results in OSA+R for texture attribute recognition
(Sect. 2) and in MSRC+R for semantic segmentation are
analogous; it is worth noting that, when ground-truth segments are used in this experiment, the best model achieves
a nearly perfect 99:7%classiﬁcation rate in MSRC.


key: chroma:document
string_value:
a nearly perfect 99:7%classiﬁcation rate in MSRC.
4.2. Region recognition: objects and scenes
This section shows that the FV-CNN descriptor, despite
its orderless nature that make it an excellent texture descriptor, excels at object and scene recognition as well. In the remainder of the section, and unless otherwise noted, region
descriptors are applied to images as a whole by considering
these single regions.
FV-CNN vs FC-CNN. As seen in Table 1, in PASCAL
VOC and MIT Indoor the FC-CNN descriptor performs
very well but in line with previous results for this class of
methods [6]. FV-CNN performs similarly to FC-CNN in
PASCAL VOC, but substantially better ( +5% for VGG-M
and +13% for VGG-VD) in MIT Indoor. As further discussed below, this is an example of the ability of FV-CNN
to transfer between domains better than FC-CNN. The gap
between FC-CNN and FV-CNN is the highest for the very
deep VGG-VD models (68.1% !81.1%), a trend also exhibited by other datasets as seen in Tab. 1. In the CUB
dataset, FV-CNN signiﬁcantly outperforms FC-CNN both


key: chroma:document
string_value:
dataset, FV-CNN signiﬁcantly outperforms FC-CNN both
whether the descriptor is computed from the whole image
(CUB) or from the bird bounding box (CUB+R). In the latter case, the difference is very large ( +1014%).
Comparison with alternative pooling methods. FV-CNN
is related to the method of [15], which uses a similar
underlying CNN and VLAD instead of FV for pooling.
Notably, however, FV-CNN results on MIT Indoor are
markedly better than theirs for both VGG-M and VGGVD ( 68:8%!73:5%=81:1%resp.) and marginally better (68:8%!69:1%) when the same CAFFE CNN is used
(Tab. 3). The key difference is that FV-CNN pools convolu-Accuracy (%)
CNN FC-CNN FV-CNN FC+FV-CNN
PLACES 65.0 67.6 73.1
CAFFE 58.6 69.7 71.6
VGG-M 62.5 74.2 74.4
VGG-VD 67.6 81.0 80.3
Table 3. Accuracy of various CNNs on the MIT indoor dataset.


key: chroma:document
string_value:
VGG-VD 67.6 81.0 80.3
Table 3. Accuracy of various CNNs on the MIT indoor dataset.
tional features, whereas [15] pools fully-connected descriptors extracted from square image patches. Thus, even without spatial information as used by [15], FV-CNN is not only
substantially faster, but at least as accurate. Using the same
settings, that is, the same net and the same three scales, our
approach results in an 8.5 speedup.
Comparison with the state-of-the-art. The best result obtained in PASCAL VOC is comparable to the current stateof-the-art set by the deep learning method of [44] ( 85:2%!
85:0%), but using a much more straightforward pipeline. In
MIT Places our best performance is also substantially superior ( +10% ) to the current state-of-the-art using deep convolutional networks learned on the MIT Place dataset [47]
(see also below). In the CUB dataset, our best performance
is a little short (3%) of the state-of-the-art results of [46].
However, [46] uses a category-speciﬁc part detector and
corresponding part descriptor as well as a CNN ﬁne-tuned


key: chroma:document
string_value:
However, [46] uses a category-speciﬁc part detector and
corresponding part descriptor as well as a CNN ﬁne-tuned
on the CUB data; by contrast, FV-CNN and FC-CNN are
used here as global image descriptors which, furthermore,
are the same for all the datasets considered . Compared to
the results of [46] without part-based descriptors (but still
using a part-based object detector), our global image descriptors perform substantially better ( 62:1%!69:1%).
We conclude that FV-CNN is a very strong descriptor.
Results are usually as good, and often signiﬁcantly better,
than FC-CNN. In most applications, furthermore, FV-CNN
is many times faster as it does not require evaluating the
CNN for each target image region. Finally, FC- and FVCNN can be combined outperforming the state-of-the-art in
many benchmarks.
Domain transfer. So far, the same underlying CNN features, trained on ImageNet’s ILSVCR, were used for all
datasets. Here we investigate the effect of using domainspeciﬁc features. To do so, we consider the PLACES [47],


key: chroma:document
string_value:
datasets. Here we investigate the effect of using domainspeciﬁc features. To do so, we consider the PLACES [47],
trained to recognize places on a dataset of about 2.5 million labeled images. [47] showed that, applied to the task of
scene recognition in MIT Indoor, these features outperform
similar ones trained on ILSVCR (denoted CAFFE [21] below) – a fact explained by the similarity of domains. Below,
we repeat this experiment using FC- and FV-CNN descriptors on top of VGG-M, VGG-VD, PLACES, and CAFFE.
Results are shown in Table 3. The FC-CNN results are
in line with those reported in [47] – in scene recognition
with FC-CNN the same CNN architecture performs better
if trained on the Places dataset instead of the ImageNet data
(58:6%!65:0%accuracy4). Nevertheless, stronger CNN
architectures such as VGG-M and VGG-VD can approach
and outperform PLACES even if trained on ImageNet data
(65:0%!63:0%=68:1%).
However, when it comes to using the ﬁlter banks with
FV-CNN, conclusions are very different. First, FV-CNN


key: chroma:document
string_value:
However, when it comes to using the ﬁlter banks with
FV-CNN, conclusions are very different. First, FV-CNN
outperforms FC-CNN in all cases, with substantial gains
up to 20% in correspondence of a domain transfer from
ImageNet to MIT Indoor. Second, the advantage of using domain-speciﬁc CNNs disappears . In fact, the same
CAFFE model that is 6.4% worse than PLACES with FCCNN, is actually 1.5% better when used in FV-CNN. The
conclusion is that FV-CNN appears to be immune, or at
least substantially less sensitive, to domain shifts.
Our tentative explanation of this surprising phenomenon
is that the convolutional layers are less committed to a speciﬁc dataset than the fully ones. Hence, by using those,
FV-CNN tends to be a more general than FC-CNN.
4.3. Texture segmentation
The previous section considered the problem of region
recognition when the region support is known at test time.
This section studies the problem of recognising regions
when their extent Risnotknown and also be estimated.
The ﬁrst experiment (Tab. 2) investigates the simplest possible scheme: combining the region descriptors
of Sect. 4.1 with a general-purpose image segmentation


key: chroma:document
string_value:
The ﬁrst experiment (Tab. 2) investigates the simplest possible scheme: combining the region descriptors
of Sect. 4.1 with a general-purpose image segmentation
method, namely the crisp regions of [19]. Two datasets
are evaluated: OS for material recognition and MSRC for
things & stuff. Compared to OS+R, classifying crisp regions results in a drop of about 5%points for all descriptors. As this dataset is fairly challenging with best achievable performance is 55:4%, this is a satisfactory result. But
it also illustrates that there is ample space for future improvements. In MSRC, the best accuracy is 87:0%, just a
hair above the best published result 86:5%[25]. Remarkably, these algorithms not use any dataset-speciﬁc training, nor CRF-regularised semantic inference: they simply
greedily classify regions as obtained from a general-purpose
segmentation algorithms. Qualitative segmentation results
(sampled at random) are given in Fig. 2 and 3.
Unlike crisp regions, the proposals of [3] are overlapping and a typical image contains thousands of them. We
propose a simple scheme to combine prediction from multiple proposals. For each proposal we set its label to the
highest scoring class, and score to the highest score. We
then sort the proposals in the increasing order of their score


key: chroma:document
string_value:
highest scoring class, and score to the highest score. We
then sort the proposals in the increasing order of their score
divided by their area and paste them one by one. This has
the effect of considering larger regions before smaller ones
and more conﬁdent regions after less ones for regions of
the same area. Results using FV-CNN shown in Tab. 2 in
4[47] report 68:3%for PLACES applied to MIT Indoor, a small difference explained by implementation details such as the fact that, for all the
methods, we do not perform data augmentation by jittering.
40444852566064687276Performance (%)
conv1
[7 x  7]
 96x256
conv2
[27 x  27]
256x128
conv3
[75 x  75]
512x 64
conv4
[107 x  107]
512x 64
conv5
[139 x  139]
512x 64(VGG−M) Filterbank Analysis
  
VOC07 (mAP11)
MIT Indoor (acc)
FMD (acc)
DTD (acc)
SIFTFigure 4. CNN ﬁlterbank analysis for VGG-M. Performance of
ﬁlter banks extracted from various layers network are shown on


key: chroma:document
string_value:
SIFTFigure 4. CNN ﬁlterbank analysis for VGG-M. Performance of
ﬁlter banks extracted from various layers network are shown on
various datasets. For each layer convf1; : : : ; 5gwe show, the size
of the receptive ﬁeld [ NN], and FBD, where FBis the size
of ﬁlter bank and Dis the dictionary size in the FV representation.
The performance using SIFT is shown in black plus (+) marks.
brackets (FC-CNN was too slow for our experiments). The
results are comparable to those using crisp regions, and we
obtain 55:7%accuracy on the OS dataset. Our initial attempts at schemes such as non-maximum suppression of
overlapping regions that are quite successful for object segmentation [16] performed rather poorly. We believe this is
because unlike objects, material information is fairly localized and highly irregularly shaped in an image. However,
there is room for improvement by combining evidence from
multiple segmentations.
4.4. Convolutional layer analysis
We study the performance of ﬁlter banks extracted from
different layers of a CNN in the FV-CNN framework. We


key: chroma:document
string_value:
4.4. Convolutional layer analysis
We study the performance of ﬁlter banks extracted from
different layers of a CNN in the FV-CNN framework. We
use the VGG-M network which has ﬁve convolutional layers. Results on various datasets, obtained as in Sect. 4.1
and 4.2, are shown in Fig. 4. In addition we also show the
performance using FVs constructed from dense SIFT using
a number of words such that the resulting FV is roughly the
same size of FV-CNN. The CNN ﬁlter banks from layer
3 and beyond signiﬁcantly outperform SIFT. The performance monotonically improves from layer one to ﬁve.
5. Conclusions
We have conducted a range of experiments on material
and texture attribute recognition in a large dataset of textures in clutter. This benchmark was derived from OpenSurfaces, an earlier contribution of the computer graphics community, highlights the potential for collaboration between
computer graphics and vision communities. We have also
evaluated a number of state-of-the-art texture descriptors on
these and many other benchmarks. Our main ﬁnding is that
orderless pooling of convolutional neural network features
is a remarkably good texture descriptor, versatile enough to
dubbed as a scene and object descriptor, resulting in new


key: chroma:document
string_value:
orderless pooling of convolutional neural network features
is a remarkably good texture descriptor, versatile enough to
dubbed as a scene and object descriptor, resulting in new
state-of-the-art performance in several benchmarks.
brick cardboard carpet_rug ceramic concrete fabric_cloth food glass
granite_marble hair other laminate leather metal painted paper_tissue
plastic_clear plastic_opaque stone tile wallpaper wood(a) (b) (c) (d) (e) (f)
Figure 2. OS material recognition results. Example test image with material recognition and segmentation on the OS dataset. (a) original
image. (b) ground truth segmentations from the OpenSurfaces repository (note that not all pixels are annotated). (c) FC-CNN and crispregion proposals segmentation results. (d) incorrectly predicted pixels (restricted to the ones annotated). (e-f) the same, but for FV-CNN.
building grass tree cow sheep sky aeroplane water
face car bicycle flower sign bird book chair
road cat dog body boat
(a) (b) (c) (d) (e) (f)
Figure 3. MSRC object segmentation results. (a) image, (b) ground-truth, (c-d) FC-CNN, (d-e) FV-CNN segmentation and errors.


key: chroma:document
string_value:
Acknowledgements M. Cimpoi was supported by ERC
grant VisRec no. 228180 and the XRCE UAC grant. S. Maji
acknowledges a faculty startup grant from UMass, Amherst.References
[1] E. H. Adelson. On seeing stuff: The perception of materials by humans and machines. SPIE , 4299, 2001. 1
[2] M. Amadasun and R. King. Textural features corresponding to textural properties. Systems, Man, and Cybernetics , 19(5), 1989. 1
[3] P. Arbel ´aez, J. Pont-Tuset, J. T. Barron, F. Marques, and J. Malik.
Multiscale combinatorial grouping. CVPR, 2014. 4, 5, 7
[4] S. Bell, P. Upchurch, N. Snavely, and K. Bala. Opensurfaces: A
richly annotated catalog of surface appearance. In Proc. SIGGRAPH ,
2013. 2
[5] B. Caputo, E. Hayman, and P. Mallikarjuna. Class-speciﬁc material
categorisation. In ICCV , 2005. 1, 4


key: chroma:document
string_value:
categorisation. In ICCV , 2005. 1, 4
[6] K. Chatﬁeld, K. Simonyan, A. Vedaldi, and A. Zisserman. Return
of the devil in the details: Delving deep into convolutional nets. In
Proc. BMVC , 2014. 2, 3, 4, 6
[7] M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, , and A. Vedaldi.
Describing textures in the wild. In Proc. CVPR , 2014. 1, 3, 4, 5
[8] K. J. Dana, B. van Ginneken, S. K. Nayar, and J. J. Koenderink.
Reﬂectance and texture of real world surfaces. ACM Transactions on
Graphics , 18(1):1–34, 1999. 1
[9] A. Efros and T. Leung. Texture synthesis by non-parametric sampling. In CVPR , volume 2, 1999. 1
[10] M. Everingham, A. Zisserman, C. Williams, and L. V . Gool. The
PASCAL visual obiect classes challenge 2007 (VOC2007) results.
Technical report, Pascal Challenge, 2007. 3, 4


key: chroma:document
string_value:
PASCAL visual obiect classes challenge 2007 (VOC2007) results.
Technical report, Pascal Challenge, 2007. 3, 4
[11] V . Ferrari and A. Zisserman. Learning visual attributes. In Proc.
NIPS , 2007. 1
[12] D. Forsyth. Shape from texture and integrability. In ICCV , volume 2,
pages 447–452. IEEE, 2001. 1
[13] J. G ˚arding. Shape from texture for smooth curved surfaces in perspective projection. Journal of Mathematical Imaging and Vision ,
2(4):327–350, 1992. 1
[14] R. B. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature
hierarchies for accurate object detection and semantic segmentation.
InProc. CVPR , 2014. 2, 3
[15] Y . Gong, L. Wang, R. Guo, and S. Lazebnik. Multi-scale orderless
pooling of deep convolutional activation features. In Proc. ECCV ,
2014. 2, 3, 6
[16] B. Hariharan, P. Arbel ´aez, R. Girshick, and J. Malik. Simultaneous
detection and segmentation. In Computer Vision–ECCV 2014 , pages


key: chroma:document
string_value:
detection and segmentation. In Computer Vision–ECCV 2014 , pages
297–312. Springer, 2014. 2, 7
[17] E. Hayman, B. Caputo, M. Fritz, and J.-O. Eklundh. On the significance of real-world conditions for material classiﬁcation. ECCV ,
2004. 4
[18] K. He, X. Zhang, S. Ren, and J. Sun. Spatial pyramid pooling in deep
convolutional networks for visual recognition. In Proc. ECCV , 2014.
2
[19] P. Isola, D. Zoran, D. Krishnan, and E. H. Adelson. Crisp boundary
detection using pointwise mutual information. In Proc. ECCV , 2014.
4, 7
[20] A. Jain and F. Farrokhnia. Unsupervised texture segmentation using
gabor ﬁlters. Pattern recognition , 24(12):1167–1186, 1991. 1
[21] Y . Jia. Caffe: An open source convolutional architecture for fast
feature embedding. http://caffe.berkeleyvision.org/ ,
2013. 3, 4, 6
[22] B. Julesz and J. R. Bergen. Textons, the fundamental elements in


key: chroma:document
string_value:
2013. 3, 4, 6
[22] B. Julesz and J. R. Bergen. Textons, the fundamental elements in
preattentive vision and perception of textures. Bell System Technical
Journal , 62(6, Pt 3):1619–1645, Jul-Aug 1983. 1
[23] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classiﬁcation with deep convolutional neural networks. In Proc. NIPS , 2012.
2, 3
[24] L. Ladicky, C. Russell, P. Kohli, and P. Torr. Graph cut based inference with co-occurrence statistics. In Proc. ECCV , pages 239–253,
2010. 5
[25] L. Ladicky, P. Sturgess, K. Alahari, C. Russell, and P. Torr. What,
where and how many? combining object detectors and crfs. In Proc.
ECCV , 2010. 7
[26] T. Leung and J. Malik. Representing and recognizing the visual appearance of materials using three-dimensional textons. International
Journal of Computer Vision , 43(1):29–44, 2001. 1


key: chroma:document
string_value:
Journal of Computer Vision , 43(1):29–44, 2001. 1
[27] L. Liu, L. Wang, and X. Liu. In defense of soft-assignment coding.
InComputer Vision (ICCV), 2011 IEEE International Conference on ,
pages 2486–2493. IEEE, 2011. 1[28] D. G. Lowe. Object recognition from local scale-invariant features.
InProc. ICCV , 1999. 1
[29] B. Manjunath and R. Chellappa. Unsupervised texture segmentation
using markov random ﬁeld models. IEEE Transactions on Pattern
Analysis and Machine Intelligence , 13(5):478–482, 1991. 1
[30] M. Oquab, L. Bottou, I. Laptev, and J. Sivic. Learning and Transferring Mid-Level Image Representations using Convolutional Neural
Networks. In Proc. CVPR , 2014. 2
[31] O. M. Parkhi, K. Simonyan, A. Vedaldi, and A. Zisserman. A compact and discriminative face track descriptor. In Proc. CVPR , 2014.
4


key: chroma:document
string_value:
4
[32] F. Perronnin, J. S ´anchez, and T. Mensink. Improving the ﬁsher kernel for large-scale image classiﬁcation. In Computer Vision–ECCV
2010 , pages 143–156. Springer, 2010. 1
[33] X. Qi, R. Xiao, C. G. Li, Y . Qiao, J. Guo, and X. Tang. Pairwise rotation invariant co-occurrence local binary pattern. PAMI ,
36(11):2199–2213, Nov 2014. 5
[34] A. Quattoni and A. Torralba. Recognizing indoor scenes. In Proc.
CVPR , 2009. 2, 4
[35] A. S. Razavin, H. Azizpour, J. Sullivan, and S. Carlsson. Cnn features off-the-shelf: An astounding baseline for recognition. In DeepVision workshop , 2014. 3
[36] G. Schwartz and K. Nishino. Visual material traits: Recognizing
per-pixel material context. In Proc. CVCP , 2013. 1
[37] L. Sharan, C. Liu, R. Rosenholtz, and E. H. Adelson. Recognizing
materials using perceptually inspired features. International Journal


key: chroma:document
string_value:
[37] L. Sharan, C. Liu, R. Rosenholtz, and E. H. Adelson. Recognizing
materials using perceptually inspired features. International Journal
of Computer Vision , 103(3):348–371, 2013. 1, 4, 5
[38] L. Sharan, R. Rosenholtz, and E. H. Adelson. Material perceprion:
What can you see in a brief glance? Journal of Vision , 9:784(8),
2009. 1
[39] K. Simonyan and A. Zisserman. Very deep convolutional networks
for large-scale image recognition. CoRR , abs/1409.1556, 2014. 2, 4
[40] M. Sulc and J. Matas. Fast features invariant to rotation and scale of
texture. Technical report, 2014. 5
[41] M. Varma and A. Zisserman. Texture classiﬁcation: Are ﬁlter banks
necessary? In CVPR , volume 2, pages II–691. IEEE, 2003. 1
[42] J. Wang, J. Yang, K. Yu, F. Lv, T. Huang, and Y . Gong. Localityconstrained linear coding for image classiﬁcation. In Computer Vision and Pattern Recognition (CVPR), 2010 IEEE Conference on ,


key: chroma:document
string_value:
pages 3360–3367. IEEE, 2010. 1
[43] L. Wei and M. Levoy. Fast texture synthesis using treestructured vector quantization. In SIGGRAPH , pages 479–488. ACM
Press/Addison-Wesley Publishing Co., 2000. 1
[44] Y . Wei, W. Xia, J. Huang, B. Ni, J. Dong, Y . Zhao, and S. Yan. Cnn:
Single-label to multi-label. 2014. 5, 6
[45] P. Welinder, S. Branson, T. Mita, C. Wah, and F. Schroff. Caltechucsd birds 200. Technical report, Caltech-UCSD, 2010. 4
[46] N. Zhang, J. Donahue, R. Girshickr, and T. Darrell. Part-based RCNNs for ﬁne-grained category detection. In Proc. ECCV , 2014. 5,
6
[47] B. Zhou, A. Lapedriza, J. Xiao, A. Torralba, and A. Oliva. Learning
deep features for scene recognition using places database. In Proc.
NIPS , 2014. 2, 5, 6, 7


key: chroma:document
string_value:
deep features for scene recognition using places database. In Proc.
NIPS , 2014. 2, 5, 6, 7
[48] X. Zhou, K. Yu, T. Zhang, and T. S. Huang. Image classiﬁcation
using super-vector coding of local image descriptors. In Computer
Vision–ECCV 2010 , pages 141–154. Springer, 2010. 1


key: chroma:document
string_value:
Material Recognition in the Wild with the Materials in Context Database
Sean BellPaul UpchurchNoah Snavely Kavita Bala
Department of Computer Science, Cornell University
fsbell,paulu,snavely,kb g@cs.cornell.edu
Abstract
Recognizing materials in real-world images is a challenging task. Real-world materials have rich surface texture,
geometry, lighting conditions, and clutter, which combine
to make the problem particularly difﬁcult. In this paper, we
introduce a new, large-scale, open dataset of materials in
the wild, the Materials in Context Database (MINC ), and
combine this dataset with deep learning to achieve material
recognition and segmentation of images in the wild.
MINC is an order of magnitude larger than previous material databases, while being more diverse and well-sampled
across its 23 categories. Using MINC , we train convolutional neural networks (CNNs) for two tasks: classifying
materials from patches, and simultaneous material recognition and segmentation in full images. For patch-based classiﬁcation on MINC we found that the best performing CNN
architectures can achieve 85.2% mean class accuracy. We
convert these trained CNN classiﬁers into an efﬁcient fully


key: chroma:document
string_value:
architectures can achieve 85.2% mean class accuracy. We
convert these trained CNN classiﬁers into an efﬁcient fully
convolutional framework combined with a fully connected
conditional random ﬁeld (CRF) to predict the material at
every pixel in an image, achieving 73.1% mean class accuracy. Our experiments demonstrate that having a large,
well-sampled dataset such as MINC is crucial for real-world
material recognition and segmentation.
1. Introduction
Material recognition plays a critical role in our understanding of and interactions with the world. To tell whether
a surface is easy to walk on, or what kind of grip to use
to pick up an object, we must recognize the materials that
make up our surroundings. Automatic material recognition
can be useful in a variety of applications, including robotics,
product search, and image editing for interior design. But recognizing materials in real-world images is very challenging.
Many categories of materials, such as fabric or wood, are
visually very rich and span a diverse range of appearances.
Materials can further vary in appearance due to lighting and
shape. Some categories, such as plastic and ceramic, are ofAuthors contributed equally
CNN
Sliding
CNN“wood”
Transfer weights(trained)
(discretely


key: chroma:document
string_value:
CNN
Sliding
CNN“wood”
Transfer weights(trained)
(discretely 
optimized)(fixed)AMT AMT AMT MINC
3 million
patches OpenSurfacesFlickr
Houzz(a) Constructing MINC
Material
Labels(b) Patch material classification
(c) Full scene material classificationP(material)
Dense
CRF
PatchFigure 1. Overview. (a) We construct a new dataset by combining
OpenSurfaces [ 1] with a novel three-stage Amazon Mechanical
Turk (AMT) pipeline. (b)We train various CNNs on patches from
MINC to predict material labels. (c)We transfer the weights to a
fully convolutional CNN to efﬁciently generate a probability map
across the image; we then use a fully connected CRF to predict the
material at every pixel.
ten smooth and featureless, requiring reasoning about subtle
cues or context to differentiate between them.
Large-scale datasets (e.g., ImageNet [ 21], SUN [ 31,19]
and Places [ 34]) combined with convolutional neural networks (CNNs) have been key to recent breakthroughs in
object recognition and scene classiﬁcation. Material recognition is similarly poised for advancement through large-scale
data and learning. To date, progress in material recognition


key: chroma:document
string_value:
object recognition and scene classiﬁcation. Material recognition is similarly poised for advancement through large-scale
data and learning. To date, progress in material recognition
has been facilitated by moderate-sized datasets like the Flickr
Material Database (FMD) [ 26]. FMD contains ten material
categories, each with 100 samples drawn from Flickr photos.
These images were carefully selected to illustrate a wide
range of appearances for these categories. FMD has been
used in research on new features and learning methods for
material perception and recognition [ 17,10,20,25]. While
FMD was an important step towards material recognition, it
is not sufﬁcient for classifying materials in real-world im1arXiv:1412.0623v2  [cs.CV]  14 Apr 2015
agery. This is due to the relatively small set of categories,
the relatively small number of images per category, and also
because the dataset has been designed around hand-picked
iconic images of materials. The OpenSurfaces dataset [ 1]
addresses some of these problems by introducing 105,000
material segmentations from real-world images, and is signiﬁcantly larger than FMD. However, in OpenSurfaces many
material categories are under-sampled, with only tens of
images.


key: chroma:document
string_value:
material categories are under-sampled, with only tens of
images.
A major contribution of our paper is a new, well-sampled
material dataset, called the Materials in Context Database
(MINC ), with 3 million material samples. MINC is more
diverse, has more examples in less common categories, and
is much larger than existing datasets. MINC draws data from
both Flickr images, which include many “regular” scenes,
as well as Houzz images from professional photographers of
staged interiors. These sources of images each have different
characteristics that together increase the range of materials
that can be recognized. See Figure 2 for examples of our
data. We make our full dataset available online at http:
//minc.cs.cornell.edu/.
We use this data for material recognition by training different CNN architectures on this new dataset. We perform
experiments that illustrate the effect of network architecture, image context, and training data size on subregions
(i.e., patches) of a full scene image. Further, we build on
our patch classiﬁcation results and demonstrate simultaneous material recognition and segmentation of an image by
performing dense classiﬁcation over the image with a fully
connected conditional random ﬁeld (CRF) model [ 12]. By


key: chroma:document
string_value:
performing dense classiﬁcation over the image with a fully
connected conditional random ﬁeld (CRF) model [ 12]. By
replacing the fully connected layers of the CNN with convolutional layers [ 24], the computational burden is signiﬁcantly
lower than a naive sliding window approach.
In summary, we make two new contributions:
We introduce a new material dataset, MINC , and 3stage crowdsourcing pipeline for efﬁciently collecting
millions of click labels (Section 3.2).
Our new semantic segmentation method combines a
fully-connected CRF with unary predictions based on
CNN learned features (Section 4.2) for simultaneous
material recognition and segmentation.
2. Prior Work
Material Databases. Much of the early work on material
recognition focused on classifying speciﬁc instances of textures or material samples. For instance, the CUReT [ 4]
database contains 61 material samples, each captured under
205 different lighting and viewing conditions. This led to
research on the task of instance-level texture or material classiﬁcation [ 15,30], and an appreciation of the challenges of
building features that are invariant to pose and illumination.


key: chroma:document
string_value:
building features that are invariant to pose and illumination.
Later, databases with more diverse examples from each ma-terial category began to appear, such as KTH-TIPS [ 9,2],
and led explorations of how to generalize from one example
of a material to another—from one sample of wood to a completely different sample, for instance. Real-world texture
attributes have also recently been explored [3].
In the domain of categorical material databases, Sharan et
al. released FMD [ 26] (described above). Subsequently,
Bell et al. released OpenSurfaces [ 1] which contains over
20,000 real-world scenes labeled with both materials and objects, using a multi-stage crowdsourcing pipeline. Because
OpenSurfaces images are drawn from consumer photos on
Flickr, material samples have real-world context, in contrast
to prior databases (CUReT, KTH-TIPS, FMD) which feature cropped stand-alone samples. While OpenSurfaces is a
good starting point for a material database, we substantially
expand it with millions of new labels.
Material recognition. Much prior work on material recognition has focused on the classiﬁcation problem (categorizing
an image patch into a set of material categories), often using


key: chroma:document
string_value:
Material recognition. Much prior work on material recognition has focused on the classiﬁcation problem (categorizing
an image patch into a set of material categories), often using
hand-designed image features. For FMD, Liu et al. [17] introduced reﬂectance-based edge features in conjunction with
other general image features. Hu et al. [10] proposed features based on variances of oriented gradients. Qi et al. [20]
introduced a pairwise local binary pattern (LBP) feature.
Liet al. [16] synthesized a dataset based on KTH-TIPS2
and built a classiﬁer from LBP and dense SIFT. Timofte et
al. [29] proposed a classiﬁcation framework with minimal
parameter optimization. Schwartz and Nishino [ 23] introduced material traits that incorporate learned convolutional
auto-encoder features. Recently, Cimpoi et al. [3] developed a CNN and improved Fisher vector (IFV) classiﬁer that
achieves state-of-the-art results on FMD and KTH-TIPS2.
Finally, it has been shown that jointly predicting objects and
materials can improve performance [10, 33].
Convolutional neural networks. While CNNs have been


key: chroma:document
string_value:
Finally, it has been shown that jointly predicting objects and
materials can improve performance [10, 33].
Convolutional neural networks. While CNNs have been
around for a few decades, with early successes such as
LeNet [ 14], they have only recently led to state-of-theart results in object classiﬁcation and detection, leading
to enormous progress. Driven by the ILSVRC challenge [ 21], we have seen many successful CNN architectures [ 32,24,28,27], led by the work of Krizhevsky et al.
on their SuperVision (a.k.a. AlexNet ) network [ 13], with
more recent architectures including GoogLeNet [ 28]. In addition to image classiﬁcation, CNNs are the state-of-the-art
for detection and localization of objects, with recent work
including R-CNNs [ 7], Overfeat [ 24], and VGG [ 27]. Finally, relevant to our goal of per-pixel material segmentation,
Farabet et al. [6] use a multi-scale CNN to predict the class
at every pixel in a segmentation. Oquab et al. [18] employ a
sliding window approach to localize patch classiﬁcation of
objects. We build on this body of work in deep learning to


key: chroma:document
string_value:
sliding window approach to localize patch classiﬁcation of
objects. We build on this body of work in deep learning to
solve our problem of material recognition and segmentation.
2
Brick Carpet Ceramic Fabric Foliage Food Glass Hair
Leather Metal Mirror Other Painted Paper Plastic Pol. stone
Skin Sky Stone Tile Wallpaper Water Wood
Figure 2. Example patches from all 23 categories of the Materials in Context Database (MINC ). Note that we sample patches so that the
patch center is the material in question (and not necessarily the entire patch). See Table 1 for the size of each category.
3. The Materials in Context Database (MINC)
We now describe the methodology that went into building
our new material database. Why a new database? We needed
a dataset with the following properties:
Size: It should be sufﬁciently large that learning methods can generalize beyond the training set.
Well-sampled : Rare categories should be represented
with a large number of examples.
Diversity : Images should span a wide range of appearances of each material in real-world settings.
Number of categories : It should contain many different materials found in the real world.
3.1. Sources of data
We decided to start with the public, crowdsourced
OpenSurfaces dataset [ 1] as the seed for MINC since it is


key: chroma:document
string_value:
3.1. Sources of data
We decided to start with the public, crowdsourced
OpenSurfaces dataset [ 1] as the seed for MINC since it is
drawn from Flickr imagery of everyday, real-world scenes
with reasonable diversity. Furthermore, it has a large number
of categories and the most samples of all prior databases.
While OpenSurfaces data is a good start, it has a few limitations. Many categories in OpenSurfaces are not well sampled. While the largest category, wood , has nearly 20K samples, smaller categories, such as water , have only tens of examples. This imbalance is due to the way the OpenSurfaces
dataset was annotated; workers on Amazon Mechanical Turk
(AMT) were free to choose any material subregion to segment. Workers often gravitated towards certain common
types of materials or salient objects, rather than being encouraged to label a diverse set of materials. Further, the
images come from a single source (Flickr).
We decided to augment OpenSurfaces with substantially
more data, especially for underrepresented material cate-gories, with the initial goal of gathering at least 10K samples
per material category. We decided to gather this data from
another source of imagery, professional photos on the interior design website Houzz (houzz.com). Our motivation for
using this different source of data was that, despite Houzz


key: chroma:document
string_value:
another source of imagery, professional photos on the interior design website Houzz (houzz.com). Our motivation for
using this different source of data was that, despite Houzz
photos being more “staged” (relative to Flickr photos), they
actually represent a larger variety of materials. For instance,
Houzz photos contain a wide range of types of polished
stone. With these sources of image data, we now describe
how we gather material annotations.
3.2. Segments, Clicks, and Patches
What speciﬁc kinds of material annotations make for a
good database? How should we collect these annotations?
The type of annotations to collect is guided in large part by
the tasks we wish to generate training data for. For some
tasks such as scene recognition, whole-image labels can
sufﬁce [ 31,34]. For object detection, labeled bounding
boxes as in PASCAL are often used [ 5]. For segmentation or
scene parsing tasks, per-pixel segmentations are required [ 22,
8]. Each style of annotation comes with a cost proportional
to its complexity. For materials, we decided to focus on two
problems, guided by prior work:
Patch material classiﬁcation. Given an image patch,
what kind of material is it at the center?


key: chroma:document
string_value:
problems, guided by prior work:
Patch material classiﬁcation. Given an image patch,
what kind of material is it at the center?
Full scene material classiﬁcation. Given a full image, produce a full per-pixel segmentation and labeling.
Also known as semantic segmentation orscene parsing
(but in our case, focused on materials). Note that classiﬁcation can be a component of segmentation, e.g., with
sliding window approaches.
3
(a) Which images 
contain wood ?(b) Click on 3 
points of wood(c) What is this 
material?Figure 3. AMT pipeline schematic for collecting clicks. (a)
Workers ﬁlter by images that contain a certain material, (b)workers click on materials, and (c)workers validate click locations by
re-labeling each point. Example responses are shown in orange.
Segments. OpenSurfaces contains material segmentations—
carefully drawn polygons that enclose same-material regions.
To form the basis of MINC, we selected OpenSurfaces segments with high conﬁdence (inter-worker agreement) and
manually curated segments with low conﬁdence, giving a
total of 72K shapes. To better balance the categories, we


key: chroma:document
string_value:
manually curated segments with low conﬁdence, giving a
total of 72K shapes. To better balance the categories, we
manually segmented a few hundred extra samples for sky,
foliage andwater .
Since some of the OpenSurfaces categories are difﬁcult
for humans, we consolidated these categories. We found
that many AMT workers could not disambiguate stone from
concrete ,clear plastic from opaque plastic , and granite
from marble . Therefore, we merged these into stone ,plastic ,
andpolished stone respectively. Without this merging, many
ground truth examples in these categories would be incorrect.
The ﬁnal list of 23 categories is shown in Table 1. The
category other is different in that it was created by combining
various smaller categories.
Clicks. Since we want to expand our dataset to millions
of samples, we decided to augment OpenSurfaces segments
by collecting clicks : single points in an image along with a
material label, which are much cheaper and faster to collect.
Figure 3 shows our pipeline for collecting clicks.
Initially, we tried asking workers to click on examples
of a given material in a photo. However, we found that
workers would get frustrated if the material was absent in
too many of the photos. Thus, we added an initial ﬁrst


key: chroma:document
string_value:
workers would get frustrated if the material was absent in
too many of the photos. Thus, we added an initial ﬁrst
stage where workers ﬁlter out such photos. To increase
the accuracy of our labels, we verify the click labels by
asking different workers to specify the material for each
click without providing them with the label from the previous
stage.
To ensure that we obtain high quality annotations and
avoid collecting labels from workers who are not making an
effort, we include secret known answers (sentinels) in the
ﬁrst and third stages, and block workers with an accuracy
below 50% and 85% respectively. We do not use sentinels
in the second stage since it would require per-pixel ground
truth labels, and it turned out not to be necessary. Workers
generally performed all three tasks so we could identify bad
workers in the ﬁrst or third task.Patches Category Patches Category Patches Category
564,891 Wood 114,085 Polished stone 35,246 Skin
465,076 Painted 98,891 Carpet 29,616 Stone
397,982 Fabric 83,644 Leather 28,108 Ceramic
216,368 Glass 75,084 Mirror 26,103 Hair
188,491 Metal 64,454 Brick 25,498 Food
147,346 Tile 55,364 Water 23,779 Paper


key: chroma:document
string_value:
216,368 Glass 75,084 Mirror 26,103 Hair
188,491 Metal 64,454 Brick 25,498 Food
147,346 Tile 55,364 Water 23,779 Paper
142,150 Sky 39,612 Other 14,954 Wallpaper
120,957 Foliage 38,975 Plastic
Table 1. MINC patch counts by category. Patches were created
from both OpenSurfaces segments and our newly collected clicks .
See Section 3.2 for details.
Material clicks were collected for both OpenSurfaces
images and the new Houzz images. This allowed us to use
labels from OpenSurfaces to generate the sentinel data; we
included 4 sentinels per task. With this streamlined pipeline
we collected 2,341,473 annotations at an average cost of
$0.00306 per annotation (stage 1: $0.02 / 40 images, stage
2: $0.10 / 50 images, 2, stage 3: $0.10 / 50 points).
Patches. Labeled segments and clicks form the core of
MINC . For training CNNs and other types of classiﬁers,
it is useful to have data in the form of ﬁxed-sized patches.
We convert both forms of data into a uniﬁed dataset format:
square image patches. We use a patch center andpatch scale


key: chroma:document
string_value:
We convert both forms of data into a uniﬁed dataset format:
square image patches. We use a patch center andpatch scale
(a multiplier of the smaller image dimension) to deﬁne the
image subregion that makes a patch. For our patch classiﬁcation experiments, we use 23.3% of the smaller image
dimension. Increasing the patch scale provides more context
but reduces the spatial resolution. Later in Section 5 we
justify our choice with experiments that vary the patch scale
for AlexNet.
We place a patch centered around each click label. For
each segment , if we were to place a patch at every interior
pixel then we would have a very large and redundant dataset.
Therefore, we Poisson-disk subsample each segment , separating patch centers by at least 9.1% of the smaller image
dimension. These segments generated 655,201 patches (an
average of 9.05 patches per segment ). In total, we generated 2,996,674 labeled patches from 436,749 images. Patch
counts are shown in Table 1, and example patches from
various categories are illustrated in Figure 2.
4. Material recognition in real-world images
Our goal is to train a system that recognizes the material
at every pixel in an image. We split our training procedure
into multiple stages and analyze the performance of the


key: chroma:document
string_value:
Our goal is to train a system that recognizes the material
at every pixel in an image. We split our training procedure
into multiple stages and analyze the performance of the
network at each stage. First, we train a CNN that produces a
single prediction for a given input patch. Then, we convert
the CNN into a sliding window and predict materials on a
dense grid across the image. We do this at multiple scales
and average to obtain a unary term. Finally, a dense CRF
[12] combines the unary term with fully connected pairwise
reasoning to output per-pixel material predictions. The entire
system is depicted in Figure 1, and described more below.
4
Sky
Water
FoliageTile
(a) Multiscale input (b) Probability map (12 of 23 shown) (c) Predicted labelsSliding
CNNDense
CRF
Wood
Plastic Sliding
CNNSliding
CNN
Avg.
Glass
Hair
Fabric
Polished
stone
Stone
 PaperWaterSky
Foliage
TilePaperWood
StoneWood
TileFigure 4. Pipeline for full scene material classiﬁcation. An image (a)is resized to multiple scales [1=p
2;1;p
2]. The same sliding CNN


key: chroma:document
string_value:
2;1;p
2]. The same sliding CNN
predicts a probability map (b)across the image for each scale; the results are upsampled and averaged. A fully connected CRF predicts a
ﬁnal label for each pixel (c). This example shows predictions from a single GoogLeNet converted into a sliding CNN (no average pooling).
4.1. Training procedure
MINC contains 3 million patches that we split into training, validation and test sets. Randomly splitting would result
in nearly identical patches (e.g., from the same OpenSurfaces segment) being put in training and test, thus inﬂating
the test score. To prevent correlation, we group photos into
clusters of near-duplicates, then assign each cluster to one
of train, validate or test. We make sure that there are at least
75segments of each category in the test set to ensure there
are enough segments to evaluate segmentation accuracy. To
detect near-duplicates, we compare AlexNet CNN features
computed from each photo (see the supplemental for details).
For exact duplicates, we discard all but one of the copies.
We train all of our CNNs by ﬁne-tuning the network
starting from the weights obtained by training on 1.2 million images from ImageNet (ILSVRC2012). When training


key: chroma:document
string_value:
starting from the weights obtained by training on 1.2 million images from ImageNet (ILSVRC2012). When training
AlexNet , we use stochastic gradient descent with batchsize
128, dropout rate 0.5, momentum 0.9, and a base learning
rate of 103that decreases by a factor of 0.25 every 50,000
iterations. For GoogLeNet , we use batchsize 69, dropout 0.4,
and learning rate t= 104p
1t=250000 for iteration t.
Our training set has a different number of examples per
class, so we cycle through the classes and randomly sample
an example from each class. Failing to properly balance the
examples results in a 5.7% drop in mean class accuracy (on
the validation set). Further, since it has been shown to reduce
overﬁtting, we randomly augment samples by taking crops
(227227out of 256256), horizontal mirror ﬂips, spatial
scales in the range [1=p
2;p
2], aspect ratios from 3:4 to 4:3,
and amplitude shifts in [0:95;1:05]. Since we are looking at
local regions, we subtract a per-channel mean (R: 124, G:


key: chroma:document
string_value:
and amplitude shifts in [0:95;1:05]. Since we are looking at
local regions, we subtract a per-channel mean (R: 124, G:
117, B: 104) rather than a mean image [13].
4.2. Full scene material classiﬁcation
Figure 4 shows an overview of our method for simultaneously segmenting and recognizing materials. Given a
CNN that can classify individual points in the image, we
convert it to a sliding window detector and densely classify
a grid across the image. Speciﬁcally, we replace the last
fully connected layers with convolutional layers, so that the
network is fully convolutional and can classify images ofany shape. After conversion, the weights are ﬁxed and not
ﬁne-tuned. With our converted network, the strides of each
layer cause the network to output a prediction every 32 pixels. We obtain predictions every 16 pixels by shifting the
input image by half-strides (16 pixels). While this appears to
require 4x the computation, Sermanet et al. [24] showed that
the convolutions can be reused and only the pool5 through
fc8 layers need to be recomputed for the half-stride shifts.
Adding half-strides resulted in a minor 0.2% improvement
in mean class accuracy across segments (after applying the


key: chroma:document
string_value:
Adding half-strides resulted in a minor 0.2% improvement
in mean class accuracy across segments (after applying the
dense CRF, described below), and about the same mean class
accuracy at click locations.
The input image is resized so that a patch maps to a
256x256 square. Thus, for a network trained at patch scale
s, the resized input has smaller dimension d= 256=s. Note
thatdis inversely proportional to scale, so increased context
leads to lower spatial resolution. We then add padding so
that the output probability map is aligned with the input
when upsampled. We repeat this at 3 different scales (smaller
dimensiond=p
2,d,dp
2), upsample each output probability
map with bilinear interpolation, and average the predictions.
To make the next step more efﬁcient, we upsample the output
to a ﬁxed smaller dimension of 550.
We then use the dense CRF of Kr¨ahenb ¨uhlet al. [12] to
predict a label at every pixel, using the following energy:
E(xjI) =X
i i(xi) +X
i<j ij(xi;xj) (1)
 i(xi) =logpi(xi) (2)


key: chroma:document
string_value:
i i(xi) +X
i<j ij(xi;xj) (1)
 i(xi) =logpi(xi) (2)
 ij(xi;xj) =wp(xi6=xj)k(fifj) (3)
where iis the unary energy (negative log of the aggregated softmax probabilities) and  ijis the pairwise term
that connects every pair of pixels in the image. We use a
single pairwise term with a Potts label compatibility term
weighted by wpand unit Gaussian kernel k. For the featuresfi, we convert the RGB image to L*a*b*and use color
(IL
i;Ia
i;Ib
i)and position (px;py)as pairwise features for
each pixel: fi=h
px
i
pd;py
i
pd;IL
i
L;Ia
i
ab;Ib
i
abi
, wheredis the
smaller image dimension. Figure 4 shows an example unary
termpiand the resulting segmentation x.
5
Figure 5. Varying patch scale . We train/test patches of different
scales (the patch locations do not vary). The optimum is a trade-off


key: chroma:document
string_value:
5
Figure 5. Varying patch scale . We train/test patches of different
scales (the patch locations do not vary). The optimum is a trade-off
between context and spatial resolution. CNN: AlexNet.
Architecture Validation Test
AlexNet [13] 82.2% 81.4%
GoogLeNet [28] 85.9% 85.2%
VGG-16 [27] 85.6% 84.8%
Table 2. Patch material classiﬁcation results. Mean class accuracy for different CNNs trained on MINC. See Section 5.1.
Sky 97.3% Food 90.4% Wallpaper 83.4% Glass 78.5%
Hair 95.3% Leather 88.2% Tile 82.7% Fabric 77.8%
Foliage 95.1% Other 87.9% Ceramic 82.7% Metal 77.7%
Skin 93.9% Pol. stone 85.8% Stone 82.7% Mirror 72.0%
Water 93.6% Brick 85.1% Paper 81.8% Plastic 70.9%
Carpet 91.6% Painted 84.2% Wood 81.3%
Table 3. Patch test accuracy by category. CNN: GoogLeNet . See
the supplemental material for a full confusion matrix.
5. Experiments and Results


key: chroma:document
string_value:
Table 3. Patch test accuracy by category. CNN: GoogLeNet . See
the supplemental material for a full confusion matrix.
5. Experiments and Results
5.1. Patch material classiﬁcation
In this section, we evaluate the effect of many different
design decisions for training methods for material classiﬁcation and segmentation, including various CNN architectures,
patch sizes, and amounts of data.
CNN Architectures. Our ultimate goal is full material segmentation, but we are also interested in exploring which
CNN architectures give the best results for classifying single patches. Among the networks and parameter variations we tried we found the best performing networks were
AlexNet [13],VGG-16 [27] and GoogLeNet [28].AlexNet
andGoogLeNet are re-implementations by BVLC [ 11], and
VGG-16 is conﬁguration D (a 16 layer network) of [ 27].
All models were obtained from the Caffe Model Zoo [ 11].
Our experiments use AlexNet for evaluating material classiﬁcation design decisions and combinations of AlexNet and
GoogLeNet for evaluating material segmentation. Tables 2
and 3 summarize patch material classiﬁcation results on our
dataset. Figure 10 shows correct and incorrect predictions
made with high conﬁdence.


key: chroma:document
string_value:
and 3 summarize patch material classiﬁcation results on our
dataset. Figure 10 shows correct and incorrect predictions
made with high conﬁdence.
Input patch scale. To classify a point in an image we must
decide how much context to include around it. The context,
expressed as a fraction of image size, is the patch scale. A
priori, it is not clear which scale is best since small patches
have better spatial resolution, but large patches have more
0.5e6 1e6 1.5e6 2e6 2.5e6
Number of training patches0.600.650.700.750.800.85Mean class accuracy (test set)66.1%74.8%77.9%79.3%80.2%80.9% 81.2% 81.4%Equal size
79.6%Figure 6. Varying database size. Patch accuracy when trained on
random subsets of MINC .Equal size is using equal samples per
category (size determined by smallest category). CNN: AlexNet.
Peak accuracy
per category
Figure 7. Accuracy vs patch scale by category . Dots: peak accuracy for each category; colored lines: sky,wallpaper ,mirror ; gray
lines: other categories. CNN: AlexNet . While most materials are


key: chroma:document
string_value:
lines: other categories. CNN: AlexNet . While most materials are
optimally recognized at 23.3% or 32% patch scale, recognition of
sky,wallpaper andmirror improve with increasing context.
contextual information. Holding patch centers ﬁxed we varied scale and evaluated classiﬁcation accuracy with AlexNet .
Results and a visualization of patch scales are shown in Figure 5. Scale 32% performs the best. Individual categories
had peaks at middle scale with some exceptions; we ﬁnd
thatmirror ,wallpaper andskyimprove with increasing context (Figure 7). We used 23.3% (which has nearly the same
accuracy but higher spatial resolution) for our experiments.
Dataset size. To measure the effect of size on patch classiﬁcation accuracy we trained AlexNet with patches from
randomly sampled subsets of all 369,104 training images
and tested on our full test set (Figure 6). As expected, using
more data improved performance. In addition, we still have
not saturated performance with 2.5 million training patches;
even higher accuracies may be possible with more training
data (though with diminishing returns).
Dataset balance. Although we’ve shown that more data is
better we also ﬁnd that a balanced dataset is more effective.
We trained AlexNet with all patches of our smallest category


key: chroma:document
string_value:
better we also ﬁnd that a balanced dataset is more effective.
We trained AlexNet with all patches of our smallest category
(wallpaper ) and randomly sampled the larger categories
(the largest, wood , being 40x larger) to be equal size. We
then measured mean class accuracy on the same full test
set. As shown in Figure 6, “Equal size” is more accurate
than a dataset of the same size and just 1.7% lower than
the full training set (which is 9x larger). This result further
demonstrates the value of building up datasets in a balanced
manner, focusing on expanding the smallest, least common
categories.
6
Brick
WoodCarpet
WaterWallpaperTileStoneSkySkinPolished stonePlasticPaperPainted
OtherMirrorMetalLeatherHairGlassFoodFoliageFabricCeramicFigure 8. Full-scene material classiﬁcation examples: high-accuracy test set predictions by our method. CNN: GoogLeNet (with the
average pooling layer removed). Right: legend for material colors. See Table 4 for quantitative evaluation.
Input image (a)Labels from CRF (b)Labels from CRF
(test set) trained on segments trained on clicks


key: chroma:document
string_value:
Input image (a)Labels from CRF (b)Labels from CRF
(test set) trained on segments trained on clicks
Figure 9. Optimizing for click accuracy leads to sloppy boundaries. In(a), we optimize for mean class accuracy across segments,
resulting in high quality boundaries. In (b), we optimize for mean
class accuracy at click locations. Since the clicks are not necessarily close to object boundaries, there is no penalty for sloppy
boundaries. CNN: GoogLeNet (without average pooling).
5.2. Full scene material segmentation
The full test set for our patch dataset contains 41,801
photos, but most of them contain only a few labels. Since
we want to evaluate the per-pixel classiﬁcation performance,
we select a subset of 5,000 test photos such that each photo
contains a large number of segments and clicks, and small
categories are well sampled. We greedily solve for the best
such set of photos. We similarly select 2,500 of 25,844
validation photos. Our splits for all experiments are included
online with the dataset. To train the CRF for our model, we
try various parameter settings ( p,ab,L,wp) and select the
model that performs best on the validation set. In total, we


key: chroma:document
string_value:
try various parameter settings ( p,ab,L,wp) and select the
model that performs best on the validation set. In total, we
evaluate 1799 combinations of CNNs and CRF parameters.
See the supplemental material for a detailed breakdown.
We evaluate multiple versions of GoogLeNet : both the
original architecture and a version with the average pooling
layer (at the end) changed to 5x5, 3x3, and 1x1 (i.e. no
average pooling). We evaluate AlexNet trained at multiple
patch scales (Figure 5). When using an AlexNet trained
at a different scale, we keep the same scale for testing. We
also experiment with ensembles of GoogLeNet andAlexNet ,Architecture(a)Segments only (b)Clicks only
Class Total Class Total
AlexNet Scale: 11.6% 64.3% 72.6% 79.9% 77.2%
AlexNet Scale: 23.3% 69.6% 76.6% 83.3% 81.1%
AlexNet Scale: 32.0% 70.1% 77.1% 83.2% 80.7%
AlexNet Scale: 46.5% 69.6% 75.4% 80.8% 77.7%


key: chroma:document
string_value:
AlexNet Scale: 46.5% 69.6% 75.4% 80.8% 77.7%
AlexNet Scale: 66.2% 67.7% 72.0% 77.2% 72.6%
GoogLeNet 7x7 avg. pool 64.4% 71.6% 63.6% 63.4%
GoogLeNet 5x5 avg. pool 67.6% 74.6% 70.9% 69.8%
GoogLeNet 3x3 avg. pool 70.4% 77.7% 76.1% 74.7%
GoogLeNet No avg. pool 70.4% 78.8% 79.1% 77.4%
Ensemble 2 CNNs 73.1% 79.8% 84.5% 83.1%
Ensemble 3 CNNs 73.1% 79.3% 85.9% 83.5%
Ensemble 4 CNNs 72.1% 78.4% 85.8% 83.2%
Ensemble 5 CNNs 71.7% 78.3% 85.5% 83.2%
Table 4. Full scene material classiﬁcation results. Mean class
and total accuracy on the test set. When training, we optimize the
CRF parameters for mean class accuracy, but report both mean class


key: chroma:document
string_value:
and total accuracy on the test set. When training, we optimize the
CRF parameters for mean class accuracy, but report both mean class
and total accuracy (mean accuracy across all examples). In one
experiment (a), we train and test only on segments; in a separate
experiment (b), we train and test only on clicks. Accuracies for
segments are averaged across all pixels that fall in that segment.
combined with either arithmetic or geometric mean.
Since we have two types of data, clicks andsegments , we
run two sets of experiments: (a) we train and test only on
segments, and in a separate experiment (b) we train and test
only on clicks. These two training objectives result in very
different behavior, as illustrated in Figure 9. In experiment
(a), the accuracy across segments are optimized, producing
clean boundaries. In experiment (b), the CRF maximizes
accuracy only at click locations, thus resulting in sloppy
boundaries. As shown in Table 4, the numerical scores for
the two experiments are also very different: segments are
more challenging than clicks. While clicks are sufﬁcient to
train a CNN, they are not sufﬁcient to train a CRF.
Focusing on segmentation accuracy, we see from Table 4(a) that our best single model is GoogLeNet without


key: chroma:document
string_value:
Focusing on segmentation accuracy, we see from Table 4(a) that our best single model is GoogLeNet without
average pooling (6% better than with pooling). The best
ensemble is 2 CNNs: GoogLeNet (no average pooling) and
AlexNet (patch scale: 46.5%), combined with arithmetic
mean. Larger ensembles perform worse since we are aver7
Correct
Fabric (99%) Foliage (99%) Food (99%) Leather (99%)Correct
Metal (99%) Mirror (99%) Painted (99%) Plastic (99%)Incorrect
T: Wood T: Polished stone T: Water T: Fabric
P: Stone (90%) P: Water (35%) P: Carpet (27%) P: Foliage (67%)
Figure 10. High conﬁdence predictions. Top two rows: correct
predictions. Bottom row: incorrect predictions ( T: true, P: predicted). Percentages indicate conﬁdence (the predictions shown are
at least this conﬁdent). CNN: GoogLeNet.
aging worse CNNs. In Figure 8, we show example labeling
results on test images.
5.3. Comparing MINC to FMD
Compared to FMD, the size and diversity of MINC is


key: chroma:document
string_value:
results on test images.
5.3. Comparing MINC to FMD
Compared to FMD, the size and diversity of MINC is
valuable for classifying real-world imagery. Table 5 shows
the effect of training on all of FMD and testing on MINC
(and vice versa). The results suggests that training on FMD
alone is not sufﬁcient for real-world classiﬁcation. Though it
may seem that our dataset is “easy,” since the best classiﬁcations scores are lower for FMD than for MINC , we ﬁnd that
difﬁculty is in fact closely tied to dataset size (Section 5.1).
Taking 100 random samples per category, AlexNet achieves
54.20.7% on MINC (64.6 1.3% when considering only
the 10 FMD categories) and 66.5% on FMD.
5.4. Comparing CNNs with prior methods
Cimpoi [ 3] is the best prior material classiﬁcation method
on FMD. We ﬁnd that by replacing DeCAF with oversampled AlexNet features we can improve on their FMD results.
We then show that on MINC , a ﬁnetuned CNN is even better.


key: chroma:document
string_value:
We then show that on MINC , a ﬁnetuned CNN is even better.
To improve on [ 3], we take their SIFT IFV, combine it
with AlexNet fc7 features, and add oversampling [ 13] (see
supplemental for details). With a linear SVM we achieve
69.60.3% on FMD. Previous results are listed in Table 6.
Having found that SIFT IFV+fc7 is the new best on
FMD, we compare it to a ﬁnetuned CNN on a subset of MINC (2500 patches per category, one patch per
photo). Fine-tuning AlexNet achieves 76.00.2% whereasTest
FMD MINC
TrainFMD 66.5% 26.1%
MINC 41.7% 85.0%(10 categories
in common)
Table 5. Cross-dataset experiments. We train on one dataset and
test on another dataset. Since MINC contains 23 categories, we
limit MINC to the 10 categories in common. CNN: AlexNet.
Method Accuracy Trials
Sharan et al. [25] 57.10.6% 14 splits
Cimpoi et al. [3] 67.10.4% 14 splits
Fine-tuned AlexNet 66.51.5% 5 folds


key: chroma:document
string_value:
Cimpoi et al. [3] 67.10.4% 14 splits
Fine-tuned AlexNet 66.51.5% 5 folds
SIFT IFV+fc7 69.60.3% 10 splits
Table 6. FMD experiments. By replacing DeCAF features with
oversampled AlexNet features we improve on the best FMD result.
SIFT IFV+fc7 achieves 67.4 0.5% with a linear SVM
(oversampling, 5 splits). This experiment shows that
a ﬁnetuned CNN is a better method for MINC than
SIFT IFV+fc7.
6. Conclusion
Material recognition is a long-standing, challenging problem. We introduce a new large, open, material database,
MINC , that includes a diverse range of materials of everyday scenes and staged designed interiors, and is at least an
order of magnitude larger than prior databases. Using this
large database we conduct an evaluation of recent deep learning algorithms for simultaneous material classiﬁcation and
segmentation, and achieve results that surpass prior attempts
at material recognition.
Some lessons we have learned are:
Training on a dataset which includes the surrounding
context is crucial for real-world material classiﬁcation.


key: chroma:document
string_value:
at material recognition.
Some lessons we have learned are:
Training on a dataset which includes the surrounding
context is crucial for real-world material classiﬁcation.
Labeled clicks are cheap and sufﬁcient to train a CNN
alone. However, to obtain high quality segmentation
results, training a CRF on polygons results in much
better boundaries than training on clicks.
Many future avenues of work remain. Expanding the
dataset to a broader range of categories will require new
ways to mine images that have more variety, and new annotation tasks that are cost-effective. Inspired by attributes
for textures [ 3], in the future we would like to identify material attributes and expand our database to include them. We
also believe that further exploration of joint material and object classiﬁcation and segmentation will be fruitful [ 10] and
lead to improvements in both tasks. Our database, trained
models, and all experimental results are available online at
http://minc.cs.cornell.edu/.
Acknowledgements. This work was supported in part by
Google, Amazon AWS for Education, a NSERC PGS-D
scholarship, the National Science Foundation (grants IIS1149393, IIS-1011919, IIS-1161645), and the Intel Science
and Technology Center for Visual Computing.
8
References


key: chroma:document
string_value:
and Technology Center for Visual Computing.
8
References
[1]S. Bell, P. Upchurch, N. Snavely, and K. Bala. OpenSurfaces:
A richly annotated catalog of surface appearance. ACM Trans.
on Graphics (SIGGRAPH) , 32(4), 2013. 1, 2, 3
[2]B. Caputo, E. Hayman, and P. Mallikarjuna. Class-speciﬁc
material categorisation. In ICCV , pages 1597–1604, 2005. 2
[3]M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and
A. Vedaldi. Describing textures in the wild. In CVPR , pages
3606–3613. IEEE, 2014. 2, 8
[4]K. J. Dana, B. Van Ginneken, S. K. Nayar, and J. J. Koenderink. Reﬂectance and texture of real-world surfaces. ACM
Transactions on Graphics (TOG) , 18(1):1–34, 1999. 2
[5]M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn,
and A. Zisserman. The Pascal Visual Object Classes (VOC)


key: chroma:document
string_value:
and A. Zisserman. The Pascal Visual Object Classes (VOC)
Challenge. IJCV , 88(2):303–338, June 2010. 3
[6]C. Farabet, C. Couprie, L. Najman, and Y . LeCun. Learning
hierarchical features for scene labeling. PAMI , 35(8):1915–
1929, 2013. 2
[7]R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich
feature hierarchies for accurate object detection and semantic
segmentation. In CVPR , 2014. 2
[8]S. Gould, R. Fulton, and D. Koller. Decomposing a scene
into geometric and semantically consistent regions. In ICCV ,
2009. 3
[9]E. Hayman, B. Caputo, M. Fritz, and J. olof Eklundh. On the
signiﬁcance of real-world conditions for material classiﬁcation. In ECCV , 2004. 2
[10] D. Hu, L. Bo, and X. Ren. Toward robust material recognition
for everyday objects. In BMVC , pages 1–11. Citeseer, 2011.
1, 2, 8


key: chroma:document
string_value:
for everyday objects. In BMVC , pages 1–11. Citeseer, 2011.
1, 2, 8
[11] Y . Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell. Caffe: Convolutional
architecture for fast feature embedding. In Proceedings of
the ACM International Conference on Multimedia , pages 675–
678. ACM, 2014. 6
[12] P. Kr ¨ahenb ¨uhl and V . Koltun. Parameter learning and convergent inference for dense random ﬁelds. In ICML , pages
513–521, 2013. 2, 4, 5
[13] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet
classiﬁcation with deep convolutional neural networks. In
Advances in neural information processing systems , pages
1097–1105, 2012. 2, 5, 6, 8
[14] Y . LeCun, B. Boser, J. S. Denker, D. Henderson, R. E.
Howard, W. Hubbard, and L. D. Jackel. Backpropagation


key: chroma:document
string_value:
Howard, W. Hubbard, and L. D. Jackel. Backpropagation
applied to handwritten zip code recognition. Neural computation, 1(4):541–551, 1989. 2
[15] T. Leung and J. Malik. Representing and recognizing the visual appearance of materials using three-dimensional textons.
IJCV , 43(1):29–44, June 2001. 2
[16] W. Li and M. Fritz. Recognizing materials from virtual examples. In ECCV , pages 345–358. Springer, 2012. 2
[17] C. Liu, L. Sharan, E. H. Adelson, and R. Rosenholtz. Exploring features in a bayesian framework for material recognition.
InCVPR , pages 239–246. IEEE, 2010. 1, 2
[18] M. Oquab, L. Bottou, I. Laptev, and J. Sivic. Learning and
transferring mid-level image representations using convolutional neural networks. In CVPR , 2014. 2[19] G. Patterson, C. Xu, H. Su, and J. Hays. The SUN Attribute
Database: Beyond Categories for Deeper Scene Understanding. IJCV , 108(1-2):59–81, 2014. 1


key: chroma:document
string_value:
Database: Beyond Categories for Deeper Scene Understanding. IJCV , 108(1-2):59–81, 2014. 1
[20] X. Qi, R. Xiao, J. Guo, and L. Zhang. Pairwise rotation
invariant co-occurrence local binary pattern. In ECCV , pages
158–171. Springer, 2012. 1, 2
[21] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma,
Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg,
and L. Fei-Fei. ImageNet Large Scale Visual Recognition
Challenge, 2014. 1, 2
[22] B. C. Russell, A. Torralba, K. P. Murphy, and W. T. Freeman. LabelMe: A database and web-based tool for image
annotation. IJCV , 77(1-3):157–173, May 2008. 3
[23] G. Schwartz and K. Nishino. Visual material traits: Recognizing per-pixel material context. In Proceedings of the
International Conference on Computer Vision Workshops (ICCVW) , pages 883–890. IEEE, 2013. 2


key: chroma:document
string_value:
International Conference on Computer Vision Workshops (ICCVW) , pages 883–890. IEEE, 2013. 2
[24] P. Sermanet, D. Eigen, X. Zhang, M. Mathieu, R. Fergus,
and Y . LeCun. Overfeat: Integrated recognition, localization
and detection using convolutional networks. In International
Conference on Learning Representations (ICLR2014) . CBLS,
April 2014. 2, 5
[25] L. Sharan, C. Liu, R. Rosenholtz, and E. Adelson. Recognizing materials using perceptually inspired features. IJCV ,
2013. 1, 8
[26] L. Sharan, R. Rosenholtz, and E. Adelson. Material perception: What can you see in a brief glance? Journal of Vision ,
9(8):784–784, 2009. 1, 2
[27] K. Simonyan and A. Zisserman. Very deep convolutional
networks for large-scale image recognition. arXiv preprint
arXiv:1409.1556 , 2014. 2, 6
[28] C. Szegedy, W. Liu, Y . Jia, P. Sermanet, S. Reed, D. Anguelov,


key: chroma:document
string_value:
[28] C. Szegedy, W. Liu, Y . Jia, P. Sermanet, S. Reed, D. Anguelov,
D. Erhan, V . Vanhoucke, and A. Rabinovich. Going deeper
with convolutions. CVPR , 2015. 2, 6
[29] R. Timofte and L. J. Van Gool. A training-free classiﬁcation
framework for textures, writers, and materials. In BMVC ,
pages 1–12, 2012. 2
[30] M. Varma and A. Zisserman. A statistical approach to texture
classiﬁcation from single images. IJCV , 62(1-2):61–81, Apr.
2005. 2
[31] J. Xiao, K. A. Ehinger, J. Hays, A. Torralba, and A. Oliva.
SUN Database: Exploring a large collection of scene categories. IJCV , 2014. 1, 3
[32] M. D. Zeiler and R. Fergus. Visualizing and understanding
convolutional networks. In ECCV , pages 818–833. Springer,
2014. 2
[33] S. Zheng, M.-M. Cheng, J. Warrell, P. Sturgess, V . Vineet,


key: chroma:document
string_value:
2014. 2
[33] S. Zheng, M.-M. Cheng, J. Warrell, P. Sturgess, V . Vineet,
C. Rother, and P. H. Torr. Dense semantic image segmentation
with objects and attributes. In CVPR , pages 3214–3221. IEEE,
2014. 2
[34] B. Zhou, A. Lapedriza, J. Xiao, A. Torralba, and A. Oliva.
Learning deep features for scene recognition using Places
database. NIPS , 2014. 1, 3
9


key: chroma:document
string_value:
DeepEdge: A Multi-Scale Bifurcated Deep Network
for Top-Down Contour Detection
Gedas Bertasius
University of Pennsylvania
gberta@seas.upenn.eduJianbo Shi
University of Pennsylvania
jshi@seas.upenn.eduLorenzo Torresani
Dartmouth College
lt@dartmouth.edu
Abstract
Contour detection has been a fundamental component
in many image segmentation and object detection systems.
Most previous work utilizes low-level features such as texture or saliency to detect contours and then use them as cues
for a higher-level task such as object detection. However,
we claim that recognizing objects and predicting contours
are two mutually related tasks. Contrary to traditional approaches, we show that we can invert the commonly established pipeline: instead of detecting contours with low-level
cues for a higher-level recognition task, we exploit objectrelated features as high-level cues for contour detection.
We achieve this goal by means of a multi-scale deep network that consists of ﬁve convolutional layers and a bifurcated fully-connected sub-network. The section from the input layer to the ﬁfth convolutional layer is ﬁxed and directly


key: chroma:document
string_value:
lifted from a pre-trained network optimized over a largescale object classiﬁcation task. This section of the network
is applied to four different scales of the image input. These
four parallel and identical streams are then attached to
a bifurcated sub-network consisting of two independentlytrained branches. One branch learns to predict the contour likelihood (with a classiﬁcation objective) whereas the
other branch is trained to learn the fraction of human labelers agreeing about the contour presence at a given point
(with a regression criterion).
We show that without any feature engineering our multiscale deep learning approach achieves state-of-the-art results in contour detection.
1. Introduction
Contour detection is typically considered a low-level
problem, and used to aid higher-level tasks such as object
detection [1, 3, 31, 24]. However, it can be argued that
the tasks of detecting objects and predicting contours are
closely related. For instance, given the contours we can
easily infer which objects are present in the image. Con0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 100.10.20.30.40.50.60.70.80.91
RecallPrecision
  
[F=.80] Human


key: chroma:document
string_value:
RecallPrecision
  
[F=.80] Human
[F=.753] DeepEdge
[F=.753] N4−fields [10]
[F=.747] MCG [2]
[F=.746] SE [8]
[F=.739] SCG [27]
[F=.737] PMI [14]
[F=.727] Sketch Tokens [19]
[F=.726] gPb−owt−ucm [1]Figure 1: Contour detection accuracy on the BSDS500
dataset. Our method attains higher average precision compared to prior methods and state-of-the-art F-score. At low
recall, DeepEdge achieves nearly 100% precision..
versely, if we are given exact locations of the objects we
could predict contours just as easily. A commonly established pipeline in computer vision starts with with low-level
contour prediction and then moves up to higher-level object
detection. However, since we claim that these two tasks are
mutually related, we propose to invert this process. Instead
of using contours as low-level cues for object detection, we
want to use object-speciﬁc information as high-level cues
for contour detection. Thus, in a sense our scheme can be


key: chroma:document
string_value:
want to use object-speciﬁc information as high-level cues
for contour detection. Thus, in a sense our scheme can be
viewed as a top-down approach where object-level cues inform the low-level contour detection process.
In this work, we present a uniﬁed multi-scale deep learning approach that uses higher-level object information to
1arXiv:1412.1123v3  [cs.CV]  23 Apr 2015
predict contours. Speciﬁcally, we present a front-to-end
convolutional architecture where contours are learned directly from raw pixels. Our proposed deep learning architecture reuses features computed by the ﬁrst ﬁve convolutional layers of the network of Krizhevsky et al. [17]. We refer to this network as the KNet . Because the KNet has been
trained for object classiﬁcation, reusing its features enable
our method to incorporate object-level information for contour prediction. In the experimental section, we will show
that this high-level object information greatly improves contour detection results.
Furthermore, our deﬁned architecture operates on multiple scales simultaneously and combines local and global
information from the image, which leads to signiﬁcantly
improved contour detection accuracy rates.


key: chroma:document
string_value:
information from the image, which leads to signiﬁcantly
improved contour detection accuracy rates.
We connect the features computed by the convolutional
layers of the KNet at four different scales of the input with
alearned subnetwork that bifurcates into two branches (the
architecture of our model is illustrated in Fig. 2).
What should the learning objective be? When a human
observer decides if a pixel is a boundary edge, a number
of supporting evidence is used with object level reasoning.
While it is impossible to record such information, we do
have the fraction of observers in agreement for each pixel.
We argue that a learning objective that predicts the fraction
of human labelers in agreement can mimic human reasoning
better.
Thus, in the bifurcated sub-network we optimize the two
branches with different learning objectives. The weights in
one branch are optimized with an edge classiﬁcation objective, while the other branch is trained to predict the fraction of human labelers in agreement, i.e., using a regression
criterion. We show that predictions from the classiﬁcation
branch yield high edge recall, while the outputs of the regression branch have high precision. Thus, fusing these two
outputs allows us to obtain excellent results with respect to


key: chroma:document
string_value:
branch yield high edge recall, while the outputs of the regression branch have high precision. Thus, fusing these two
outputs allows us to obtain excellent results with respect to
both metrics and produce state-of-the-art F-score and average precision.
In summary, the use of higher-level object features, independent optimization of edge classiﬁcation and regression
objectives, as well as a uniﬁed multi-scale architecture are
the key characteristics that allow our method to achieve the
state-of-the-art in contour detection (see Fig. 1).
2. Related Work
Deep Learning. In the recent years, deep convolutional
networks have achieved remarkable results in a wide array
of computer vision tasks [32, 25, 33, 17]. However, thus
far, applications of convolutional networks focused on highlevel vision tasks such as face recognition, image classiﬁcation, pose estimation or scene labeling [32, 25, 33, 17]. Excellent results in these tasks beg the question whether convolutional networks could perform equally well in lowerFigure 2: Visualization of multi-scale DeepEdge network
architecture. To extract candidate contour points, we run the
Canny edge detector. Then, around each candidate point,
we extract patches at four different scales and simultaneously run them through the ﬁve convolutional layers of the


key: chroma:document
string_value:
Canny edge detector. Then, around each candidate point,
we extract patches at four different scales and simultaneously run them through the ﬁve convolutional layers of the
KNet [17]. We connect these convolutional layers to two
separately-trained network branches. The ﬁrst branch is
trained for classiﬁcation, while the second branch is trained
as a regressor. At testing time, the scalar outputs from these
two sub-networks are averaged to produce the ﬁnal score.
level vision tasks such as contour detection. In this paper,
we present a convolutional architecture that achieves stateof-the-art results in a contour detection task, thus demonstrating that convolutional networks can be applied successfully for lower-level vision tasks as well.
Edge Detection. Most of the contour detection methods
can be divided into two branches: local and global methods. Local methods perform contour detection by reasoning about small patches inside the image. Some recent
local methods include sketch tokens [18] and structured
edges [7], Both of these methods are trained in a supervised fashion using a random forest classiﬁer. Sketch tokens [18] pose contour detection as a multi-class classiﬁcation task and predicts a label for each of the pixels individually. Structured edges [7], on the other hand, attempt to


key: chroma:document
string_value:
predict the labels of multiple pixels simultaneously.
Global methods predict contours based on the information from the full image. Some of the most successful approaches in this genre are the MCG detector [2], gPb detector [1] and sparse code gradients [26]. While sparse
code gradients use supervised SVM learning [4], both gPb
and MCG rely on some form of spectral methods. Other
spectral-based methods include Normalized Cuts [30] and
PMI [13].
Recently, there have also been attempts to apply deep
learning methods to the task of contour detection. While
SCT [20] is a sparse coding approach, both N4ﬁelds [9]
and DeepNet [16] use Convolutional Neural Networks
(CNNs) to predict contours. N4ﬁelds rely on dictionary
a) Input Image
 c)2ndlayer
 d)3rdlayer
 e)4thlayer
 e)5thlayer
Figure 3: Visualization of the activation values at the selected convolutional ﬁlters of the KNet (ﬁlters are resized to the
original image dimensions). The ﬁlters in the second layer ﬁre on oriented edges inside the image. The third and fourth


key: chroma:document
string_value:
original image dimensions). The ﬁlters in the second layer ﬁre on oriented edges inside the image. The third and fourth
convolutional layers produce an outline of the shape of the object. The ﬁfth layer ﬁres on the speciﬁc parts of the object.
learning and the use of the Nearest Neighbor algorithm
within a CNN framework while DeepNet uses a traditional
CNN architecture to predict contours.
In comparison to these prior approaches, our work offers
several contributions. First, we deﬁne a novel multi-scale
bifurcated CNN architecture that enables our network to
achieve state-of-the-art contour detection results. Second,
we avoid manual feature engineering by learning contours
directly from raw data. Finally, we believe that we are the
ﬁrst to propose the use of high-level object features for contour detection, thus inverting the traditional pipeline relying on low-level cues for mid-level feature extraction. Our
experiments show that this top-down approach for contour
detection yields state-of-the-art results.
3. The DeepEdge Network
In this section, we describe our proposed deep learning approach for contour detection. For simplicity, we ﬁrst


key: chroma:document
string_value:
3. The DeepEdge Network
In this section, we describe our proposed deep learning approach for contour detection. For simplicity, we ﬁrst
present our architecture in the single-scale scenario (subsection 3.1) and then discuss how to take advantage of multiple
scales (subsection 3.2).
3.1. Single-Scale Architecture
Selection of Candidate Edge Points. To extract a set
of candidate contours with high recall we apply the Canny
edge detector [5] to the input image. For each of these
points we then extract a patch of ﬁxed size such that our
candidate point is the center of the patch. Patches that do
not ﬁt into the image boundaries are padded with the mirror
reﬂections of itself.
Extraction of High-Level Features. We then resize the
patch of ﬁxed size to match the input dimensions of the
KNet [17] and use this network to extract object-level features. The KNet is an appropriate model for our setting as it
has been trained over a large number of object classes (the
1000 categories of the ImageNet dataset [28]) and thus captures features that are generic and useful for many object
categories. While such features have been optimized for the
task of object class recognition, they have been shown to


key: chroma:document
string_value:
categories. While such features have been optimized for the
task of object class recognition, they have been shown to
be highly effective for other image-analysis tasks, includ-ing object detection [10], attribute prediction [34], and image style recognition [15]. The network was trained on 1:2
million images and it includes more than 60million parameters. Its architecture consists of 5convolutional layers and
3fully connected layers. As we intend to use the KNet as
a feature extractor for boundary detection, we utilize only
the ﬁrst 5convolutional layers, which preserve explicit location information before the “spatial scrambling” of the
fully connection layers (note that a spatially variant representation is crucially necessary to predict the presence of
contours at individual pixels). The ﬁrst two KNet convolutional layers learn low-level information. As we move
into the deeper layers, however, we observe that the network
learns higher-level object information. The second convolutional layer seems to encode coherent edge structures. The
third convolutional layer ﬁres at locations corresponding to
prototypical object shapes. The fourth layer appears to generate high responses for full shapes of the object, whereas
the ﬁfth layer ﬁres on the speciﬁc object parts (See Fig. 3).


key: chroma:document
string_value:
the ﬁfth layer ﬁres on the speciﬁc object parts (See Fig. 3).
In order to obtain a representation that captures this hierarchical information, we perform feature extraction at each
of the ﬁve convolutional layers, as shown in Fig. 5. Specifically, we consider a small sub-volume of the feature map
stack produced at each layer. The sub-volume is centered
at the center of the patch in order to assess the presence of
a contour in a small area around the candidate point. We
then perform max,average , and center pooling on this subvolume. This yields a feature descriptor of size 3Fwhere
Fis the number of feature maps computed by the convolutional layer. While max and average pooling are well established operations in deep learning, we deﬁne center pooling
as selecting the center-value from each of the feature maps.
The motivation for center pooling is that for each candidate
point we want to predict the contour presence at that particular point. Because the candidate point is located at the center of the input patch, center pooling extracts the activation
value from the location that corresponds to our candidate
point location.
A Bifurcated Sub-Network. We connect the feature
maps computed via pooling from the ﬁve convolutional


key: chroma:document
string_value:
point location.
A Bifurcated Sub-Network. We connect the feature
maps computed via pooling from the ﬁve convolutional
layers to two separately-trained network branches. Each
Figure 4: A few samples of ground truth data illustrating the difference between the classiﬁcation (ﬁrst row) and the regression
(second row) objectives. The classiﬁcation branch is trained to detect contours that are marked by at least one of the human
annotators. Conversely, the regression branch is optimized to the contour values that depict the fraction of human annotators
agreeing on the contour.
branch consists of two fully-connected layers. The ﬁrst
branch is trained using binary labels, i.e., to perform contour classiﬁcation. This branch is making less selective predictions by classifying whether a given point is a contour or
not. In a sense, this classiﬁcation branch abstracts details
related to the edge structure (orientation, strength, etc) and
simply tries to predict the presence/absence of an edge at
a given point. Due to such abstractions, the classiﬁcation
branch produces contour predictions with high recall.
The second branch is optimized as a regressor to predict


key: chroma:document
string_value:
a given point. Due to such abstractions, the classiﬁcation
branch produces contour predictions with high recall.
The second branch is optimized as a regressor to predict
the fraction of human labelers agreeing about the contour
presence at a particular point. Due to a regression objective,
this branch is much more selective than the ﬁrst branch. Intuitively, the second branch is trained to learn the structural
differences between the contours that are marked by a different fraction of human labelers. For instance, the area that
was labeled as a contour by 80% of human labelers must
be signiﬁcantly different than the area that was labeled as
a contour by 20% human labelers. The regression branch
is trying to learn such differences by predicting the fraction of human labelers who would mark a particular point
as a contour. Thus, in a sense, we are training the regression branch to implicitly mimic how human labelers reason about the contour presence at a given point. In the experimental section, we demonstrate that due to its selectivity, the regression branch produces contour predictions with
very high precision. In Fig. 4, we present several samples
of ground truth data that illustrate the different properties of
our two end-objectives.
The number of hidden layers in the ﬁrst and second fully


key: chroma:document
string_value:
of ground truth data that illustrate the different properties of
our two end-objectives.
The number of hidden layers in the ﬁrst and second fully
connected layers of both branches are 1024 and512, respectively. Both branches optimize the sum of squared difference loss over the (binary or continuous) labels. At test-ing time, the scalar outputs computed from these two subnetworks are averaged to produce a ﬁnal score indicative of
the probability that the candidate point is a contour. Visualization of this architecture is presented in Fig. 5.
In order to train our sub-network, we generate patch examples and labels using training images with ground truth
annotations from multiple human labelers. To generate the
binary labels, we ﬁrst sample 40;000positive examples that
were marked as contours by at least one of the labelers. To
generate negative examples we consider the points that were
selected as candidate contour points by the Canny edge detector but that have not been marked as contours by any of
the human labelers. These are essentially false positives.
For training, we use a random subset of 40;000 of such
points in order to have equal proportion of negative and
positive examples. These 80;000examples are then used
to train our classiﬁcation sub-network.


key: chroma:document
string_value:
points in order to have equal proportion of negative and
positive examples. These 80;000examples are then used
to train our classiﬁcation sub-network.
In addition to the binary labels, we also generate continuous labels that are used to train the regression network.
For this purpose, we deﬁne the regression label of a point to
be the fraction of human labelers that marked the point as a
contour. These 80;000examples with continuous labels are
then also used to train our regression sub-network.
3.2. Multi-Scale Architecture
In the previous section we presented a convolutional architecture for contour prediction utilizing a single scale.
However, in practice, we found that a multi-scale approach
works much better. In this section, we show how to modify
the single-scale architecture so that it can exploit multiple
scales simultaneously.
Rather than extracting a patch at a single scale as we did
in the previous section, in a multi-scale setting we extract
|{z}
|{z}Fixed WeightsLearned WeightsClassiﬁcation Branch
Regression Branch
}
}Figure 5: Detailed illustration of our proposed architecture in a single-scale setting. First, an input patch, centered around the


key: chroma:document
string_value:
Regression Branch
}
}Figure 5: Detailed illustration of our proposed architecture in a single-scale setting. First, an input patch, centered around the
candidate point, goes through ﬁve convolutional layers of the KNet . To extract high-level features, at each convolutional layer
we extract a small sub-volume of the feature map around the center point, and perform max,average , and center pooling on
this sub-volume. The pooled values feed a bifurcated sub-network. At testing time, the scalar outputs computed from the
branches of a bifurcated sub-networks are averaged to produce a ﬁnal contour prediction.
patches around the candidate point for different patch sizes
so that they cover different spatial extents of the image. We
then resize the patches to ﬁt the KNet input and pump them
in parallel through the ﬁve convolutional layers. Our highlevel features are then built by performing max, average and
center pooling in a small sub-volume of the feature map at
each convolutional layer and at each scale. This effectively
increases the dimensionality of the feature vector by a factor
equal to the number of scales compared to the single-scale
setting. These pooled features are then connected as before


key: chroma:document
string_value:
increases the dimensionality of the feature vector by a factor
equal to the number of scales compared to the single-scale
setting. These pooled features are then connected as before
to the two separately-trained network branches. A visualization of our multi-scale architecture is shown in Fig. 2.
3.3. Implementation Details
In this section, we describe additional implementation
details of our model. Our deep network is implemented using the software library Caffe [14].
We use four different scales for our patches. The sizes
of these patches are 6464;128128;196196and a
full-sized image. All of the patches are then resized to the
KNet input dimensions of 227227.
When extracting high-level features from the convolutional layers of KNet , we use sub-volumes of convolutionalfeature maps having spatial sizes 77;55;33;33,
and33for the convolutional layers 1;2;3;4;5, respectively. Note that we shrink the size of the subvolume as we
go deeper in the network since the feature maps get smaller
due to pooling. Our choice of subvolume sizes is made to
ensure we are roughly considering the same spatial extent
of the original image at each layer.
As illustrated in Fig. 5, during the training the weights


key: chroma:document
string_value:
ensure we are roughly considering the same spatial extent
of the original image at each layer.
As illustrated in Fig. 5, during the training the weights
in the convolutional layers are ﬁxed and only the weights in
the fully connected layers of the two branches are learned.
To train our model we use the learning rate of 0:1, the
dropout fraction of 0:5,50number of epochs, and the size
of the batch equal to 100.
As described earlier, to train classiﬁcation and regression branches we sample 80;000examples with binary labels. We also generate continuous labels for these 80;000
examples. In addition, we sample a hold-out dataset of size
40;000. This hold-out dataset is used for the hard-positive
mining step [22].
For the ﬁrst 25epochs we train classiﬁcation and regression branches independently on the original 80;000sample training dataset. After the ﬁrst 25epochs, we test both
branches on the hold-out dataset and detect false negative
predictions made by each branch. We then use these false
negative examples along with the same number of randomly
selected true negative examples to augment our original
80;000training dataset. For the remaining 25epochs, we
train both branches on this augmented dataset.


key: chroma:document
string_value:
selected true negative examples to augment our original
80;000training dataset. For the remaining 25epochs, we
train both branches on this augmented dataset.
The motivation for the hard-positive mining step is to reduce the number of false negative predictions produced by
both branches. By augmenting the original 80;000sized
training data with false negative examples, we are forcing
both branches to focus on hard positive examples, and thus,
effectively reducing the number of false negative predictions.
4. Experiments
In this section, we present our results on the BSDS500
dataset [23], which is arguably the most established benchmark for contour detection. This dataset contains 200training images, 100 validation images, and 200 test images.
Contour detection accuracy is evaluated using three standard measures: ﬁxed contour threshold (ODS), per-image
best threshold (OIS), and average precision (AP).
In section 4.1 we quantitatively compare our approach
to the state-of-the-art. In sections 4.2-4.5 we study how the
performance of our system changes as we modify some of
the architecture choices (number of scales, feature maps,
pooling scheme, training objective). This will cast additional insight into the factors that critically contribute to the
high accuracy of our system.
4.1. Comparison with Other Methods


key: chroma:document
string_value:
pooling scheme, training objective). This will cast additional insight into the factors that critically contribute to the
high accuracy of our system.
4.1. Comparison with Other Methods
We compare the results produced by our approach and
previously proposed contour detection methods. Table 1
summarizes the results. We note that our algorithm achieves
contour detection accuracy that is higher or equal to stateof-the-art results according to two of the three metrics.
Fig. 1 shows the precision and recall curve for the methods considered in our comparison. It also lists the F-score
for each method (in the legend). We observe that there is the
accuracy margin separating our approach from prior techniques. In particular, for low-recall our method achieves almost perfect precision rate. It also produces state-of-the-art
F-score.
4.2. Single Scale versus Multiple Scales
In this section we study the beneﬁts of a multi-scale architecture. Results in Table 2 report accuracy for different numbers and choices of scales. The ﬁrst four rows in
the table illustrate the results achieved using a single-scale
approach. Speciﬁcally, these four cases show performance
obtained when training and testing our system with an input


key: chroma:document
string_value:
the table illustrate the results achieved using a single-scale
approach. Speciﬁcally, these four cases show performance
obtained when training and testing our system with an input
patch of size 6464;128128;196196or a full-sized image, respectively. Note that adding information from multiple scales leads to signiﬁcantly higher F-scores and higherMethod ODS OIS AP
Felz., Hutt. [8] 0.610 0.640 0.560
Mean Shift [6] 0.640 0.680 0.560
Ncuts [30] 0.640 0.680 0.450
SCT [20] 0.710 0.720 0.740
gPb-owt-ucm [1] 0.726 0.757 0.696
Sketch Tokens [18] 0.727 0.746 0.780
PMI [13] 0.737 0.771 0.783
DeepNet [16] 0.738 0.759 0.758
SCG [26] 0.739 0.758 0.773
SE [7] 0.746 0.767 0.803
MCG [2] 0.747 0.779 0.759
N4-ﬁelds [9] 0.753 0.769 0.784


key: chroma:document
string_value:
MCG [2] 0.747 0.779 0.759
N4-ﬁelds [9] 0.753 0.769 0.784
DeepEdge 0.753 0.772 0.807
Table 1: Edge detection results on the BSDS500 benchmark. Our DeepEdge method achieves state-of-the-art contour detections results according to both F-score and AP
metrics.
Scale ODS OIS AP
64 0.71 0.73 0.76
128 0.72 0.74 0.78
196 0.71 0.73 0.76
Full Image 0.67 0.69 0.57
64, 128 0.72 0.75 0.78
64, 128,196 0.72 0.75 0.78
64,128,196,Full Image 0.75 0.77 0.81
Table 2: Results illustrating the effect of using a multi-scale
architecture. Considering multiple scales for contour detection yields signiﬁcantly higher accuracy relative to a single
scale approach.
average precisions. Thus, these results suggest that a multiscale approach is highly advantageous in comparison to a
single scale setting.
4.3. Advantages of Higher-Level Features
In this section, we examine the validity of our earlier


key: chroma:document
string_value:
single scale setting.
4.3. Advantages of Higher-Level Features
In this section, we examine the validity of our earlier
claim that higher-level object-features enhance contour detection accuracy. In Table 3, we present individual contour
detection results using features from the different convolutional layers of KNet . Note that the 4thconvolutional layer
produces the most effective features when considering one
layer at a time. From our earlier discussion we know that
the4thconvolutional layer encodes higher-level object information related to shape and speciﬁc object parts. This
indicates that object speciﬁc cues are particularly beneﬁcial
for contour detection accuracy.
We also observe that by incorporating features from all
Input Image
 Canny Edges
 Raw DeepEdges
 Thresholded DeepEdges
 Ground Truth Edges
Figure 6: Qualitative results produced by our method. Notice how our method learns to distinguish between strong and
weak contours. For instance, in the last row of predictions, contours corresponding to zebra stripes are assigned much lower
probabilities than contours that correspond to the actual object boundaries separating the zebras from the background.
Conv. Layers ODS OIS AP
1st0.66 0.68 0.69
2nd0.71 0.74 0.76


key: chroma:document
string_value:
Conv. Layers ODS OIS AP
1st0.66 0.68 0.69
2nd0.71 0.74 0.76
3rd0.74 0.75 0.79
4th0.74 0.76 0.79
5th0.73 0.74 0.77
All 0.75 0.77 0.81
Table 3: This table shows the advantage of using higherlevel features from the KNet convolutional layers. Individually, the 4thconvolutional layer produces the best contour
prediction results, which implies that higher-level object information is indeed beneﬁcial for contour detection. Combining the features from all convolutional layers leads to
state-of-the-art results.
the convolutional layers, our method achieves state-of-theart contour detection results. This suggests that the features
computed by different layers are complementary and that
considering information from the entire hierarchy is advantageous.4.4. Pooling Scheme
When presenting the architecture of our model, we discussed three different types of pooling: max,average , and
center pooling. These three techniques were used to pool
the values from the sub-volumes extracted around the center point in each convolutional ﬁlter as illustrated in Fig. 5.
We now show how each type of pooling affects contour


key: chroma:document
string_value:
We now show how each type of pooling affects contour
detection results. Table 4 illustrates that, individually, center pooling yields the best contour detection results. This is
expected because the candidate point for which we are trying to predict a contour probability is located at the center
of the input patch.
However, we note that combining all three types of pooling, achieves better contour detection results than any single
pooling technique alone.
4.5. Bifurcation and Training Objective
Next, we want to show that that the two independentlytrained classiﬁcation and regression branches in the bifurcated sub-network provide complementary information that
yields improved contour detection accuracy. In Table 5, we
present contour detection results achieved by using predictions from the individual branches of the bifurcated subnetwork.
Pooling Type ODS OIS AP
Average 0.73 0.75 0.78
Max 0.69 0.72 0.73
Center 0.74 0.76 0.8
Avg+Max+Cen 0.75 0.77 0.81
Table 4: Effect of different pooling schemes on contour detection results. Center pooling produces better results than
max or average pooling. Combining all three types of pooling further improves the results.
Branch ODS OIS AP


key: chroma:document
string_value:
max or average pooling. Combining all three types of pooling further improves the results.
Branch ODS OIS AP
Classiﬁcation 0.75 0.76 0.78
Regression 0.74 0.76 0.80
Classiﬁcation+Regression 0.75 0.77 0.81
Table 5: Contour detection accuracy of the two branches
in our bifurcated sub-network. The classiﬁcation branch
yields solid F-score results whereas the regression branch
achieves high average precision. Averaging the outputs
from these two branches further improve the results.
From these results, we observe that using predictions
just from the classiﬁcation branch produces high F-score
whereas using predictions only from the regression branch
yields high average precision. Combining the predictions
from both branches improves the results according to both
metrics thus, supporting our claim that separately optimizing edge classiﬁcation and regression objectives is beneﬁcial to contour detection.
4.6. Qualitative Results
Finally, we present qualitative results produced by our
method. In Figure 6 we show for each input image example, the set of candidate points produced by the Canny edge
detector, the un-thresholded predictions of our method, the


key: chroma:document
string_value:
method. In Figure 6 we show for each input image example, the set of candidate points produced by the Canny edge
detector, the un-thresholded predictions of our method, the
thresholded predictions, and the ground truth contour map
computed as an average of the multiple manual annotations.
To generate the thresholded predictions, we use a probability threshold of 0:5.
Note that our method successfully distinguishes between
strong and weak contours. Speciﬁcally, observe that in the
last row of Figure 6, our method assigns lower probability
to contours corresponding to zebra stripes compared to the
contours of the actual object boundary separating the zebras
from the background. Thus, in the thresholded version of
the prediction, the weak contours inside the zebra bodies
are removed and we obtain contour predictions that look
very similar to the ground truth.
Due to locality of our method, it may be beneﬁcial
to apply spectral methods [21, 30] or conditional randomﬁelds [27] on top of our method to further improve its performance.
4.7. Computational Cost
In its current form, our method requires about 60KKNet
evaluations ( 15Kper scale) to extract the features. Based
on the runtimes reported in [14], if executed on a GPU our


key: chroma:document
string_value:
evaluations ( 15Kper scale) to extract the features. Based
on the runtimes reported in [14], if executed on a GPU our
method would take about 5 minutes and could be made
faster using the approach described in [12].
An alternative way to dramatically reduce the runtime of
DeepEdge is to interpolate the entries of the feature maps
produced by applying the KNet to the full image rather than
to individual patches. Such an approach would reduce the
number of CNN evaluations needed from 60Kto4(one
for each scale), which would allow our method to run in
real time even on CPUs. We note that interpolation of features in deep layers has been used successfully in several
recent vision papers [29, 11, 19]. Thus, we believe that such
an approach could yield nearly equivalent contour detection
accuracy, up to a small possible degradation caused by interpolation.
Since in this work we were primarily interested in studying the effective advantage enabled by object-level features
in contour detection, we have not invested any effort in optimizing the implementation of our method. This will be one
of our immediate goals in the future.
5. Conclusions
In this work, we presented a multi-scale bifurcated deep
network for top-down contour detection. In the past, contour detection has been approached as a bottom-up task


key: chroma:document
string_value:
network for top-down contour detection. In the past, contour detection has been approached as a bottom-up task
where low-level features are engineered ﬁrst, then contour
detection is performed, and ﬁnally contours may be used
as cues for object detection. However, due to a close relationship between object and contour detection tasks, we
proposed to invert this pipeline and perform contour detection in a top-down fashion. We demonstrated how to use
higher-level object cues to predict contours and showed that
considering higher-level object-features leads to a substantial gain in contour detection accuracy.
Additionally, we demonstrated that our multi-scale architecture is beneﬁcial to contour prediction as well. By
considering multiple scales, our method incorporates local
and global information around the candidate contour points,
which leads to signiﬁcantly better contour detection results.
Furthermore, we showed that independent optimization of
contour classiﬁcation and regression objectives improves
contour prediction accuracy as well. As our experiments indicate, DeepEdge achieves higher average precision results
compared to any prior or concurrent work.
In conclusion, our results suggest that pure CNN systems can be applied successfully to contour detection and
possibly to many other low-level vision tasks.
6. Acknowledgements


key: chroma:document
string_value:
In conclusion, our results suggest that pure CNN systems can be applied successfully to contour detection and
possibly to many other low-level vision tasks.
6. Acknowledgements
We thank Piotr Teterwak, Du Tran and Mohammad Haris
Baig for helpful discussions. This research was funded in
part by NSF award CNS-1205521.
References
[1] P. Arbelaez, M. Maire, C. Fowlkes, and J. Malik. Contour
detection and hierarchical image segmentation. IEEE Trans.
Pattern Anal. Mach. Intell. , 33(5):898–916, May 2011. 1, 2,
6
[2] P. Arbelaez, J. Pont-Tuset, J. Barron, F. Marqu ´es, and J. Malik. Multiscale combinatorial grouping. In Computer Vision
and Pattern Recognition (CVPR) , 2014. 2, 6
[3] E. Borenstein. Combining top-down and bottom-up segmentation. In In Proceedings IEEE workshop on Perceptual Organization in Computer Vision, CVPR , page 46, 2004. 1
[4] C. J. C. Burges. A tutorial on support vector machines for
pattern recognition. Data Mining and Knowledge Discovery ,
2:121–167, 1998. 2


key: chroma:document
string_value:
[4] C. J. C. Burges. A tutorial on support vector machines for
pattern recognition. Data Mining and Knowledge Discovery ,
2:121–167, 1998. 2
[5] J. Canny. A computational approach to edge detection. IEEE
Trans. Pattern Anal. Mach. Intell. , 8(6):679–698, June 1986.
3
[6] D. Comaniciu, P. Meer, and S. Member. Mean shift: A robust
approach toward feature space analysis. IEEE Transactions
on Pattern Analysis and Machine Intelligence , 24:603–619,
2002. 6
[7] P. Doll ´ar and C. L. Zitnick. Fast edge detection using structured forests. PAMI , 2015. 2, 6
[8] P. F. Felzenszwalb and D. P. Huttenlocher. Efﬁcient
graph-based image segmentation. Int. J. Comput. Vision ,
59(2):167–181, Sept. 2004. 6
[9] Y . Ganin and V . S. Lempitsky. N4-ﬁelds: Neural network
nearest neighbor ﬁelds for image transforms. ACCV , 2014.
2, 6


key: chroma:document
string_value:
nearest neighbor ﬁelds for image transforms. ACCV , 2014.
2, 6
[10] R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature hierarchies for accurate object detection and semantic
segmentation. In Proceedings of the IEEE Conference on
Computer Vision and Pattern Recognition (CVPR) , 2014. 3
[11] B. Hariharan, P. A. Arbel ´aez, R. B. Girshick, and J. Malik.
Hypercolumns for object segmentation and ﬁne-grained localization. CoRR , abs/1411.5752, 2014. 8
[12] F. N. Iandola, M. W. Moskewicz, S. Karayev, R. B. Girshick,
T. Darrell, and K. Keutzer. Densenet: Implementing efﬁcient
convnet descriptor pyramids. CoRR , abs/1404.1869, 2014. 8
[13] P. Isola, D. Zoran, D. Krishnan, and E. H. Adelson. Crisp
boundary detection using pointwise mutual information. In
ECCV , 2014. 2, 6


key: chroma:document
string_value:
boundary detection using pointwise mutual information. In
ECCV , 2014. 2, 6
[14] Y . Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell. Caffe: Convolutional architecture for fast feature embedding. arXiv preprint
arXiv:1408.5093 , 2014. 5, 8
[15] S. Karayev, A. Hertzmann, H. Winnemoeller, A. Agarwala, and T. Darrell. Recognizing image style. CoRR ,
abs/1311.3715, 2013. 3[16] J. J. Kivinen, C. K. Williams, N. Heess, and D. Technologies. Visual boundary prediction: A deep neural prediction
network and quality dissection. AISTATS , 1(2):9, 2014. 2, 6
[17] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet
classiﬁcation with deep convolutional neural networks. In
F. Pereira, C. Burges, L. Bottou, and K. Weinberger, editors, Advances in Neural Information Processing Systems 25 ,


key: chroma:document
string_value:
F. Pereira, C. Burges, L. Bottou, and K. Weinberger, editors, Advances in Neural Information Processing Systems 25 ,
pages 1097–1105. Curran Associates, Inc., 2012. 2, 3
[18] J. Lim, C. L. Zitnick, and P. Doll ´ar. Sketch tokens: A learned
mid-level representation for contour and object detection. In
CVPR , 2013. 2, 6
[19] J. Long, E. Shelhamer, and T. Darrell. Fully convolutional
networks for semantic segmentation. CoRR , abs/1411.4038,
2014. 8
[20] M. Maire, S. X. Yu, and P. Perona. Reconstructive sparse
code transfer for contour detection and semantic labeling. In
Asian Conference on Computer Vision (ACCV) , 2014. 2, 6
[21] J. Malik, S. Belongie, T. Leung, and J. Shi. Contour and texture analysis for image segmentation. Int. J. Comput. Vision ,
43(1):7–27, June 2001. 8
[22] T. Malisiewicz, A. Gupta, and A. A. Efros. Ensemble of
exemplar-svms for object detection and beyond. In ICCV ,


key: chroma:document
string_value:
exemplar-svms for object detection and beyond. In ICCV ,
2011. 5
[23] D. Martin, C. Fowlkes, D. Tal, and J. Malik. A database
of human segmented natural images and its application to
evaluating segmentation algorithms and measuring ecological statistics. In Proc. 8th Int’l Conf. Computer Vision , volume 2, pages 416–423, July 2001. 6
[24] A. Opelt and A. Zisserman. A boundary-fragment-model for
object detection. In In ECCV , pages 575–588, 2006. 1
[25] P. H. O. Pinheiro and R. Collobert. Recurrent convolutional
neural networks for scene parsing. CoRR , abs/1306.2795,
2013. 2
[26] X. Ren and L. Bo. Discriminatively Trained Sparse Code
Gradients for Contour Detection. In Advances in Neural Information Processing Systems , December 2012. 2, 6
[27] X. Ren, C. C. Fowlkes, and J. Malik. Scale-invariant contour
completion using condition random ﬁelds. Technical report.
8


key: chroma:document
string_value:
completion using condition random ﬁelds. Technical report.
8
[28] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh,
S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein,
A. C. Berg, and L. Fei-Fei. ImageNet Large Scale Visual
Recognition Challenge, 2014. 3
[29] P. Sermanet, D. Eigen, X. Zhang, M. Mathieu, R. Fergus,
and Y . LeCun. Overfeat: Integrated recognition, localization and detection using convolutional networks. CoRR ,
abs/1312.6229, 2013. 8
[30] J. Shi and J. Malik. Normalized cuts and image segmentation. IEEE Transactions on Pattern Analysis and Machine
Intelligence , 22:888–905, 1997. 2, 6, 8
[31] J. Shotton. Contour-based learning for object detection. In
In Proc. ICCV , pages 503–510, 2005. 1
[32] Y . Taigman, M. Yang, M. Ranzato, and L. Wolf. Deepface:


key: chroma:document
string_value:
[32] Y . Taigman, M. Yang, M. Ranzato, and L. Wolf. Deepface:
Closing the gap to human-level performance in face veriﬁcation. In Conference on Computer Vision and Pattern Recognition (CVPR) , 2014. 2
[33] A. Toshev and C. Szegedy. Deeppose: Human pose estimation via deep neural networks. CoRR , abs/1312.4659, 2013.
2
[34] N. Zhang, M. Paluri, M. Ranzato, T. Darrell, and L. Bourdev.
Panda: Pose aligned networks for deep attribute modeling.
CoRR , abs/1311.5591, 2013. 3


key: chroma:document
string_value:
Published as a conference paper at ICLR 2015
SEMANTIC IMAGE SEGMENTATION WITH DEEP CONVOLUTIONAL NETS AND FULLY CONNECTED CRF S
Liang-Chieh Chen
Univ. of California, Los Angeles
lcchen@cs.ucla.edu
George Papandreou
Google Inc.
gpapan@google.com
Iasonas Kokkinos
CentraleSup ´elec and INRIA
iasonas.kokkinos@ecp.fr
Kevin Murphy
Google Inc.
kpmurphy@google.com
Alan L. Yuille
Univ. of California, Los Angeles
yuille@stat.ucla.edu
ABSTRACT
Deep Convolutional Neural Networks (DCNNs) have recently shown state of the
art performance in high level vision tasks, such as image classiﬁcation and object detection. This work brings together methods from DCNNs and probabilistic
graphical models for addressing the task of pixel-level classiﬁcation (also called
”semantic image segmentation”). We show that responses at the ﬁnal layer of
DCNNs are not sufﬁciently localized for accurate object segmentation. This is


key: chroma:document
string_value:
DCNNs are not sufﬁciently localized for accurate object segmentation. This is
due to the very invariance properties that make DCNNs good for high level tasks.
We overcome this poor localization property of deep networks by combining the
responses at the ﬁnal DCNN layer with a fully connected Conditional Random
Field (CRF). Qualitatively, our “DeepLab” system is able to localize segment
boundaries at a level of accuracy which is beyond previous methods. Quantitatively, our method sets the new state-of-art at the PASCAL VOC-2012 semantic
image segmentation task, reaching 71.6% IOU accuracy in the test set. We show
how these results can be obtained efﬁciently: Careful network re-purposing and a
novel application of the ’hole’ algorithm from the wavelet community allow dense
computation of neural net responses at 8 frames per second on a modern GPU.
1 I NTRODUCTION
Deep Convolutional Neural Networks (DCNNs) had been the method of choice for document recognition since LeCun et al. (1998), but have only recently become the mainstream of high-level vision


key: chroma:document
string_value:
research. Over the past two years DCNNs have pushed the performance of computer vision systems to soaring heights on a broad array of high-level problems, including image classiﬁcation
(Krizhevsky et al., 2013; Sermanet et al., 2013; Simonyan & Zisserman, 2014; Szegedy et al., 2014;
Work initiated when G.P. was with the Toyota Technological Institute at Chicago. The ﬁrst two authors
contributed equally to this work.
1arXiv:1412.7062v4  [cs.CV]  7 Jun 2016
Published as a conference paper at ICLR 2015
Papandreou et al., 2014), object detection (Girshick et al., 2014), ﬁne-grained categorization (Zhang
et al., 2014), among others. A common theme in these works is that DCNNs trained in an end-to-end
manner deliver strikingly better results than systems relying on carefully engineered representations,
such as SIFT or HOG features. This success can be partially attributed to the built-in invariance of
DCNNs to local image transformations, which underpins their ability to learn hierarchical abstractions of data (Zeiler & Fergus, 2014). While this invariance is clearly desirable for high-level vision


key: chroma:document
string_value:
tasks, it can hamper low-level tasks, such as pose estimation (Chen & Yuille, 2014; Tompson et al.,
2014) and semantic segmentation - where we want precise localization, rather than abstraction of
spatial details.
There are two technical hurdles in the application of DCNNs to image labeling tasks: signal downsampling, and spatial ‘insensitivity’ (invariance). The ﬁrst problem relates to the reduction of signal
resolution incurred by the repeated combination of max-pooling and downsampling (‘striding’) performed at every layer of standard DCNNs (Krizhevsky et al., 2013; Simonyan & Zisserman, 2014;
Szegedy et al., 2014). Instead, as in Papandreou et al. (2014), we employ the ‘atrous’ (with holes)
algorithm originally developed for efﬁciently computing the undecimated discrete wavelet transform
(Mallat, 1999). This allows efﬁcient dense computation of DCNN responses in a scheme substantially simpler than earlier solutions to this problem (Giusti et al., 2013; Sermanet et al., 2013).


key: chroma:document
string_value:
The second problem relates to the fact that obtaining object-centric decisions from a classiﬁer requires invariance to spatial transformations, inherently limiting the spatial accuracy of the DCNN
model. We boost our model’s ability to capture ﬁne details by employing a fully-connected Conditional Random Field (CRF). Conditional Random Fields have been broadly used in semantic segmentation to combine class scores computed by multi-way classiﬁers with the low-level information
captured by the local interactions of pixels and edges (Rother et al., 2004; Shotton et al., 2009) or
superpixels (Lucchi et al., 2011). Even though works of increased sophistication have been proposed
to model the hierarchical dependency (He et al., 2004; Ladicky et al., 2009; Lempitsky et al., 2011)
and/or high-order dependencies of segments (Delong et al., 2012; Gonfaus et al., 2010; Kohli et al.,
2009; Chen et al., 2013; Wang et al., 2015), we use the fully connected pairwise CRF proposed by
Kr¨ahenb ¨uhl & Koltun (2011) for its efﬁcient computation, and ability to capture ﬁne edge details


key: chroma:document
string_value:
Kr¨ahenb ¨uhl & Koltun (2011) for its efﬁcient computation, and ability to capture ﬁne edge details
while also catering for long range dependencies. That model was shown in Kr ¨ahenb ¨uhl & Koltun
(2011) to largely improve the performance of a boosting-based pixel-level classiﬁer, and in our work
we demonstrate that it leads to state-of-the-art results when coupled with a DCNN-based pixel-level
classiﬁer.
The three main advantages of our “DeepLab” system are (i) speed: by virtue of the ‘atrous’ algorithm, our dense DCNN operates at 8 fps, while Mean Field Inference for the fully-connected CRF
requires 0.5 second, (ii) accuracy: we obtain state-of-the-art results on the PASCAL semantic segmentation challenge, outperforming the second-best approach of Mostajabi et al. (2014) by a margin
of 7.2 %and (iii) simplicity: our system is composed of a cascade of two fairly well-established modules, DCNNs and CRFs.
2 R ELATED WORK


key: chroma:document
string_value:
of 7.2 %and (iii) simplicity: our system is composed of a cascade of two fairly well-established modules, DCNNs and CRFs.
2 R ELATED WORK
Our system works directly on the pixel representation, similarly to Long et al. (2014). This is in contrast to the two-stage approaches that are now most common in semantic segmentation with DCNNs:
such techniques typically use a cascade of bottom-up image segmentation and DCNN-based region
classiﬁcation, which makes the system commit to potential errors of the front-end segmentation system. For instance, the bounding box proposals and masked regions delivered by (Arbel ´aez et al.,
2014; Uijlings et al., 2013) are used in Girshick et al. (2014) and (Hariharan et al., 2014b) as inputs
to a DCNN to introduce shape information into the classiﬁcation process. Similarly, the authors of
Mostajabi et al. (2014) rely on a superpixel representation. A celebrated non-DCNN precursor to
these works is the second order pooling method of (Carreira et al., 2012) which also assigns labels
to the regions proposals delivered by (Carreira & Sminchisescu, 2012). Understanding the perils


key: chroma:document
string_value:
to the regions proposals delivered by (Carreira & Sminchisescu, 2012). Understanding the perils
of committing to a single segmentation, the authors of Cogswell et al. (2014) build on (Yadollahpour et al., 2013) to explore a diverse set of CRF-based segmentation proposals, computed also by
(Carreira & Sminchisescu, 2012). These segmentation proposals are then re-ranked according to a
DCNN trained in particular for this reranking task. Even though this approach explicitly tries to
handle the temperamental nature of a front-end segmentation algorithm, there is still no explicit ex2
Published as a conference paper at ICLR 2015
ploitation of the DCNN scores in the CRF-based segmentation algorithm: the DCNN is only applied
post-hoc, while it would make sense to directly try to use its results during segmentation.
Moving towards works that lie closer to our approach, several other researchers have considered
the use of convolutionally computed DCNN features for dense image labeling. Among the ﬁrst
have been Farabet et al. (2013) who apply DCNNs at multiple image resolutions and then employ a
segmentation tree to smooth the prediction results; more recently, Hariharan et al. (2014a) propose to


key: chroma:document
string_value:
segmentation tree to smooth the prediction results; more recently, Hariharan et al. (2014a) propose to
concatenate the computed inter-mediate feature maps within the DCNNs for pixel classiﬁcation, and
Dai et al. (2014) propose to pool the inter-mediate feature maps by region proposals. Even though
these works still employ segmentation algorithms that are decoupled from the DCNN classiﬁer’s
results, we believe it is advantageous that segmentation is only used at a later stage, avoiding the
commitment to premature decisions.
More recently, the segmentation-free techniques of (Long et al., 2014; Eigen & Fergus, 2014) directly apply DCNNs to the whole image in a sliding window fashion, replacing the last fully connected layers of a DCNN by convolutional layers. In order to deal with the spatial localization
issues outlined in the beginning of the introduction, Long et al. (2014) upsample and concatenate
the scores from inter-mediate feature maps, while Eigen & Fergus (2014) reﬁne the prediction result
from coarse to ﬁne by propagating the coarse results to another DCNN.
The main difference between our model and other state-of-the-art models is the combination of


key: chroma:document
string_value:
from coarse to ﬁne by propagating the coarse results to another DCNN.
The main difference between our model and other state-of-the-art models is the combination of
pixel-level CRFs and DCNN-based ‘unary terms’. Focusing on the closest works in this direction,
Cogswell et al. (2014) use CRFs as a proposal mechanism for a DCNN-based reranking system,
while Farabet et al. (2013) treat superpixels as nodes for a local pairwise CRF and use graph-cuts for
discrete inference; as such their results can be limited by errors in superpixel computations, while ignoring long-range superpixel dependencies. Our approach instead treats every pixel as a CRF node,
exploits long-range dependencies, and uses CRF inference to directly optimize a DCNN-driven cost
function. We note that mean ﬁeld had been extensively studied for traditional image segmentation/edge detection tasks, e.g., (Geiger & Girosi, 1991; Geiger & Yuille, 1991; Kokkinos et al.,
2008), but recently Kr ¨ahenb ¨uhl & Koltun (2011) showed that the inference can be very efﬁcient for
fully connected CRF and particularly effective in the context of semantic segmentation.


key: chroma:document
string_value:
fully connected CRF and particularly effective in the context of semantic segmentation.
After the ﬁrst version of our manuscript was made publicly available, it came to our attention that
two other groups have independently and concurrently pursued a very similar direction, combining
DCNNs and densely connected CRFs (Bell et al., 2014; Zheng et al., 2015). There are several
differences in technical aspects of the respective models. Bell et al. (2014) focus on the problem
of material classiﬁcation, while Zheng et al. (2015) unroll the CRF mean-ﬁeld inference steps to
convert the whole system into an end-to-end trainable feed-forward network.
We have updated our proposed “DeepLab” system with much improved methods and results in our
latest work (Chen et al., 2016). We refer the interested reader to the paper for details.
3 C ONVOLUTIONAL NEURAL NETWORKS FOR DENSE IMAGE LABELING
Herein we describe how we have re-purposed and ﬁnetuned the publicly available Imagenetpretrained state-of-art 16-layer classiﬁcation network of (Simonyan & Zisserman, 2014) (VGG-16)
into an efﬁcient and effective dense feature extractor for our dense semantic image segmentation


key: chroma:document
string_value:
into an efﬁcient and effective dense feature extractor for our dense semantic image segmentation
system.
3.1 E FFICIENT DENSE SLIDING WINDOW FEATURE EXTRACTION WITH THE HOLE
ALGORITHM
Dense spatial score evaluation is instrumental in the success of our dense CNN feature extractor. As
a ﬁrst step to implement this, we convert the fully-connected layers of VGG-16 into convolutional
ones and run the network in a convolutional fashion on the image at its original resolution. However
this is not enough as it yields very sparsely computed detection scores (with a stride of 32 pixels). To
compute scores more densely at our target stride of 8 pixels, we develop a variation of the method
previously employed by Giusti et al. (2013); Sermanet et al. (2013). We skip subsampling after
the last two max-pooling layers in the network of Simonyan & Zisserman (2014) and modify the
convolutional ﬁlters in the layers that follow them by introducing zeros to increase their length ( 2in
3
Published as a conference paper at ICLR 2015
Input stride
Output stride
Figure 1: Illustration of the hole algorithm in 1-D, when kernel size = 3 ,input stride = 2 , and


key: chroma:document
string_value:
Input stride
Output stride
Figure 1: Illustration of the hole algorithm in 1-D, when kernel size = 3 ,input stride = 2 , and
output stride = 1 .
the last three convolutional layers and 4in the ﬁrst fully connected layer). We can implement this
more efﬁciently by keeping the ﬁlters intact and instead sparsely sample the feature maps on which
they are applied on using an input stride of 2 or 4 pixels, respectively. This approach, illustrated
in Fig. 1 is known as the ‘hole algorithm’ (‘atrous algorithm’) and has been developed before for
efﬁcient computation of the undecimated wavelet transform (Mallat, 1999). We have implemented
this within the Caffe framework (Jia et al., 2014) by adding to the im2col function (it converts multichannel feature maps to vectorized patches) the option to sparsely sample the underlying feature
map. This approach is generally applicable and allows us to efﬁciently compute dense CNN feature
maps at any target subsampling rate without introducing any approximations.
We ﬁnetune the model weights of the Imagenet-pretrained VGG-16 network to adapt it to the image


key: chroma:document
string_value:
We ﬁnetune the model weights of the Imagenet-pretrained VGG-16 network to adapt it to the image
classiﬁcation task in a straightforward fashion, following the procedure of Long et al. (2014). We
replace the 1000-way Imagenet classiﬁer in the last layer of VGG-16 with a 21-way one. Our
loss function is the sum of cross-entropy terms for each spatial position in the CNN output map
(subsampled by 8 compared to the original image). All positions and labels are equally weighted in
the overall loss function. Our targets are the ground truth labels (subsampled by 8). We optimize the
objective function with respect to the weights at all network layers by the standard SGD procedure
of Krizhevsky et al. (2013).
During testing, we need class score maps at the original image resolution. As illustrated in Figure 2
and further elaborated in Section 4.1, the class score maps (corresponding to log-probabilities) are
quite smooth, which allows us to use simple bilinear interpolation to increase their resolution by a
factor of 8 at a negligible computational cost. Note that the method of Long et al. (2014) does not
use the hole algorithm and produces very coarse scores (subsampled by a factor of 32) at the CNN


key: chroma:document
string_value:
use the hole algorithm and produces very coarse scores (subsampled by a factor of 32) at the CNN
output. This forced them to use learned upsampling layers, signiﬁcantly increasing the complexity
and training time of their system: Fine-tuning our network on PASCAL VOC 2012 takes about 10
hours, while they report a training time of several days (both timings on a modern GPU).
3.2 C ONTROLLING THE RECEPTIVE FIELD SIZE AND ACCELERATING DENSE
COMPUTATION WITH CONVOLUTIONAL NETS
Another key ingredient in re-purposing our network for dense score computation is explicitly controlling the network’s receptive ﬁeld size. Most recent DCNN-based image recognition methods
rely on networks pre-trained on the Imagenet large-scale classiﬁcation task. These networks typically have large receptive ﬁeld size: in the case of the VGG-16 net we consider, its receptive ﬁeld
is224224(with zero-padding) and 404404pixels if the net is applied convolutionally. After


key: chroma:document
string_value:
is224224(with zero-padding) and 404404pixels if the net is applied convolutionally. After
converting the network to a fully convolutional one, the ﬁrst fully connected layer has 4,096 ﬁlters of large 77spatial size and becomes the computational bottleneck in our dense score map
computation.
We have addressed this practical problem by spatially subsampling (by simple decimation) the ﬁrst
FC layer to 44(or33) spatial size. This has reduced the receptive ﬁeld of the network down to
128128(with zero-padding) or 308308(in convolutional mode) and has reduced computation time
for the ﬁrst FC layer by 23times. Using our Caffe-based implementation and a Titan GPU, the
resulting VGG-derived network is very efﬁcient: Given a 306306input image, it produces 3939
4
Published as a conference paper at ICLR 2015
dense raw feature scores at the top of the network at a rate of about 8 frames/sec during testing. The
speed during training is 3 frames/sec. We have also successfully experimented with reducing the
number of channels at the fully connected layers from 4,096 down to 1,024, considerably further


key: chroma:document
string_value:
speed during training is 3 frames/sec. We have also successfully experimented with reducing the
number of channels at the fully connected layers from 4,096 down to 1,024, considerably further
decreasing computation time and memory footprint without sacriﬁcing performance, as detailed in
Section 5. Using smaller networks such as Krizhevsky et al. (2013) could allow video-rate test-time
dense feature computation even on light-weight GPUs.
4 D ETAILED BOUNDARY RECOVERY : FULLY -CONNECTED CONDITIONAL
RANDOM FIELDS AND MULTI -SCALE PREDICTION
4.1 D EEPCONVOLUTIONAL NETWORKS AND THE LOCALIZATION CHALLENGE
As illustrated in Figure 2, DCNN score maps can reliably predict the presence and rough position
of objects in an image but are less well suited for pin-pointing their exact outline. There is a natural
trade-off between classiﬁcation accuracy and localization accuracy with convolutional networks:
Deeper models with multiple max-pooling layers have proven most successful in classiﬁcation tasks,
however their increased invariance and large receptive ﬁelds make the problem of inferring position
from the scores at their top output levels more challenging.


key: chroma:document
string_value:
however their increased invariance and large receptive ﬁelds make the problem of inferring position
from the scores at their top output levels more challenging.
Recent work has pursued two directions to address this localization challenge. The ﬁrst approach is
to harness information from multiple layers in the convolutional network in order to better estimate
the object boundaries (Long et al., 2014; Eigen & Fergus, 2014). The second approach is to employ
a super-pixel representation, essentially delegating the localization task to a low-level segmentation
method. This route is followed by the very successful recent method of Mostajabi et al. (2014).
In Section 4.2, we pursue a novel alternative direction based on coupling the recognition capacity
of DCNNs and the ﬁne-grained localization accuracy of fully connected CRFs and show that it is
remarkably successful in addressing the localization challenge, producing accurate semantic segmentation results and recovering object boundaries at a level of detail that is well beyond the reach
of existing methods.
4.2 F ULLY -CONNECTED CONDITIONAL RANDOM FIELDS FOR ACCURATE LOCALIZATION
Image/G.T. DCNN output CRF Iteration 1 CRF Iteration 2 CRF Iteration 10


key: chroma:document
string_value:
Image/G.T. DCNN output CRF Iteration 1 CRF Iteration 2 CRF Iteration 10
Figure 2: Score map (input before softmax function) and belief map (output of softmax function) for
Aeroplane. We show the score (1st row) and belief (2nd row) maps after each mean ﬁeld iteration.
The output of last DCNN layer is used as input to the mean ﬁeld inference. Best viewed in color.
Traditionally, conditional random ﬁelds (CRFs) have been employed to smooth noisy segmentation
maps (Rother et al., 2004; Kohli et al., 2009). Typically these models contain energy terms that
couple neighboring nodes, favoring same-label assignments to spatially proximal pixels. Qualitatively, the primary function of these short-range CRFs has been to clean up the spurious predictions
of weak classiﬁers built on top of local hand-engineered features.
Compared to these weaker classiﬁers, modern DCNN architectures such as the one we use in this
work produce score maps and semantic label predictions which are qualitatively different. As illustrated in Figure 2, the score maps are typically quite smooth and produce homogeneous classiﬁcation


key: chroma:document
string_value:
work produce score maps and semantic label predictions which are qualitatively different. As illustrated in Figure 2, the score maps are typically quite smooth and produce homogeneous classiﬁcation
results. In this regime, using short-range CRFs can be detrimental, as our goal should be to recover
detailed local structure rather than further smooth it. Using contrast-sensitive potentials (Rother
5
Published as a conference paper at ICLR 2015
Deep 
Convolutional 
Neural 
Network
InputAeroplane
Coarse Score map
Bi-linear Interpolation Fully Connected CRF Final Output
Figure 3: Model Illustration. The coarse score map from Deep Convolutional Neural Network (with
fully convolutional layers) is upsampled by bi-linear interpolation. A fully connected CRF is applied
to reﬁne the segmentation result. Best viewed in color.
et al., 2004) in conjunction to local-range CRFs can potentially improve localization but still miss
thin-structures and typically requires solving an expensive discrete optimization problem.
To overcome these limitations of short-range CRFs, we integrate into our system the fully connected
CRF model of Kr ¨ahenb ¨uhl & Koltun (2011). The model employs the energy function
E(x) =X
ii(xi) +X


key: chroma:document
string_value:
E(x) =X
ii(xi) +X
ijij(xi;xj) (1)
where xis the label assignment for pixels. We use as unary potential i(xi) =logP(xi), where
P(xi)is the label assignment probability at pixel ias computed by DCNN. The pairwise potential
isij(xi;xj) =(xi;xj)PK
m=1wmkm(fi;fj), where(xi;xj) = 1 ifxi6=xj, and zero
otherwise ( i.e., Potts Model). There is one pairwise term for each pair of pixels iandjin the image
no matter how far from each other they lie, i.e. the model’s factor graph is fully connected. Each km
is the Gaussian kernel depends on features (denoted as f) extracted for pixel iandjand is weighted
by parameter wm. We adopt bilateral position and color terms, speciﬁcally, the kernels are
w1exp
jjpipjjj2
22jjIiIjjj2
22

+w2exp
jjpipjjj2
22

(2)


key: chroma:document
string_value:
22

+w2exp
jjpipjjj2
22

(2)
where the ﬁrst kernel depends on both pixel positions (denoted as p) and pixel color intensities
(denoted asI), and the second kernel only depends on pixel positions. The hyper parameters ,
and
control the “scale” of the Gaussian kernels.
Crucially, this model is amenable to efﬁcient approximate probabilistic inference (Kr ¨ahenb ¨uhl &
Koltun, 2011). The message passing updates under a fully decomposable mean ﬁeld approximationb(x) =Q
ibi(xi)can be expressed as convolutions with a Gaussian kernel in feature space.
High-dimensional ﬁltering algorithms (Adams et al., 2010) signiﬁcantly speed-up this computation
resulting in an algorithm that is very fast in practice, less that 0.5 sec on average for Pascal VOC
images using the publicly available implementation of (Kr ¨ahenb ¨uhl & Koltun, 2011).
4.3 M ULTI -SCALE PREDICTION


key: chroma:document
string_value:
images using the publicly available implementation of (Kr ¨ahenb ¨uhl & Koltun, 2011).
4.3 M ULTI -SCALE PREDICTION
Following the promising recent results of (Hariharan et al., 2014a; Long et al., 2014) we have also
explored a multi-scale prediction method to increase the boundary localization accuracy. Specifically, we attach to the input image and the output of each of the ﬁrst four max pooling layers a
two-layer MLP (ﬁrst layer: 128 3x3 convolutional ﬁlters, second layer: 128 1x1 convolutional ﬁlters) whose feature map is concatenated to the main network’s last layer feature map. The aggregate
feature map fed into the softmax layer is thus enhanced by 5 * 128 = 640 channels. We only adjust
the newly added weights, keeping the other network parameters to the values learned by the method
of Section 3. As discussed in the experimental section, introducing these extra direct connections
from ﬁne-resolution layers improves localization performance, yet the effect is not as dramatic as
the one obtained with the fully-connected CRF.
6
Published as a conference paper at ICLR 2015
Method mean IOU (%)
DeepLab 59.80
DeepLab-CRF 63.74


key: chroma:document
string_value:
6
Published as a conference paper at ICLR 2015
Method mean IOU (%)
DeepLab 59.80
DeepLab-CRF 63.74
DeepLab-MSc 61.30
DeepLab-MSc-CRF 65.21
DeepLab-7x7 64.38
DeepLab-CRF-7x7 67.64
DeepLab-LargeFOV 62.25
DeepLab-CRF-LargeFOV 67.64
DeepLab-MSc-LargeFOV 64.21
DeepLab-MSc-CRF-LargeFOV 68.70Method mean IOU (%)
MSRA-CFM 61.8
FCN-8s 62.2
TTI-Zoomout-16 64.4
DeepLab-CRF 66.4
DeepLab-MSc-CRF 67.1
DeepLab-CRF-7x7 70.3
DeepLab-CRF-LargeFOV 70.3
DeepLab-MSc-CRF-LargeFOV 71.6
(a) (b)
Table 1: (a) Performance of our proposed models on the PASCAL VOC 2012 ‘val’ set (with training
in the augmented ‘train’ set). The best performance is achieved by exploiting both multi-scale


key: chroma:document
string_value:
in the augmented ‘train’ set). The best performance is achieved by exploiting both multi-scale
features and large ﬁeld-of-view. (b) Performance of our proposed models (with training in the
augmented ‘trainval’ set) compared to other state-of-art methods on the PASCAL VOC 2012 ‘test’
set.
5 E XPERIMENTAL EVALUATION
Dataset We test our DeepLab model on the PASCAL VOC 2012 segmentation benchmark (Everingham et al., 2014), consisting of 20 foreground object classes and one background class. The
original dataset contains 1;464,1;449, and 1;456images for training, validation, and testing, respectively. The dataset is augmented by the extra annotations provided by Hariharan et al. (2011),
resulting in 10;582training images. The performance is measured in terms of pixel intersectionover-union (IOU) averaged across the 21 classes.
Training We adopt the simplest form of piecewise training, decoupling the DCNN and CRF training stages, assuming the unary terms provided by the DCNN are ﬁxed during CRF training.
For DCNN training we employ the VGG-16 network which has been pre-trained on ImageNet. We


key: chroma:document
string_value:
For DCNN training we employ the VGG-16 network which has been pre-trained on ImageNet. We
ﬁne-tuned the VGG-16 network on the VOC 21-way pixel-classiﬁcation task by stochastic gradient
descent on the cross-entropy loss function, as described in Section 3.1. We use a mini-batch of 20
images and initial learning rate of 0:001(0:01for the ﬁnal classiﬁer layer), multiplying the learning
rate by 0.1 at every 2000 iterations. We use momentum of 0:9and a weight decay of 0:0005 .
After the DCNN has been ﬁne-tuned, we cross-validate the parameters of the fully connected CRF
model in Eq. (2) along the lines of Kr ¨ahenb ¨uhl & Koltun (2011). We use the default values of
w2= 3 and
= 3 and we search for the best values of w1,, andby cross-validation on a
small subset of the validation set (we use 100 images). We employ coarse-to-ﬁne search scheme.


key: chroma:document
string_value:
small subset of the validation set (we use 100 images). We employ coarse-to-ﬁne search scheme.
Speciﬁcally, the initial search range of the parameters are w12[5;10],2[50 : 10 : 100] and
2[3 : 1 : 10] (MATLAB notation), and then we reﬁne the search step sizes around the ﬁrst
round’s best values. We ﬁx the number of mean ﬁeld iterations to 10 for all reported experiments.
Evaluation on Validation set We conduct the majority of our evaluations on the PASCAL ‘val’
set, training our model on the augmented PASCAL ‘train’ set. As shown in Tab. 1 (a), incorporating
the fully connected CRF to our model (denoted by DeepLab-CRF) yields a substantial performance
boost, about 4% improvement over DeepLab. We note that the work of Kr ¨ahenb ¨uhl & Koltun
(2011) improved the 27:6%result of TextonBoost (Shotton et al., 2009) to 29:1%, which makes the
improvement we report here (from 59:8%to63:7%) all the more impressive.


key: chroma:document
string_value:
improvement we report here (from 59:8%to63:7%) all the more impressive.
Turning to qualitative results, we provide visual comparisons between DeepLab and DeepLab-CRF
in Fig. 7. Employing a fully connected CRF signiﬁcantly improves the results, allowing the model
to accurately capture intricate object boundaries.
Multi-Scale features We also exploit the features from the intermediate layers, similar to Hariharan et al. (2014a); Long et al. (2014). As shown in Tab. 1 (a), adding the multi-scale features to our
7
Published as a conference paper at ICLR 2015
Method kernel size input stride receptive ﬁeld # parameters mean IOU (%) Training speed (img/sec)
DeepLab-CRF-7x7 77 4 224 134.3M 67.64 1.44
DeepLab-CRF 44 4 128 65.1M 63.74 2.90
DeepLab-CRF-4x4 44 8 224 65.1M 67.14 2.90
DeepLab-CRF-LargeFOV 33 12 224 20.5M 67.64 4.84
Table 2: Effect of Field-Of-View. We show the performance (after CRF) and training speed on the


key: chroma:document
string_value:
Table 2: Effect of Field-Of-View. We show the performance (after CRF) and training speed on the
PASCAL VOC 2012 ‘val’ set as the function of (1) the kernel size of ﬁrst fully connected layer, (2)
the input stride value employed in the atrous algorithm.
DeepLab model (denoted as DeepLab-MSc) improves about 1:5%performance, and further incorporating the fully connected CRF (denoted as DeepLab-MSc-CRF) yields about 4% improvement.
The qualitative comparisons between DeepLab and DeepLab-MSc are shown in Fig. 4. Leveraging
the multi-scale features can slightly reﬁne the object boundaries.
Field of View The ‘atrous algorithm’ we employed allows us to arbitrarily control the Field-ofView (FOV) of the models by adjusting the input stride, as illustrated in Fig. 1. In Tab. 2, we
experiment with several kernel sizes and input strides at the ﬁrst fully connected layer. The method,
DeepLab-CRF-7x7, is the direct modiﬁcation from VGG-16 net, where the kernel size = 77and
input stride = 4. This model yields performance of 67:64% on the ‘val’ set, but it is relatively slow


key: chroma:document
string_value:
input stride = 4. This model yields performance of 67:64% on the ‘val’ set, but it is relatively slow
(1:44images per second during training). We have improved model speed to 2:9images per second
by reducing the kernel size to 44. We have experimented with two such network variants with
different FOV sizes, DeepLab-CRF and DeepLab-CRF-4x4; the latter has large FOV ( i.e., large
input stride) and attains better performance. Finally, we employ kernel size 33and input stride =
12, and further change the ﬁlter sizes from 4096 to 1024 for the last two layers. Interestingly, the
resulting model, DeepLab-CRF-LargeFOV , matches the performance of the expensive DeepLabCRF-7x7. At the same time, it is 3:36times faster to run and has signiﬁcantly fewer parameters
(20.5M instead of 134.3M).
The performance of several model variants is summarized in Tab. 1, showing the beneﬁt of exploiting
multi-scale features and large FOV .
Figure 4: Incorporating multi-scale features improves the boundary segmentation. We show the


key: chroma:document
string_value:
multi-scale features and large FOV .
Figure 4: Incorporating multi-scale features improves the boundary segmentation. We show the
results obtained by DeepLab and DeepLab-MSc in the ﬁrst and second row, respectively. Best
viewed in color.
Mean Pixel IOU along Object Boundaries To quantify the accuracy of the proposed model near
object boundaries, we evaluate the segmentation accuracy with an experiment similar to Kohli et al.
(2009); Kr ¨ahenb ¨uhl & Koltun (2011). Speciﬁcally, we use the ‘void’ label annotated in val set,
which usually occurs around object boundaries. We compute the mean IOU for those pixels that
are located within a narrow band (called trimap) of ‘void’ labels. As shown in Fig. 5, exploiting
the multi-scale features from the intermediate layers and reﬁning the segmentation results by a fully
connected CRF signiﬁcantly improve the results around object boundaries.
Comparison with State-of-art In Fig. 6, we qualitatively compare our proposed model, DeepLabCRF, with two state-of-art models: FCN-8s (Long et al., 2014) and TTI-Zoomout-16 (Mostajabi


key: chroma:document
string_value:
et al., 2014) on the ‘val’ set (the results are extracted from their papers). Our model is able to
capture the intricate object boundaries.
8
Published as a conference paper at ICLR 2015
0 510 15 20 25 30 35 405560657075808590Pixelwise Accuracy (%)
Trimap Width (pixels)  
DL−MSc−CRF
DeepLab−CRF
DeepLab−MSc
DeepLab
0 510 15 20 25 30 35 4035404550556065mean IOU (%)
Trimap Width (pixels)  
DL−MSc−CRF
DeepLab−CRF
DeepLab−MSc
DeepLab
(a) (b) (c)
Figure 5: (a) Some trimap examples (top-left: image. top-right: ground-truth. bottom-left: trimap
of 2 pixels. bottom-right: trimap of 10 pixels). Quality of segmentation result within a band around
the object boundaries for the proposed methods. (b) Pixelwise accuracy. (c) Pixel mean IOU.
(a) FCN-8s vs. DeepLab-CRF (b) TTI-Zoomout-16 vs. DeepLab-CRF


key: chroma:document
string_value:
(a) FCN-8s vs. DeepLab-CRF (b) TTI-Zoomout-16 vs. DeepLab-CRF
Figure 6: Comparisons with state-of-the-art models on the val set. First row: images. Second row:
ground truths. Third row: other recent models (Left: FCN-8s, Right: TTI-Zoomout-16). Fourth
row: our DeepLab-CRF. Best viewed in color.
Reproducibility We have implemented the proposed methods by extending the excellent Caffe
framework (Jia et al., 2014). We share our source code, conﬁguration ﬁles, and trained models that
allow reproducing the results in this paper at a companion web site https://bitbucket.org/
deeplab/deeplab-public .
Test set results Having set our model choices on the validation set, we evaluate our model variants
on the PASCAL VOC 2012 ofﬁcial ‘test’ set. As shown in Tab. 3, our DeepLab-CRF and DeepLabMSc-CRF models achieve performance of 66:4%and67:1%mean IOU1, respectively. Our models


key: chroma:document
string_value:
outperform all the other state-of-the-art models (speciﬁcally, TTI-Zoomout-16 (Mostajabi et al.,
2014), FCN-8s (Long et al., 2014), and MSRA-CFM (Dai et al., 2014)). When we increase the FOV
of the models, DeepLab-CRF-LargeFOV yields performance of 70:3%, the same as DeepLab-CRF7x7, while its training speed is faster. Furthermore, our best model, DeepLab-MSc-CRF-LargeFOV ,
attains the best performance of 71:6%by employing both multi-scale features and large FOV .
1http://host.robots.ox.ac.uk:8080/leaderboard/displaylb.php?
challengeid=11&compid=6
9
Published as a conference paper at ICLR 2015
Figure 7: Visualization results on VOC 2012-val. For each row, we show the input image, the
segmentation result delivered by the DCNN (DeepLab), and the reﬁned segmentation result of the
Fully Connected CRF (DeepLab-CRF). We show our failure modes in the last three rows. Best
viewed in color.
10


key: chroma:document
string_value:
Fully Connected CRF (DeepLab-CRF). We show our failure modes in the last three rows. Best
viewed in color.
10
Published as a conference paper at ICLR 2015
Method bkg aero bike bird boat bottle bus car cat chair cow table dog horse mbike person plant sheep sofa train tv mean
MSRA-CFM -75.7 26.7 69.5 48.8 65.6 81.0 69.2 73.3 30.0 68.7 51.5 69.1 68.1 71.7 67.5 50.4 66.5 44.4 58.9 53.5 61.8
FCN-8s -76.8 34.2 68.9 49.4 60.3 75.3 74.7 77.6 21.4 62.5 46.8 71.8 63.9 76.5 73.9 45.2 72.4 37.4 70.9 55.1 62.2
TTI-Zoomout-16 89.8 81.9 35.1 78.2 57.4 56.5 80.5 74.0 79.8 22.4 69.6 53.7 74.0 76.0 76.6 68.8 44.3 70.2 40.2 68.9 55.3 64.4


key: chroma:document
string_value:
DeepLab-CRF 92.1 78.4 33.1 78.2 55.6 65.3 81.3 75.5 78.6 25.3 69.2 52.7 75.2 69.0 79.1 77.6 54.7 78.3 45.1 73.3 56.2 66.4
DeepLab-MSc-CRF 92.6 80.4 36.8 77.4 55.2 66.4 81.5 77.5 78.9 27.1 68.2 52.7 74.3 69.6 79.4 79.0 56.9 78.8 45.2 72.7 59.3 67.1
DeepLab-CRF-7x7 92.8 83.9 36.6 77.5 58.4 68.0 84.6 79.7 83.1 29.5 74.6 59.3 78.9 76.0 82.1 80.6 60.3 81.7 49.2 78.0 60.7 70.3
DeepLab-CRF-LargeFOV 92.6 83.5 36.6 82.5 62.3 66.5 85.4 78.5 83.7 30.4 72.9 60.4 78.5 75.5 82.1 79.7 58.2 82.0 48.8 73.7 63.3 70.3


key: chroma:document
string_value:
DeepLab-MSc-CRF-LargeFOV 93.1 84.4 54.5 81.5 63.6 65.9 85.1 79.1 83.4 30.7 74.1 59.8 79.0 76.1 83.2 80.8 59.7 82.2 50.4 73.1 63.7 71.6
Table 3: Labeling IOU (%) on the PASCAL VOC 2012 test set, using the trainval set for training.
6 D ISCUSSION
Our work combines ideas from deep convolutional neural networks and fully-connected conditional
random ﬁelds, yielding a novel method able to produce semantically accurate predictions and detailed segmentation maps, while being computationally efﬁcient. Our experimental results show that
the proposed method signiﬁcantly advances the state-of-art in the challenging PASCAL VOC 2012
semantic image segmentation task.
There are multiple aspects in our model that we intend to reﬁne, such as fully integrating its two
main components (CNN and CRF) and train the whole system in an end-to-end fashion, similar to
Kr¨ahenb ¨uhl & Koltun (2013); Chen et al. (2014); Zheng et al. (2015). We also plan to experiment


key: chroma:document
string_value:
Kr¨ahenb ¨uhl & Koltun (2013); Chen et al. (2014); Zheng et al. (2015). We also plan to experiment
with more datasets and apply our method to other sources of data such as depth maps or videos. Recently, we have pursued model training with weakly supervised annotations, in the form of bounding
boxes or image-level labels (Papandreou et al., 2015).
At a higher level, our work lies in the intersection of convolutional neural networks and probabilistic
graphical models. We plan to further investigate the interplay of these two powerful classes of
methods and explore their synergistic potential for solving challenging computer vision tasks.
ACKNOWLEDGMENTS
This work was partly supported by ARO 62250-CS, NIH Grant 5R01EY022247-03, EU Project
RECONFIG FP7-ICT-600825 and EU Project MOBOT FP7-ICT-2011-600796. We also gratefully
acknowledge the support of NVIDIA Corporation with the donation of GPUs used for this research.
We would like to thank the anonymous reviewers for their detailed comments and constructive feedback.
PAPER REVISIONS
Here we present the list of major paper revisions for the convenience of the readers.
v1 Submission to ICLR 2015. Introduces the model DeepLab-CRF, which attains the performance


key: chroma:document
string_value:
v1 Submission to ICLR 2015. Introduces the model DeepLab-CRF, which attains the performance
of66:4%on PASCAL VOC 2012 test set.
v2 Rebuttal for ICLR 2015. Adds the model DeepLab-MSc-CRF, which incorporates multi-scale
features from the intermediate layers. DeepLab-MSc-CRF yields the performance of 67:1%on
PASCAL VOC 2012 test set.
v3 Camera-ready for ICLR 2015. Experiments with large Field-Of-View. On PASCAL VOC 2012
test set, DeepLab-CRF-LargeFOV achieves the performance of 70:3%. When exploiting both mutliscale features and large FOV , DeepLab-MSc-CRF-LargeFOV attains the performance of 71:6%.
v4 Reference to our updated “DeepLab” system (Chen et al., 2016) with much improved results.
REFERENCES
Adams, A., Baek, J., and Davis, M. A. Fast high-dimensional ﬁltering using the permutohedral
lattice. In Computer Graphics Forum , 2010.


key: chroma:document
string_value:
lattice. In Computer Graphics Forum , 2010.
Arbel ´aez, P., Pont-Tuset, J., Barron, J. T., Marques, F., and Malik, J. Multiscale combinatorial
grouping. In CVPR , 2014.
11
Published as a conference paper at ICLR 2015
Bell, S., Upchurch, P., Snavely, N., and Bala, K. Material recognition in the wild with the materials
in context database. arXiv:1412.0623 , 2014.
Carreira, J. and Sminchisescu, C. Cpmc: Automatic object segmentation using constrained parametric min-cuts. PAMI , 2012.
Carreira, J., Caseiro, R., Batista, J., and Sminchisescu, C. Semantic segmentation with second-order
pooling. In ECCV , 2012.
Chen, L.-C., Papandreou, G., and Yuille, A. Learning a dictionary of shape epitomes with applications to image labeling. In ICCV , 2013.
Chen, L.-C., Schwing, A., Yuille, A., and Urtasun, R. Learning deep structured models.
arXiv:1407.2538 , 2014.


key: chroma:document
string_value:
arXiv:1407.2538 , 2014.
Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., and Yuille, A. L. Deeplab: Semantic
image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs.
arXiv:1606.00915 , 2016.
Chen, X. and Yuille, A. L. Articulated pose estimation by a graphical model with image dependent
pairwise relations. In NIPS , 2014.
Cogswell, M., Lin, X., Purushwalkam, S., and Batra, D. Combining the best of graphical models
and convnets for semantic segmentation. arXiv:1412.4313 , 2014.
Dai, J., He, K., and Sun, J. Convolutional feature masking for joint object and stuff segmentation.
arXiv:1412.1283 , 2014.
Delong, A., Osokin, A., Isack, H. N., and Boykov, Y . Fast approximate energy minimization with
label costs. IJCV , 2012.
Eigen, D. and Fergus, R. Predicting depth, surface normals and semantic labels with a common


key: chroma:document
string_value:
label costs. IJCV , 2012.
Eigen, D. and Fergus, R. Predicting depth, surface normals and semantic labels with a common
multi-scale convolutional architecture. arXiv:1411.4734 , 2014.
Everingham, M., Eslami, S. M. A., Gool, L. V ., Williams, C. K. I., Winn, J., and Zisserma, A. The
pascal visual object classes challenge a retrospective. IJCV , 2014.
Farabet, C., Couprie, C., Najman, L., and LeCun, Y . Learning hierarchical features for scene labeling.PAMI , 2013.
Geiger, D. and Girosi, F. Parallel and deterministic algorithms from mrfs: Surface reconstruction.
PAMI , 13(5):401–412, 1991.
Geiger, D. and Yuille, A. A common framework for image segmentation. IJCV , 6(3):227–243,
1991.
Girshick, R., Donahue, J., Darrell, T., and Malik, J. Rich feature hierarchies for accurate object
detection and semantic segmentation. In CVPR , 2014.


key: chroma:document
string_value:
detection and semantic segmentation. In CVPR , 2014.
Giusti, A., Ciresan, D., Masci, J., Gambardella, L., and Schmidhuber, J. Fast image scanning with
deep max-pooling convolutional neural networks. In ICIP , 2013.
Gonfaus, J. M., Boix, X., Van de Weijer, J., Bagdanov, A. D., Serrat, J., and Gonzalez, J. Harmony
potentials for joint classiﬁcation and segmentation. In CVPR , 2010.
Hariharan, B., Arbel ´aez, P., Bourdev, L., Maji, S., and Malik, J. Semantic contours from inverse
detectors. In ICCV , 2011.
Hariharan, B., Arbel ´aez, P., Girshick, R., and Malik, J. Hypercolumns for object segmentation and
ﬁne-grained localization. arXiv:1411.5752 , 2014a.
Hariharan, B., Arbel ´aez, P., Girshick, R., and Malik, J. Simultaneous detection and segmentation.
InECCV , 2014b.
12
Published as a conference paper at ICLR 2015


key: chroma:document
string_value:
InECCV , 2014b.
12
Published as a conference paper at ICLR 2015
He, X., Zemel, R. S., and Carreira-Perpindn, M. Multiscale conditional random ﬁelds for image
labeling. In CVPR , 2004.
Jia, Y ., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., and Darrell,
T. Caffe: Convolutional architecture for fast feature embedding. arXiv:1408.5093 , 2014.
Kohli, P., Ladicky, L., and Torr, P. H. Robust higher order potentials for enforcing label consistency.
IJCV , 2009.
Kokkinos, I., Deriche, R., Faugeras, O., and Maragos, P. Computational analysis and learning for a
biologically motivated model of boundary detection. Neurocomputing , 71(10):1798–1812, 2008.
Kr¨ahenb ¨uhl, P. and Koltun, V . Efﬁcient inference in fully connected crfs with gaussian edge potentials. In NIPS , 2011.


key: chroma:document
string_value:
Kr¨ahenb ¨uhl, P. and Koltun, V . Parameter learning and convergent inference for dense random ﬁelds.
InICML , 2013.
Krizhevsky, A., Sutskever, I., and Hinton, G. E. Imagenet classiﬁcation with deep convolutional
neural networks. In NIPS , 2013.
Ladicky, L., Russell, C., Kohli, P., and Torr, P. H. Associative hierarchical crfs for object class image
segmentation. In ICCV , 2009.
LeCun, Y ., Bottou, L., Bengio, Y ., and Haffner, P. Gradient-based learning applied to document
recognition. In Proc. IEEE , 1998.
Lempitsky, V ., Vedaldi, A., and Zisserman, A. Pylon model for semantic segmentation. In NIPS ,
2011.
Long, J., Shelhamer, E., and Darrell, T. Fully convolutional networks for semantic segmentation.
arXiv:1411.4038 , 2014.
Lucchi, A., Li, Y ., Boix, X., Smith, K., and Fua, P. Are spatial and global constraints really necessary


key: chroma:document
string_value:
Lucchi, A., Li, Y ., Boix, X., Smith, K., and Fua, P. Are spatial and global constraints really necessary
for segmentation? In ICCV , 2011.
Mallat, S. A Wavelet Tour of Signal Processing . Acad. Press, 2 edition, 1999.
Mostajabi, M., Yadollahpour, P., and Shakhnarovich, G. Feedforward semantic segmentation with
zoom-out features. arXiv:1412.0774 , 2014.
Papandreou, G., Kokkinos, I., and Savalle, P.-A. Untangling local and global deformations in deep
convolutional networks for image classiﬁcation and sliding window detection. arXiv:1412.0296 ,
2014.
Papandreou, G., Chen, L.-C., Murphy, K., and Yuille, A. L. Weakly- and semi-supervised learning
of a DCNN for semantic image segmentation. arXiv:1502.02734 , 2015.
Rother, C., Kolmogorov, V ., and Blake, A. Grabcut: Interactive foreground extraction using iterated
graph cuts. In SIGGRAPH , 2004.


key: chroma:document
string_value:
Rother, C., Kolmogorov, V ., and Blake, A. Grabcut: Interactive foreground extraction using iterated
graph cuts. In SIGGRAPH , 2004.
Sermanet, P., Eigen, D., Zhang, X., Mathieu, M., Fergus, R., and LeCun, Y . Overfeat: Integrated
recognition, localization and detection using convolutional networks. arXiv:1312.6229 , 2013.
Shotton, J., Winn, J., Rother, C., and Criminisi, A. Textonboost for image understanding: Multiclass object recognition and segmentation by jointly modeling texture, layout, and context. IJCV ,
2009.
Simonyan, K. and Zisserman, A. Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556 , 2014.
Szegedy, C., Liu, W., Jia, Y ., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V ., and
Rabinovich, A. Going deeper with convolutions. arXiv:1409.4842 , 2014.
13
Published as a conference paper at ICLR 2015


key: chroma:document
string_value:
Rabinovich, A. Going deeper with convolutions. arXiv:1409.4842 , 2014.
13
Published as a conference paper at ICLR 2015
Tompson, J., Jain, A., LeCun, Y ., and Bregler, C. Joint Training of a Convolutional Network and a
Graphical Model for Human Pose Estimation. In NIPS , 2014.
Uijlings, J., van de Sande, K., Gevers, T., and Smeulders, A. Selective search for object recognition.
IJCV , 2013.
Wang, P., Shen, X., Lin, Z., Cohen, S., Price, B., and Yuille, A. Towards uniﬁed depth and semantic
prediction from a single image. In CVPR , 2015.
Yadollahpour, P., Batra, D., and Shakhnarovich, G. Discriminative re-ranking of diverse segmentations. In CVPR , 2013.
Zeiler, M. D. and Fergus, R. Visualizing and understanding convolutional networks. In ECCV , 2014.
Zhang, N., Donahue, J., Girshick, R., and Darrell, T. Part-based r-cnns for ﬁne-grained category
detection. In ECCV , 2014.


key: chroma:document
string_value:
detection. In ECCV , 2014.
Zheng, S., Jayasumana, S., Romera-Paredes, B., Vineet, V ., Su, Z., Du, D., Huang, C., and Torr, P.
Conditional random ﬁelds as recurrent neural networks. arXiv:1502.03240 , 2015.
14


key: chroma:document
string_value:
arXiv:1502.03167v3  [cs.LG]  2 Mar 2015BatchNormalization: AcceleratingDeepNetworkTrainingb y
ReducingInternalCovariateShift
Sergey Ioffe
GoogleInc., sioffe@google.comChristianSzegedy
GoogleInc., szegedy@google.com
Abstract
TrainingDeepNeuralNetworksiscomplicatedbythefact
that the distributionofeach layer’sinputschangesduring
training, as the parametersof the previouslayers change.
This slows down the training by requiringlower learning
ratesandcarefulparameterinitialization,andmakesitno toriously hard to train models with saturating nonlinearities. We refer to this phenomenon as internal covariate
shift, and address the problem by normalizing layer inputs. Ourmethoddrawsitsstrengthfrommakingnormalizationapartofthemodelarchitectureandperformingthe
normalization for each training mini-batch . Batch Normalizationallowsustousemuchhigherlearningratesand
be less careful about initialization. It also acts as a regularizer, in some cases eliminating the need for Dropout.
Applied to a state-of-the-art image classiﬁcation model,
Batch Normalizationachievesthe same accuracy with 14
times fewer training steps, and beats the original model


key: chroma:document
string_value:
Batch Normalizationachievesthe same accuracy with 14
times fewer training steps, and beats the original model
by a signiﬁcant margin. Using an ensemble of batchnormalizednetworks,weimproveuponthebestpublished
result on ImageNet classiﬁcation: reaching 4.9% top-5
validation error (and 4.8% test error), exceeding the accuracyofhumanraters.
1 Introduction
Deep learning has dramatically advanced the state of the
art in vision, speech, and many other areas. Stochastic gradient descent (SGD) has proved to be an effective way of training deep networks, and SGD variants
such as momentum (Sutskeveret al., 2013) and Adagrad
(Duchiet al.,2011)havebeenusedtoachievestate ofthe
art performance. SGD optimizes the parameters Θof the
network,soasto minimizetheloss
Θ = argmin
Θ1
NN/summationdisplay
i=1ℓ(xi,Θ)


key: chroma:document
string_value:
network,soasto minimizetheloss
Θ = argmin
Θ1
NN/summationdisplay
i=1ℓ(xi,Θ)
wherex1...Nisthetrainingdataset. With SGD,thetrainingproceedsinsteps,andateachstepweconsidera minibatchx1...mofsizem. The mini-batchis usedtoapproximate the gradient of the loss functionwith respect to the
parameters,bycomputing
1
m∂ℓ(xi,Θ)
∂Θ.Usingmini-batchesofexamples,asopposedtooneexampleatatime,ishelpfulinseveralways. First,thegradient
ofthelossoveramini-batchisanestimateofthegradient
overthetrainingset, whose qualityimprovesas thebatch
size increases. Second, computation over a batch can be
much more efﬁcient than mcomputations for individual
examples, due to the parallelism afforded by the modern
computingplatforms.
While stochastic gradient is simple and effective, it
requires careful tuning of the model hyper-parameters,
speciﬁcallythelearningrateusedinoptimization,aswell


key: chroma:document
string_value:
requires careful tuning of the model hyper-parameters,
speciﬁcallythelearningrateusedinoptimization,aswell
as the initial values for the model parameters. The trainingiscomplicatedbythefactthattheinputstoeachlayer
areaffectedbytheparametersofallprecedinglayers–so
that small changes to the network parameters amplify as
thenetworkbecomesdeeper.
The change in the distributions of layers’ inputs
presents a problem because the layers need to continuously adapt to the new distribution. When the input distributiontoalearningsystemchanges,itissaidtoexperiencecovariateshift (Shimodaira, 2000). This is typically
handled via domain adaptation (Jiang, 2008). However,
the notion of covariate shift can be extended beyond the
learningsystemasawhole,toapplytoitsparts,suchasa
sub-networkora layer. Considera networkcomputing
ℓ=F2(F1(u,Θ1),Θ2)
whereF1andF2are arbitrary transformations, and the
parameters Θ1,Θ2are to be learned so as to minimize
the lossℓ. Learning Θ2can be viewed as if the inputs
x =F1(u,Θ1)arefedintothesub-network


key: chroma:document
string_value:
the lossℓ. Learning Θ2can be viewed as if the inputs
x =F1(u,Θ1)arefedintothesub-network
ℓ=F2(x,Θ2).
Forexample,agradientdescentstep
Θ2←Θ2−α
mm/summationdisplay
i=1∂F2(xi,Θ2)
∂Θ2
(forbatchsize mandlearningrate α)isexactlyequivalent
to that for a stand-alone network F2with input x. Therefore, the input distribution properties that make training
more efﬁcient – such as having the same distribution between the training and test data – apply to training the
sub-network as well. As such it is advantageous for the
distributionof xtoremainﬁxedovertime. Then, Θ2does
1
not have to readjust to compensate for the change in the
distributionof x.
Fixed distribution of inputs to a sub-network would
havepositiveconsequencesforthelayers outsidethesubnetwork,as well. Consider a layer with a sigmoid activation function z =g(Wu+b)whereuis the layer input,
the weight matrix Wand bias vector bare the layer parameters to be learned, and g(x) =1


key: chroma:document
string_value:
the weight matrix Wand bias vector bare the layer parameters to be learned, and g(x) =1
1+exp(−x). As|x|
increases, g′(x)tends to zero. This means that for all dimensionsof x =Wu+bexceptthosewithsmallabsolute
values,thegradientﬂowingdownto uwillvanishandthe
model will train slowly. However, since xis affected by
W,band the parameters of all the layers below, changes
tothoseparametersduringtrainingwilllikelymovemany
dimensions of xinto the saturated regime of the nonlinearity and slow down the convergence. This effect is
ampliﬁed as the network depth increases. In practice,
the saturation problem and the resulting vanishing gradientsareusuallyaddressedbyusingRectiﬁedLinearUnits
(Nair&Hinton, 2010) ReLU(x) = max( x,0), careful
initialization (Bengio&Glorot, 2010; Saxeet al., 2013),
and small learning rates. If, however, we could ensure
that the distribution of nonlinearity inputs remains more
stable as the network trains, then the optimizer would be
less likely to get stuck in the saturated regime, and the
trainingwouldaccelerate.
We refer to the change in the distributions of internal


key: chroma:document
string_value:
less likely to get stuck in the saturated regime, and the
trainingwouldaccelerate.
We refer to the change in the distributions of internal
nodes of a deep network, in the course of training, as Internal Covariate Shift . Eliminating it offers a promise of
faster training. We propose a new mechanism, which we
callBatch Normalization , that takes a step towards reducing internal covariate shift, and in doing so dramatically accelerates the training of deep neural nets. It accomplishes this via a normalization step that ﬁxes the
meansandvariancesoflayerinputs. BatchNormalization
also has a beneﬁcial effect on the gradient ﬂow through
the network, by reducing the dependence of gradients
on the scale of the parameters or of their initial values.
This allows us to use much higher learning rates without the risk of divergence. Furthermore, batch normalization regularizes the model and reduces the need for
Dropout(Srivastavaet al., 2014). Finally, Batch Normalization makes it possible to use saturating nonlinearities
by preventingthe network from getting stuck in the saturatedmodes.
In Sec. 4.2, we apply Batch Normalization to the bestperforming ImageNet classiﬁcation network, and show
that we can match its performance using only 7% of the


key: chroma:document
string_value:
that we can match its performance using only 7% of the
training steps, and can further exceed its accuracy by a
substantial margin. Using an ensemble of such networks
trained with Batch Normalization, we achieve the top-5
error rate that improves upon the best known results on
ImageNetclassiﬁcation.2 Towards Reducing Internal
CovariateShift
We deﬁne Internal Covariate Shift as the change in the
distribution of network activations due to the change in
networkparametersduringtraining. Toimprovethetraining, we seek to reduce the internal covariate shift. By
ﬁxingthe distributionof the layer inputs xas the training
progresses,weexpecttoimprovethetrainingspeed. Ithas
been long known (LeCunetal., 1998b; Wiesler &Ney,
2011) that the network training convergesfaster if its inputsarewhitened–i.e.,linearlytransformedtohavezero
meansandunitvariances,anddecorrelated. Aseachlayer
observestheinputsproducedbythelayersbelow,itwould
be advantageousto achieve the same whiteningof the inputsof each layer. By whitening the inputsto each layer,


key: chroma:document
string_value:
be advantageousto achieve the same whiteningof the inputsof each layer. By whitening the inputsto each layer,
we would take a step towards achieving the ﬁxed distributions of inputs that would remove the ill effects of the
internalcovariateshift.
We couldconsiderwhiteningactivationsat everytraining step or at some interval, either by modifying the
network directly or by changing the parameters of the
optimization algorithm to depend on the network activation values (Wiesleret al., 2014; Raikoetal., 2012;
Poveyet al., 2014; Desjardins&Kavukcuoglu). However, if these modiﬁcations are interspersed with the optimization steps, then the gradient descent step may attempt to update the parameters in a way that requires
the normalization to be updated, which reduces the effect of the gradient step. For example, consider a layer
with the input uthat addsthe learned bias b, and normalizes the result by subtracting the mean of the activation
computed over the training data: /hatwidex=x−E[x]where
x=u+b,X={x1...N}is the set of values of xover
the training set, and E [x] =1
N/summationtextN
i=1xi. If a gradient


key: chroma:document
string_value:
the training set, and E [x] =1
N/summationtextN
i=1xi. If a gradient
descent step ignores the dependence of E [x]onb, then it
will update b←b+ ∆b, where∆b∝−∂ℓ/∂/hatwidex. Then
u+ (b+ ∆b)−E[u+ (b+ ∆b)] =u+b−E[u+b].
Thus, the combination of the update to band subsequent
change in normalization led to no change in the output
of the layer nor, consequently, the loss. As the training
continues, bwill grow indeﬁnitely while the loss remains
ﬁxed. Thisproblemcangetworseifthenormalizationnot
only centers but also scales the activations. We have observed this empirically in initial experiments, where the
model blows up when the normalization parameters are
computedoutsidethe gradientdescentstep.
The issue with the above approach is that the gradient
descent optimization does not take into account the fact
that the normalization takes place. To address this issue,
we would like to ensure that, for any parameter values,
the network alwaysproducesactivationswith the desired
distribution. Doing so would allow the gradient of the


key: chroma:document
string_value:
we would like to ensure that, for any parameter values,
the network alwaysproducesactivationswith the desired
distribution. Doing so would allow the gradient of the
loss with respect to the model parameters to account for
the normalization, and for its dependence on the model
parameters Θ. Let again xbe a layer input, treated as a
2
vector, andXbe the set of these inputs over the training
dataset. Thenormalizationcanthenbewrittenasatransformation
/hatwidex =Norm(x,X)
which depends not only on the given training example x
but on all examples X– each of which depends on Θif
xis generatedby anotherlayer. For backpropagation,we
wouldneedtocomputetheJacobians
∂Norm(x,X)
∂xand∂Norm(x,X)
∂X;
ignoring the latter term would lead to the explosion describedabove. Withinthisframework,whiteningthelayer
inputs is expensive, as it requires computing the covariance matrix Cov [x] =Ex∈X[xxT]−E[x]E[x]Tand its
inverse square root, to produce the whitened activations
Cov[x]−1/2(x−E[x]), as well as the derivatives of these


key: chroma:document
string_value:
inverse square root, to produce the whitened activations
Cov[x]−1/2(x−E[x]), as well as the derivatives of these
transformsforbackpropagation.Thismotivatesustoseek
an alternative that performs input normalization in a way
that is differentiable and does not require the analysis of
theentiretrainingset aftereveryparameterupdate.
Some of the previous approaches (e.g.
(Lyu&Simoncelli, 2008)) use statistics computed
over a single training example, or, in the case of image
networks, over differentfeature maps at a given location.
However, this changes the representation ability of a
network by discarding the absolute scale of activations.
We want to a preservethe informationin the network,by
normalizing the activations in a training example relative
tothe statisticsoftheentiretrainingdata.
3 Normalization via Mini-Batch
Statistics
Since the full whitening of each layer’s inputs is costly
and not everywhere differentiable, we make two necessary simpliﬁcations. The ﬁrst is that instead of whitening
the features in layer inputs and outputs jointly, we will
normalizeeachscalarfeatureindependently,bymakingit
have the mean of zero and the variance of 1. For a layer


key: chroma:document
string_value:
normalizeeachscalarfeatureindependently,bymakingit
have the mean of zero and the variance of 1. For a layer
withd-dimensionalinput x = (x(1)...x(d)),wewillnormalizeeachdimension
/hatwidex(k)=x(k)−E[x(k)]/radicalbig
Var[x(k)]
wheretheexpectationandvariancearecomputedoverthe
trainingdataset. Asshownin(LeCunetal.,1998b),such
normalizationspeedsupconvergence,evenwhenthefeaturesarenotdecorrelated.
Notethatsimplynormalizingeachinputofalayermay
change what the layer can represent. For instance, normalizing the inputsof a sigmoid wouldconstrain them to
the linear regime of the nonlinearity. To address this, we
makesurethat thetransformationinsertedin thenetwork
can represent the identity transform . To accomplish this,weintroduce,foreachactivation x(k),apairofparameters
γ(k),β(k),whichscale andshift thenormalizedvalue:
y(k)=γ(k)/hatwidex(k)+β(k).
These parameters are learned along with the original
model parameters, and restore the representation power


key: chroma:document
string_value:
y(k)=γ(k)/hatwidex(k)+β(k).
These parameters are learned along with the original
model parameters, and restore the representation power
ofthenetwork. Indeed,bysetting γ(k)=/radicalbig
Var[x(k)]and
β(k)=E[x(k)], we couldrecoverthe originalactivations,
ifthatwerethe optimalthingto do.
Inthebatchsettingwhereeachtrainingstepisbasedon
theentire trainingset, we woulduse the wholeset to normalize activations. However,this is impracticalwhen using stochastic optimization. Therefore, we make the secondsimpliﬁcation: since we use mini-batchesin stochastic gradient training, each mini-batch produces estimates
ofthemeanandvariance ofeachactivation. Thisway,the
statistics used for normalization can fully participate in
the gradient backpropagation. Note that the use of minibatchesis enabledbycomputationof per-dimensionvariances rather than joint covariances; in the joint case, regularizationwouldbe requiredsince the mini-batchsize is
likely to be smaller than the number of activations being
whitened,resultinginsingularcovariancematrices.


key: chroma:document
string_value:
likely to be smaller than the number of activations being
whitened,resultinginsingularcovariancematrices.
Consider a mini-batch Bof sizem. Since the normalization is applied to each activation independently, let us
focusonaparticularactivation x(k)andomitkforclarity.
We havemvaluesofthisactivationinthemini-batch,
B={x1...m}.
Letthenormalizedvaluesbe /hatwidex1...m,andtheirlineartransformationsbe y1...m. We referto thetransform
BNγ,β:x1...m→y1...m
as theBatch Normalizing Transform . We present the BN
TransforminAlgorithm1. Inthealgorithm, ǫisaconstant
addedtothemini-batchvariancefornumericalstability.
Input:Valuesof xovera mini-batch: B={x1...m};
Parametersto belearned: γ,β
Output:{yi=BNγ,β(xi)}
µB←1
mm/summationdisplay
i=1xi // mini-batchmean
σ2
B←1
mm/summationdisplay
i=1(xi−µB)2// mini-batchvariance


key: chroma:document
string_value:
σ2
B←1
mm/summationdisplay
i=1(xi−µB)2// mini-batchvariance
/hatwidexi←xi−µB/radicalbig
σ2
B+ǫ// normalize
yi←γ/hatwidexi+β≡BNγ,β(xi) // scale andshift
Algorithm 1: Batch Normalizing Transform, applied to
activation xoveramini-batch.
TheBNtransformcanbeaddedtoanetworktomanipulate any activation. In the notation y=BNγ,β(x), we
3
indicate that the parameters γandβare to be learned,
but it should be noted that the BN transform does not
independently process the activation in each training example. Rather, BN γ,β(x)depends both on the training
exampleand the other examples in the mini-batch . The
scaled and shifted values yare passed to other network
layers. The normalized activations /hatwidexare internal to our
transformation, but their presence is crucial. The distributions of values of any /hatwidexhas the expected value of 0
and the variance of 1, as long as the elements of each
mini-batch are sampled from the same distribution, and


key: chroma:document
string_value:
and the variance of 1, as long as the elements of each
mini-batch are sampled from the same distribution, and
if we neglect ǫ. This can be seen by observing that/summationtextm
i=1/hatwidexi= 0and1
m/summationtextm
i=1/hatwidex2
i= 1, and taking expectations. Eachnormalizedactivation /hatwidex(k)canbeviewedas
an input to a sub-network composed of the linear transformy(k)=γ(k)/hatwidex(k)+β(k), followed by the other processing doneby the originalnetwork. Thesesub-network
inputs all have ﬁxed means and variances, and although
the jointdistributionofthese normalized /hatwidex(k)canchange
over the course of training, we expect that the introduction of normalized inputs accelerates the training of the
sub-networkand,consequently,thenetworkasawhole.
During training we need to backpropagate the gradient of loss ℓthrough this transformation,as well as compute the gradients with respect to the parameters of the
BN transform. We use chainrule,as follows(beforesimpliﬁcation):
∂ℓ
∂/hatwidexi=∂ℓ
∂yi·γ


key: chroma:document
string_value:
∂ℓ
∂/hatwidexi=∂ℓ
∂yi·γ
∂ℓ
∂σ2
B=/summationtextm
i=1∂ℓ
∂/hatwidexi·(xi−µB)·−1
2(σ2
B+ǫ)−3/2
∂ℓ
∂µB=/parenleftbigg/summationtextm
i=1∂ℓ
∂/hatwidexi·−1√
σ2
B+ǫ/parenrightbigg
+∂ℓ
∂σ2
B·/summationtextm
i=1−2(xi−µB)
m
∂ℓ
∂xi=∂ℓ
∂/hatwidexi·1√
σ2
B+ǫ+∂ℓ
∂σ2
B·2(xi−µB)
m+∂ℓ
∂µB·1
m
∂ℓ
∂γ=/summationtextm
i=1∂ℓ
∂yi·/hatwidexi
∂ℓ


key: chroma:document
string_value:
∂ℓ
∂γ=/summationtextm
i=1∂ℓ
∂yi·/hatwidexi
∂ℓ
∂β=/summationtextm
i=1∂ℓ
∂yi
Thus,BNtransformisadifferentiabletransformationthat
introduces normalized activations into the network. This
ensures that as the model is training, layers can continue
learningoninputdistributionsthatexhibitlessinternal covariate shift, thus accelerating the training. Furthermor e,
the learned afﬁne transform applied to these normalized
activationsallowstheBNtransformtorepresenttheidentity transformationandpreservesthenetworkcapacity.
3.1 Training and Inference with BatchNormalizedNetworks
ToBatch-Normalize anetwork,wespecifyasubsetofactivations and insert the BN transform for each of them,
according to Alg. 1. Any layer that previously received
xas the input, now receives BN (x). A model employing
Batch Normalization can be trained using batch gradient
descent,orStochasticGradientDescentwithamini-batch
sizem >1, or with any of its variants such as Adagrad(Duchiet al.,2011). Thenormalizationofactivationsthat


key: chroma:document
string_value:
sizem >1, or with any of its variants such as Adagrad(Duchiet al.,2011). Thenormalizationofactivationsthat
dependsonthemini-batchallowsefﬁcienttraining,but is
neithernecessarynordesirableduringinference;wewant
the output to depend only on the input, deterministically.
For this, once the network has been trained, we use the
normalization
/hatwidex=x−E[x]/radicalbig
Var[x]+ǫ
using the population, rather than mini-batch, statistics.
Neglecting ǫ, these normalized activations have the same
mean0 and variance1 as duringtraining. We use the unbiased variance estimate Var [x] =m
m−1·EB[σ2
B], where
theexpectationisovertrainingmini-batchesofsize mand
σ2
Baretheirsamplevariances. Usingmovingaveragesinstead, we can track the accuracy of a model as it trains.
Sincethemeansandvariancesareﬁxedduringinference,
the normalization is simply a linear transform applied to
eachactivation. Itmayfurtherbecomposedwiththescaling byγand shift by β, to yield a single linear transform


key: chroma:document
string_value:
the normalization is simply a linear transform applied to
eachactivation. Itmayfurtherbecomposedwiththescaling byγand shift by β, to yield a single linear transform
that replacesBN (x). Algorithm 2 summarizesthe procedurefortrainingbatch-normalizednetworks.
Input:NetworkNwith trainableparameters Θ;
subsetofactivations {x(k)}K
k=1
Output: Batch-normalizednetworkforinference, Ninf
BN
1:Ntr
BN←N// TrainingBN network
2:fork= 1...Kdo
3:Add transformation y(k)=BNγ(k),β(k)(x(k))to
Ntr
BN(Alg.1)
4:Modify each layer in Ntr
BNwith input x(k)to take
y(k)instead
5:end for
6:TrainNtr
BNto optimize the parameters Θ∪
{γ(k),β(k)}K
k=1
7:Ninf
BN←Ntr
BN// InferenceBN networkwithfrozen
// parameters
8:fork= 1...Kdo
9:// Forclarity, x≡x(k),γ≡γ(k),µB≡µ(k)
B, etc.


key: chroma:document
string_value:
9:// Forclarity, x≡x(k),γ≡γ(k),µB≡µ(k)
B, etc.
10:Process multiple training mini-batches B, each of
sizem,andaverageoverthem:
E[x]←EB[µB]
Var[x]←m
m−1EB[σ2
B]
11:InNinf
BN, replace the transform y=BNγ,β(x)with
y=γ√
Var[x]+ǫ·x+/parenleftbig
β−γE[x]√
Var[x]+ǫ/parenrightbig
12:end for
Algorithm2: Traininga Batch-NormalizedNetwork
3.2 Batch-Normalized Convolutional Networks
Batch Normalization can be applied to any set of activations in the network. Here, we focus on transforms
4
that consist of an afﬁne transformation followed by an
element-wisenonlinearity:
z =g(Wu+b)
whereWandbare learned parametersof the model, and
g(·)isthenonlinearitysuchassigmoidorReLU.Thisformulation covers both fully-connected and convolutional
layers. We add the BN transform immediately before the


key: chroma:document
string_value:
layers. We add the BN transform immediately before the
nonlinearity,bynormalizing x =Wu+b. Wecouldhave
also normalized the layer inputs u, but since uis likely
the output of another nonlinearity, the shape of its distributionislikelytochangeduringtraining,andconstrainin g
its ﬁrst and second moments would not eliminate the covariate shift. In contrast, Wu + bis more likely to have
a symmetric,non-sparsedistribution,that is “moreGaussian”(Hyv¨ arinen&Oja,2000);normalizingitislikelyto
produceactivationswithastable distribution.
Notethat,sincewenormalize Wu+b,thebiasbcanbe
ignoredsinceitseffectwillbecanceledbythesubsequent
meansubtraction(theroleofthebiasissubsumedby βin
Alg.1). Thus, z =g(Wu+b)is replacedwith
z =g(BN(Wu))
where the BN transformis applied independentlyto each
dimension of x =Wu, with a separate pair of learned
parameters γ(k),β(k)perdimension.
Forconvolutionallayers,we additionallywant the normalization to obey the convolutional property – so that


key: chroma:document
string_value:
parameters γ(k),β(k)perdimension.
Forconvolutionallayers,we additionallywant the normalization to obey the convolutional property – so that
different elements of the same feature map, at different
locations, are normalized in the same way. To achieve
this, we jointly normalize all the activations in a minibatch, overall locations. In Alg. 1, we let Bbe the set of
all values in a feature map across both the elements of a
mini-batch and spatial locations – so for a mini-batch of
sizemand feature maps of size p×q, we use the effective mini-batch of size m′=|B|=m·pq. We learn a
pair of parameters γ(k)andβ(k)per feature map, rather
than per activation. Alg. 2 is modiﬁed similarly, so that
duringinferencetheBNtransformappliesthesamelinear
transformationtoeachactivationina givenfeaturemap.
3.3 Batch Normalization enables higher
learning rates
In traditional deep networks, too-high learning rate may
result in the gradients that explode or vanish, as well as
getting stuck in poor local minima. Batch Normalization helps address these issues. By normalizing activations throughout the network, it prevents small changes


key: chroma:document
string_value:
getting stuck in poor local minima. Batch Normalization helps address these issues. By normalizing activations throughout the network, it prevents small changes
to the parameters from amplifying into larger and suboptimal changes in activations in gradients; for instance, it
prevents the training from getting stuck in the saturated
regimesofnonlinearities.
BatchNormalizationalsomakestrainingmoreresilient
totheparameterscale. Normally,largelearningratesmay
increasethescaleoflayerparameters,whichthenamplifythegradientduringbackpropagationandleadtothemodel
explosion. However, with Batch Normalization, backpropagation through a layer is unaffected by the scale of
itsparameters. Indeed,fora scalar a,
BN(Wu) =BN((aW)u)
andwe canshowthat
∂BN((aW)u)
∂u=∂BN(Wu)
∂u
∂BN((aW)u)
∂(aW)=1
a·∂BN(Wu)
∂W
The scale does not affect the layer Jacobian nor, consequently, the gradient propagation. Moreover, larger
weights lead to smallergradients, and Batch Normalizationwill stabilize theparametergrowth.
We further conjecture that Batch Normalization may


key: chroma:document
string_value:
weights lead to smallergradients, and Batch Normalizationwill stabilize theparametergrowth.
We further conjecture that Batch Normalization may
leadthelayerJacobianstohavesingularvaluescloseto1,
which is known to be beneﬁcial for training (Saxeet al.,
2013). Consider two consecutive layers with normalized
inputs, and the transformation between these normalized
vectors:/hatwidez =F(/hatwidex). Ifweassumethat /hatwidexand/hatwidezareGaussian
anduncorrelated,andthat F(/hatwidex)≈J/hatwidexisalineartransformationforthe givenmodelparameters,thenboth /hatwidexand/hatwidez
have unit covariances, and I=Cov[/hatwidez] =JCov[/hatwidex]JT=
JJT. Thus,JJT=I, and so all singular values of J
are equal to 1, which preserves the gradient magnitudes
during backpropagation. In reality, the transformation is
notlinear,andthenormalizedvaluesarenotguaranteedto
be Gaussian nor independent, but we nevertheless expect
Batch Normalization to help make gradient propagation
better behaved. The precise effect of Batch Normalization on gradient propagation remains an area of further
study.


key: chroma:document
string_value:
Batch Normalization to help make gradient propagation
better behaved. The precise effect of Batch Normalization on gradient propagation remains an area of further
study.
3.4 Batch Normalization regularizes the
model
When training with Batch Normalization, a training example is seen in conjunction with other examples in the
mini-batch, and the training network no longer producing deterministic values for a given training example. In
our experiments,we foundthis effect to be advantageous
to the generalization of the network. Whereas Dropout
(Srivastavaet al., 2014) is typically used to reduce overﬁtting,inabatch-normalizednetworkwefoundthatitcan
beeitherremovedorreducedinstrength.
4 Experiments
4.1 Activationsovertime
To verify the effects of internal covariate shift on training, and the ability of Batch Normalization to combat it,
weconsideredtheproblemofpredictingthedigitclasson
theMNISTdataset(LeCunetal.,1998a). Weusedavery
simple network, with a 28x28binary image as input, and
5
10K20K30K40K50K0.70.80.91
  
Without BN
With BN
−202
−202
(a) (b)WithoutBN (c)With BN


key: chroma:document
string_value:
Without BN
With BN
−202
−202
(a) (b)WithoutBN (c)With BN
Figure 1: (a) The test accuracy of the MNIST network
trained with and without Batch Normalization, vs. the
number of training steps. Batch Normalization helps the
network train faster and achieve higher accuracy. (b,
c)The evolution of input distributions to a typical sigmoid,overthecourseoftraining,shownas {15,50,85}th
percentiles. Batch Normalization makes the distribution
morestableandreducestheinternalcovariateshift.
3fully-connectedhiddenlayerswith100activationseach.
Eachhiddenlayercomputes y =g(Wu+b)withsigmoid
nonlinearity, and the weights Winitialized to small random Gaussian values. The last hidden layer is followed
by a fully-connected layer with 10 activations (one per
class) and cross-entropyloss. We trained the network for
50000steps, with 60 examplespermini-batch. We added
BatchNormalizationtoeachhiddenlayerofthenetwork,
as in Sec. 3.1. We were interested in the comparison betweenthebaselineandbatch-normalizednetworks,rather
thanachievingthestateoftheartperformanceonMNIST


