# embedding_metadata

key: chroma:document
string_value:
thanachievingthestateoftheartperformanceonMNIST
(whichthe describedarchitecturedoesnot).
Figure 1(a) shows the fraction of correct predictions
by the two networks on held-out test data, as training
progresses. The batch-normalized network enjoys the
higher test accuracy. To investigate why, we studied inputs to the sigmoid, in the original network Nand batchnormalizednetwork Ntr
BN(Alg.2)overthecourseoftraining. InFig.1(b,c)weshow,foronetypicalactivationfrom
the last hidden layer of each network, how its distribution evolves. The distributions in the original network
change signiﬁcantly over time, both in their mean and
the variance, which complicates the training of the subsequent layers. In contrast, the distributions in the batch normalizednetworkaremuchmorestableastrainingprogresses,whichaidsthe training.
4.2 ImageNetclassiﬁcation
We applied Batch Normalization to a new variant of the
Inception network (Szegedyetal., 2014), trained on the
ImageNet classiﬁcation task (Russakovskyet al., 2014).
The network has a large number of convolutional and
pooling layers, with a softmax layer to predict the image
class, out of 1000 possibilities. Convolutional layers use


key: chroma:document
string_value:
The network has a large number of convolutional and
pooling layers, with a softmax layer to predict the image
class, out of 1000 possibilities. Convolutional layers use
ReLU asthenonlinearity. Themaindifferenceto thenetwork described in (Szegedyet al., 2014) is that the 5×5
convolutionallayersare replacedby two consecutivelayers of3×3convolutionswith up to 128ﬁlters. The network contains 13.6·106parameters, and, other than the
top softmax layer, has no fully-connected layers. Moredetails are given in the Appendix. We refer to this model
asInception intherestofthetext. Themodelwastrained
using a version of Stochastic Gradient Descent with momentum(Sutskeveretal.,2013),usingthemini-batchsize
of32. Thetrainingwasperformedusingalarge-scale,distributed architecture (similar to (Deanet al., 2012)). All
networksare evaluatedastrainingprogressesbycomputing the validation accuracy @1, i.e. the probability of
predicting the correct label out of 1000 possibilities, on
aheld-outset,usinga singlecropperimage.
Inourexperiments,weevaluatedseveralmodiﬁcations


key: chroma:document
string_value:
aheld-outset,usinga singlecropperimage.
Inourexperiments,weevaluatedseveralmodiﬁcations
ofInceptionwithBatchNormalization. Inallcases,Batch
Normalizationwasappliedtotheinputofeachnonlinearity, in a convolutional way, as described in section 3.2,
whilekeepingtherestofthearchitectureconstant.
4.2.1 AcceleratingBN Networks
SimplyaddingBatchNormalizationtoanetworkdoesnot
take full advantage of our method. To do so, we further
changed the network and its training parameters, as follows:
Increase learning rate. In a batch-normalized model,
we have been able to achieve a training speedup from
higherlearningrates,with noill sideeffects(Sec.3.3).
RemoveDropout. As describedin Sec. 3.4, Batch NormalizationfulﬁllssomeofthesamegoalsasDropout. Removing Dropout from Modiﬁed BN-Inception speeds up
training,withoutincreasingoverﬁtting.
Reduce the L2weight regularization. While in Inception anL2loss on the model parameters controls overﬁtting, in Modiﬁed BN-Inception the weight of this loss is


key: chroma:document
string_value:
reduced by a factor of 5. We ﬁnd that this improves the
accuracyontheheld-outvalidationdata.
Accelerate the learning rate decay. In training Inception, learning rate was decayed exponentially. Because
our network trains faster than Inception, we lower the
learningrate 6timesfaster.
Remove Local Response Normalization While Inception and other networks (Srivastavaet al., 2014) beneﬁt
from it, we found that with Batch Normalization it is not
necessary.
Shufﬂetrainingexamplesmorethoroughly. Weenabled
within-shardshufﬂingofthetrainingdata,whichprevents
thesameexamplesfromalwaysappearinginamini-batch
together. This led to about 1% improvements in the validation accuracy, which is consistent with the view of
Batch Normalization as a regularizer (Sec. 3.4): the randomization inherent in our method should be most beneﬁcialwhenitaffectsanexampledifferentlyeachtimeitis
seen.
Reduce the photometric distortions. Because batchnormalized networks train faster and observe each trainingexamplefewertimes,weletthetrainerfocusonmore
“real”imagesbydistortingthemless.
6


key: chroma:document
string_value:
“real”imagesbydistortingthemless.
6
5M 10M 15M 20M 25M 30M0.40.50.60.70.8
Inception
BN−Baseline
BN−x5
BN−x30
BN−x5−Sigmoid
Steps to match Inception
Figure 2: Single crop validation accuracy of Inception
and its batch-normalized variants, vs. the number of
trainingsteps.Model Stepsto72.2% Maxaccuracy
Inception 31.0·10672.2%
BN-Baseline 13.3·10672.7%
BN-x5 2.1·10673.0%
BN-x30 2.7·10674.8%
BN-x5-Sigmoid 69.8%
Figure 3: For Inception and the batch-normalized
variants, the number of training steps required to
reach the maximum accuracy of Inception(72.2%),
and the maximum accuracy achieved by the network.
4.2.2 Single-NetworkClassiﬁcation
We evaluated the following networks, all trained on the
LSVRC2012 training data, and tested on the validation
data:
Inception : the network described at the beginning of
Section4.2,trainedwiththeinitiallearningrateof0.001 5.


key: chroma:document
string_value:
data:
Inception : the network described at the beginning of
Section4.2,trainedwiththeinitiallearningrateof0.001 5.
BN-Baseline : Same as Inception with Batch Normalizationbeforeeachnonlinearity.
BN-x5: Inception with Batch Normalization and the
modiﬁcations in Sec. 4.2.1. The initial learning rate was
increased by a factor of 5, to 0.0075. The same learning
rateincreasewithoriginalInceptioncausedthemodelparameterstoreachmachineinﬁnity.
BN-x30: LikeBN-x5, but with the initial learning rate
0.045(30timesthatofInception).
BN-x5-Sigmoid : LikeBN-x5, but with sigmoid nonlinearityg(t) =1
1+exp(−x)instead of ReLU. We also attempted to train the original Inception with sigmoid, but
themodelremainedat theaccuracyequivalenttochance.
In Figure 2, we show the validation accuracy of the
networks, as a function of the number of training steps.
Inception reached the accuracy of 72.2% after 31·106
training steps. The Figure 3 shows, for each network,
the number of training steps required to reach the same


key: chroma:document
string_value:
Inception reached the accuracy of 72.2% after 31·106
training steps. The Figure 3 shows, for each network,
the number of training steps required to reach the same
72.2%accuracy,aswellasthemaximumvalidationaccuracy reached by the network and the number of steps to
reachit.
By onlyusingBatch Normalization( BN-Baseline ),we
matchtheaccuracyofInceptioninlessthanhalfthenumber of training steps. By applying the modiﬁcations in
Sec. 4.2.1, we signiﬁcantly increase the training speed of
the network. BN-x5needs 14 times fewer steps than Inception to reach the 72.2% accuracy. Interestingly, increasing the learning rate further ( BN-x30) causes the
model to train somewhat slowerinitially, but allows it to
reachahigherﬁnalaccuracy. Itreaches74.8%after 6·106
steps, i.e. 5 times fewer steps than required by Inception
toreach72.2%.
We also veriﬁed that the reduction in internal covariate shift allows deep networks with Batch Normalizationto be trained when sigmoid is used as the nonlinearity,


key: chroma:document
string_value:
We also veriﬁed that the reduction in internal covariate shift allows deep networks with Batch Normalizationto be trained when sigmoid is used as the nonlinearity,
despite the well-known difﬁculty of training such networks. Indeed, BN-x5-Sigmoid achieves the accuracy of
69.8%. WithoutBatchNormalization,Inceptionwithsigmoidneverachievesbetterthan 1/1000accuracy.
4.2.3 Ensemble Classiﬁcation
The current reported best results on the ImageNet Large
Scale Visual RecognitionCompetitionare reachedby the
Deep Image ensemble of traditional models (Wuet al.,
2015) and the ensemble model of (Heet al., 2015). The
latterreportsthetop-5errorof4.94%,asevaluatedbythe
ILSVRCserver. Herewereportatop-5validationerrorof
4.9%, and test error of 4.82% (according to the ILSVRC
server). This improvesupon the previousbest result, and
exceedstheestimatedaccuracyofhumanratersaccording
to(Russakovskyet al.,2014).
Forourensemble,weused6networks. Eachwasbased
onBN-x30,modiﬁedviasomeofthefollowing: increased


key: chroma:document
string_value:
Forourensemble,weused6networks. Eachwasbased
onBN-x30,modiﬁedviasomeofthefollowing: increased
initial weights in the convolutionallayers; using Dropout
(with the Dropout probability of 5% or 10%, vs. 40%
for the original Inception); and using non-convolutional,
per-activation Batch Normalization with last hidden layers of the model. Each network achieved its maximum
accuracyafter about 6·106training steps. The ensemble
prediction was based on the arithmetic average of class
probabilities predicted by the constituent networks. The
detailsofensembleandmulticropinferencearesimilarto
(Szegedyet al., 2014).
We demonstrate in Fig. 4 that batch normalization allowsusto set new state-of-the-artby a healthymarginon
theImageNetclassiﬁcationchallengebenchmarks.
5 Conclusion
We have presented a novel mechanism for dramatically
accelerating the training of deep networks. It is based on
the premise that covariate shift, which is known to complicate the trainingof machine learning systems, also ap7
Model Resolution Crops Models Top-1error Top-5error
GoogLeNetensemble 224 144 7 - 6.67%


key: chroma:document
string_value:
Model Resolution Crops Models Top-1error Top-5error
GoogLeNetensemble 224 144 7 - 6.67%
DeepImagelow-res 256 - 1 - 7.96%
DeepImagehigh-res 512 - 1 24.88 7.42%
DeepImageensemble variable - - - 5.98%
BN-Inceptionsinglecrop 224 1 1 25.2% 7.82%
BN-Inceptionmulticrop 224 144 1 21.99% 5.82%
BN-Inceptionensemble 224 144 6 20.1% 4.9%*
Figure 4: Batch-Normalized Inception comparison with previous stat e of the art on the provided validation set comprising50000images. *BN-Inceptionensemblehasreached4 .82%top-5erroronthe100000imagesofthetestsetof
theImageNetasreportedbythe test server.
plies to sub-networks and layers, and removing it from
internal activations of the network may aid in training.
Our proposed method draws its power from normalizing
activations, and from incorporating this normalization in
the network architecture itself. This ensures that the normalization is appropriately handled by any optimization
method that is being used to train the network. To enable stochastic optimization methods commonly used in
deep network training, we perform the normalization for


key: chroma:document
string_value:
method that is being used to train the network. To enable stochastic optimization methods commonly used in
deep network training, we perform the normalization for
eachmini-batch,andbackpropagatethegradientsthrough
the normalization parameters. Batch Normalization adds
only two extra parameters per activation, and in doing so
preserves the representation ability of the network. We
presentedanalgorithmforconstructing,training,andper forming inference with batch-normalized networks. The
resulting networks can be trained with saturating nonlinearities, are more tolerant to increased training rates, an d
oftendonotrequireDropoutforregularization.
Merely adding Batch Normalization to a state-of-theartimageclassiﬁcationmodelyieldsasubstantialspeedup
in training. By further increasing the learning rates, removing Dropout, and applying other modiﬁcations afforded by Batch Normalization, we reach the previous
state of the art with onlya small fractionof trainingsteps
–andthenbeatthestateoftheartinsingle-networkimage
classiﬁcation. Furthermore, by combining multiple models trained with Batch Normalization, we perform better
thanthebestknownsystemonImageNet,byasigniﬁcant
margin.


key: chroma:document
string_value:
thanthebestknownsystemonImageNet,byasigniﬁcant
margin.
Interestingly, our method bears similarity to the standardization layer of (G¨ ulc ¸ehre& Bengio, 2013), though
the two methodsstem from very differentgoals, and perform different tasks. The goal of Batch Normalization
is to achieve a stable distribution of activation values
throughout training, and in our experiments we apply it
before the nonlinearity since that is where matching the
ﬁrst and second moments is more likely to result in a
stable distribution. On the contrary,(G¨ ulc ¸ehre&Bengio ,
2013) apply the standardizationlayer to the outputof the
nonlinearity, which results in sparser activations. In our
large-scaleimage classiﬁcation experiments,we havenot
observedthenonlinearity inputstobesparse,neitherwith
nor without Batch Normalization. Other notable differ-entiating characteristics of Batch Normalization include
the learned scale and shift that allow the BN transform
to representidentity (the standardizationlayer did not re quirethissinceitwasfollowedbythelearnedlineartransform that, conceptually, absorbs the necessary scale and


key: chroma:document
string_value:
to representidentity (the standardizationlayer did not re quirethissinceitwasfollowedbythelearnedlineartransform that, conceptually, absorbs the necessary scale and
shift), handling of convolutional layers, deterministic i nferencethatdoesnotdependonthemini-batch,andbatchnormalizingeachconvolutionallayerin thenetwork.
In this work, we have not explored the full range of
possibilitiesthatBatchNormalizationpotentiallyenabl es.
Our future work includes applications of our method to
Recurrent Neural Networks (Pascanuet al., 2013), where
theinternalcovariateshiftandthevanishingorexploding
gradients may be especially severe, and which would allowustomorethoroughlytestthehypothesisthatnormalizationimprovesgradientpropagation(Sec.3.3). Weplan
toinvestigatewhetherBatchNormalizationcanhelpwith
domain adaptation, in its traditional sense – i.e. whether
the normalization performed by the network would allow it to more easily generalize to new data distributions,perhapswithjustarecomputationofthepopulation
meansandvariances(Alg.2). Finally,webelievethatfurthertheoreticalanalysisofthealgorithmwouldallowstil l
moreimprovementsandapplications.
References


key: chroma:document
string_value:
moreimprovementsandapplications.
References
Bengio, Yoshua and Glorot, Xavier. Understanding the
difﬁcultyoftrainingdeepfeedforwardneuralnetworks.
InProceedings of AISTATS 2010 , volume 9, pp. 249–
256,May2010.
Dean,Jeffrey,Corrado,GregS.,Monga,Rajat,Chen,Kai,
Devin,Matthieu,Le,QuocV., Mao,MarkZ.,Ranzato,
Marc’Aurelio,Senior,Andrew,Tucker,Paul,Yang,Ke,
and Ng, Andrew Y. Large scale distributed deep networks. In NIPS,2012.
Desjardins, Guillaume and Kavukcuoglu,Koray. Natural
neuralnetworks. (unpublished).
Duchi, John, Hazan, Elad, and Singer, Yoram. Adaptive
subgradientmethodsfor onlinelearning and stochastic
8
optimization. J.Mach.Learn.Res. ,12:2121–2159,July
2011. ISSN1532-4435.
G¨ ulc ¸ehre, C ¸aglar and Bengio, Yoshua. Knowledge matters: Importanceof prior informationfor optimization.


key: chroma:document
string_value:
G¨ ulc ¸ehre, C ¸aglar and Bengio, Yoshua. Knowledge matters: Importanceof prior informationfor optimization.
CoRR,abs/1301.4083,2013.
He, K., Zhang, X., Ren, S., and Sun, J. Delving Deep
into Rectiﬁers: Surpassing Human-Level Performance
on ImageNet Classiﬁcation. ArXiv e-prints , February
2015.
Hyv¨ arinen, A. and Oja, E. Independentcomponent analysis: Algorithms and applications. Neural Netw. , 13
(4-5):411–430,May2000.
Jiang, Jing. A literature survey on domain adaptation of
statistical classiﬁers, 2008.
LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P.
Gradient-based learning applied to document recognition.Proceedings of the IEEE , 86(11):2278–2324,
November1998a.
LeCun, Y., Bottou, L., Orr, G., and Muller, K. Efﬁcient
backprop. InOrr,G.andK.,Muller(eds.), NeuralNetworks: Tricks ofthetrade .Springer,1998b.


key: chroma:document
string_value:
backprop. InOrr,G.andK.,Muller(eds.), NeuralNetworks: Tricks ofthetrade .Springer,1998b.
Lyu, S and Simoncelli, E P. Nonlinear image representation using divisive normalization. In Proc. Computer
Vision and Pattern Recognition , pp. 1–8. IEEE Computer Society, Jun 23-28 2008. doi: 10.1109/CVPR.
2008.4587821.
Nair,VinodandHinton,GeoffreyE. Rectiﬁedlinearunits
improve restricted boltzmann machines. In ICML, pp.
807–814.Omnipress,2010.
Pascanu, Razvan, Mikolov, Tomas, and Bengio, Yoshua.
On the difﬁculty of training recurrent neural networks.
InProceedingsofthe30thInternationalConferenceon
MachineLearning,ICML 2013,Atlanta,GA,USA,1621June2013 ,pp.1310–1318,2013.
Povey, Daniel, Zhang, Xiaohui, and Khudanpur, Sanjeev. Parallel training of deep neural networks with
natural gradient and parameter averaging. CoRR,
abs/1410.7455,2014.


key: chroma:document
string_value:
natural gradient and parameter averaging. CoRR,
abs/1410.7455,2014.
Raiko, Tapani, Valpola, Harri, and LeCun, Yann. Deep
learning made easier by linear transformations in perceptrons. In International Conference on Artiﬁcial IntelligenceandStatistics(AISTATS) ,pp.924–932,2012.
Russakovsky,Olga,Deng,Jia,Su,Hao,Krause,Jonathan,
Satheesh, Sanjeev, Ma, Sean, Huang, Zhiheng, Karpathy, Andrej, Khosla, Aditya, Bernstein, Michael, Berg,
Alexander C., and Fei-Fei, Li. ImageNet Large Scale
Visual RecognitionChallenge,2014.Saxe, Andrew M., McClelland, James L., and Ganguli,
Surya. Exact solutions to the nonlinear dynamics
of learning in deep linear neural networks. CoRR,
abs/1312.6120,2013.
Shimodaira, Hidetoshi. Improving predictive inference
under covariate shift by weighting the log-likelihood
function. JournalofStatisticalPlanningandInference ,
90(2):227–244,October2000.


key: chroma:document
string_value:
under covariate shift by weighting the log-likelihood
function. JournalofStatisticalPlanningandInference ,
90(2):227–244,October2000.
Srivastava, Nitish, Hinton, Geoffrey, Krizhevsky, Alex,
Sutskever, Ilya, and Salakhutdinov, Ruslan. Dropout:
A simple way to preventneural networksfrom overﬁtting.J. Mach. Learn. Res. , 15(1):1929–1958, January
2014.
Sutskever, Ilya, Martens, James, Dahl, George E., and
Hinton, Geoffrey E. On the importance of initialization and momentum in deep learning. In ICML
(3), volume 28 of JMLR Proceedings , pp. 1139–1147.
JMLR.org,2013.
Szegedy, Christian, Liu, Wei, Jia, Yangqing, Sermanet,
Pierre, Reed, Scott, Anguelov, Dragomir, Erhan, Dumitru, Vanhoucke, Vincent, and Rabinovich, Andrew. Going deeper with convolutions. CoRR,
abs/1409.4842,2014.


key: chroma:document
string_value:
abs/1409.4842,2014.
Wiesler, Simon and Ney, Hermann. A convergenceanalysis of log-lineartraining. In Shawe-Taylor,J., Zemel,
R.S.,Bartlett,P.,Pereira,F.C.N.,andWeinberger,K.Q.
(eds.),AdvancesinNeuralInformationProcessingSystems24,pp.657–665,Granada,Spain,December2011.
Wiesler, Simon, Richard, Alexander, Schl¨ uter, Ralf, and
Ney, Hermann. Mean-normalized stochastic gradient
for large-scale deep learning. In IEEE International
Conference on Acoustics, Speech, and Signal Processing,pp.180–184,Florence,Italy,May2014.
Wu, Ren, Yan, Shengen, Shan, Yi, Dang, Qingqing, and
Sun,Gang. Deepimage: Scalingupimagerecognition,
2015.
Appendix
Variantofthe Inception Model Used
Figure 5 documents the changes that were performed
compared to the architecture with respect to the
GoogleNet archictecture. For the interpretation of this
table, please consult (Szegedyetal., 2014). The notable
architecture changes compared to the GoogLeNet model
include:


key: chroma:document
string_value:
table, please consult (Szegedyetal., 2014). The notable
architecture changes compared to the GoogLeNet model
include:
•The 5×5 convolutional layers are replaced by two
consecutive 3×3 convolutional layers. This increases the maximum depth of the network by 9
9
weight layers. Also it increases the number of parameters by 25% and the computational cost is increasedbyabout30%.
•The number 28×28 inception modules is increased
from2to 3.
•Inside the modules, sometimes average, sometimes
maximum-poolingis employed. This is indicated in
theentriescorrespondingtothepoolinglayersofthe
table.
•There are no across the board pooling layers between any two Inception modules, but stride-2 convolution/pooling layers are employed before the ﬁlterconcatenationin themodules3c,4e.
Our model employed separable convolution with depth
multiplier 8on the ﬁrst convolutionallayer. This reduces
thecomputationalcost while increasingthememoryconsumptionat trainingtime.
10
typepatch size/
strideoutput
sizedepth #1×1#3×3
reduce#3×3double#3×3
reducedouble
#3×3Pool+proj


key: chroma:document
string_value:
strideoutput
sizedepth #1×1#3×3
reduce#3×3double#3×3
reducedouble
#3×3Pool+proj
convolution* 7×7/2112×112×641
maxpool 3×3/256×56×64 0
convolution 3×3/156×56×192 1 64 192
maxpool 3×3/228×28×192 0
inception (3a) 28×28×256 364 64 64 64 96 avg+ 32
inception (3b) 28×28×320 364 64 96 64 96 avg+ 64
inception (3c) stride 2 28×28×576 3 0128 160 64 96max +pass through
inception (4a) 14×14×576 3224 64 96 96 128 avg+ 128
inception (4b) 14×14×576 3192 96 128 96 128 avg+ 128
inception (4c) 14×14×576 3160 128 160 128 160 avg+ 128
inception (4d) 14×14×576 396 128 192 160 192 avg+ 128
inception (4e) stride 2 14×14×1024 3 0128 192 192 256max +pass through
inception (5a) 7×7×1024 3352 192 320 160 224 avg+ 128


key: chroma:document
string_value:
inception (5a) 7×7×1024 3352 192 320 160 224 avg+ 128
inception (5b) 7×7×1024 3352 192 320 192 224 max+ 128
avgpool 7×7/11×1×1024 0
Figure5: Inceptionarchitecture
11


key: chroma:document
string_value:
1
Multiscale Combinatorial Grouping
for Image Segmentation and
Object Proposal Generation
Jordi Pont-Tuset*, Pablo Arbel ´aez*, Jonathan T. Barron, Member, IEEE,
Ferran Marques, Senior Member, IEEE, Jitendra Malik, Fellow, IEEE
Abstract —We propose a uniﬁed approach for bottom-up hierarchical image segmentation and object proposal generation for
recognition, called Multiscale Combinatorial Grouping (MCG). For this purpose, we ﬁrst develop a fast normalized cuts algorithm.
We then propose a high-performance hierarchical segmenter that makes effective use of multiscale information. Finally, we propose a
grouping strategy that combines our multiscale regions into highly-accurate object proposals by exploring efﬁciently their combinatorial
space. We also present Single-scale Combinatorial Grouping (SCG), a faster version of MCG that produces competitive proposals in
under ﬁve second per image. We conduct an extensive and comprehensive empirical validation on the BSDS500, SegVOC12, SBD,
and COCO datasets, showing that MCG produces state-of-the-art contours, hierarchical regions, and object proposals.
Index Terms —Image segmentation, object proposals, normalized cuts.
F
1 I NTRODUCTION


key: chroma:document
string_value:
Index Terms —Image segmentation, object proposals, normalized cuts.
F
1 I NTRODUCTION
TWOparadigms have shaped the ﬁeld of object recognition in the last decade. The ﬁrst one, popularized
by the Viola-Jones face detection algorithm [1], formulates object localization as window classiﬁcation. The basic scanning-window architecture, relying on histograms
of gradients and linear support vector machines, was
introduced by Dalal and Triggs [2] in the context of
pedestrian detection and is still at the core of seminal
object detectors on the PASCAL challenge such as Deformable Part Models [3].
The second paradigm relies on perceptual grouping to
provide a limited number of high-quality and categoryindependent object proposals, which can then be described with richer representations and used as input to
more sophisticated learning methods. Examples in this
family are [4], [5]. Recently, this approach has dominated
the PASCAL segmentation challenge [6], [7], [8], [9], improved object detection [10], ﬁne-grained categorization
[11] and proven competitive in large-scale classiﬁcation
[12].
Since the power of this second paradigm is critically
dependent on the accuracy and the number of object


key: chroma:document
string_value:
[11] and proven competitive in large-scale classiﬁcation
[12].
Since the power of this second paradigm is critically
dependent on the accuracy and the number of object
proposals, an increasing body of research has delved
J. Pont-Tuset and F. Marques are with the Department of Signal Theory
and Communications, Universitat Polit` ecnica de Catalunya, BarcelonaTech
(UPC), Spain. E-mail: fjordi.pont,ferran.marques g@upc.edu
P . Arbel´ aez is with the Department of Biomedical Engineering, Universidad de los Andes, Colombia. E-mail: pa.arbelaez@uniandes.edu.co
J. T. Barron, and J. Malik are with the Department of Electrical
Engineering and Computer Science, University of California at Berkeley,
Berkeley, CA 94720. E-mail: fbarron,malikg@eecs.berkeley.edu
* The ﬁrst two authors contributed equally
Fig. 1. Top: original image, instance-level ground truth
from COCO and our multiscale hierarchical segmentation. Bottom: our best object proposals among 150.
into the problem of their generation [13], [14], [15], [12],


key: chroma:document
string_value:
into the problem of their generation [13], [14], [15], [12],
[16], [17], [18], [19]. However, those approaches typically
focus on learning generic properties of objects from
a set of examples, while reasoning on a ﬁxed set of
regions and contours produced by external bottom-up
segmenters such as [20], [21].
In this paper, we propose a uniﬁed approach to
multiscale hierarchical segmentation and object proposal
generation called Multiscale Combinatorial Grouping
(MCG). Fig. 1 shows an example of our results and Fig. 2
an overview of our pipeline. Our main contributions are:
An efﬁcient normalized cuts algorithm, which in
practice provides a 20speed-up to the eigenvector
computation required for contour globalization [20],
[22] (Sect. 3.1).
A state-of-the-art hierarchical segmenter that leverages multiscale information (Sect. 3.3).
A grouping algorithm that produces accurate object
proposals by efﬁciently exploring the combinatorial
space of our multiscale regions (Sect. 5).arXiv:1503.00848v4  [cs.CV]  1 Mar 2016
2


key: chroma:document
string_value:
space of our multiscale regions (Sect. 5).arXiv:1503.00848v4  [cs.CV]  1 Mar 2016
2
F ixed-Scale
Segmentation R escaling &
A lignment C ombination  R es oluti on 
C ombinatorial
Grouping
 
I mage P yramid  Segmentati on P yramid  Al i gned  H iera rchies  Object Proposals  
 Multiscale Hierarchy
Fig. 2. Multiscale Combinatorial Grouping . Starting from a multiresolution image pyramid, we perform hierarchical
segmentation at each scale independently. We align these multiple hierarchies and combine them into a single
multiscale segmentation hierarchy. Our grouping component then produces a ranked list of object proposals by
efﬁciently exploring the combinatorial space of these regions.
We conduct a comprehensive and large-scale empirical
validation. On the BSDS500 (Sect. 4) we report signiﬁcant
progress in contour detection and hierarchical segmentation. On the VOC2012, SBD, and COCO segmentation
datasets (Sect. 6), our proposals obtain overall state-ofthe-art accuracy both as segmented proposals and as


key: chroma:document
string_value:
datasets (Sect. 6), our proposals obtain overall state-ofthe-art accuracy both as segmented proposals and as
bounding boxes. MCG is efﬁcient, its good generalization power makes it parameter free in practice, and it
provides a ranked set of proposals that are competitive
in all regimes of number of proposals.
2 R ELATED WORK
For space reasons, we focus our review on recent normalized cut algorithms and object proposals for recognition.
Fast normalized cuts : The efﬁcient computation of
normalized-cuts eigenvectors has been the subject of
recent work, as it is often the computational bottleneck
in grouping algorithms. Taylor [23] presented a technique for using a simple watershed oversegmentation
to reduce the size of the eigenvector problem, sacriﬁcing accuracy for speed. We take a similar approach
of solving the eigenvector problem in a reduced space,
though we use simple image-pyramid operations on
the afﬁnity matrix (instead of a separate segmentation
algorithm) and we see no loss in performance despite a
20speed improvement. Maire and Yu [24] presented
a novel multigrid solver for producing eigenvectors at
multiple scales, which speeds up ﬁne-scale eigenvector
computation by leveraging coarse-scale solutions. Our


key: chroma:document
string_value:
multiple scales, which speeds up ﬁne-scale eigenvector
computation by leveraging coarse-scale solutions. Our
technique also uses the scale-space structure of an image,
but instead of solving the problem at multiple scales,
we simply reduce the scale of the problem, solve it at
a reduced scale, and then upsample the solution while
preserving the structure of the image. As such, our
technique is faster and much simpler, requiring only a
few lines of code wrapped around a standard sparse
eigensolver.Object Proposals : Class-independent methods that
generate object hypotheses can be divided into those
whose output is an image window and those that generate segmented proposals.
Among the former, Alexe et al. [16] propose an objectness measure to score randomly-sampled image windows based on low-level features computed on the
superpixels of [21]. Manen et al. [25] propose to use the
Randomized Prim’s algorithm, Zitnick et al. [26] group
contours directly to produce object windows, and Cheng
et al. [27] generate box proposals at 300 images per
second. In contrast to these approaches, we focus on
the ﬁner-grained task of pixel-accurate object extraction,


key: chroma:document
string_value:
second. In contrast to these approaches, we focus on
the ﬁner-grained task of pixel-accurate object extraction,
rather than on window selection. However, by just taking the bounding box around our segmented proposals,
our results are also state of the art as window proposals.
Among the methods that produce segmented proposals, Carreira and Sminchisescu [18] hypothesize a set of
placements of fore- and background seeds and, for each
conﬁguration, solve a constrained parametric min-cut
(CPMC) problem to generate a pool of object hypotheses.
Endres and Hoiem [19] base their category-independent
object proposals on an iterative generation of a hierarchy
of regions, based on the contour detector of [20] and
occlusion boundaries of [28]. Kim and Grauman [17]
propose to match parts of the shape of exemplar objects,
regardless of their class, to detected contours by [20].
They infer the presence and shape of a proposal object
by adapting the matched object to the computed superpixels.
Uijlings et al. [12] present a selective search algorithm
based on segmentation. Starting with the superpixels
of [21] for a variety of color spaces, they produce a set of
segmentation hierarchies by region merging, which are


key: chroma:document
string_value:
based on segmentation. Starting with the superpixels
of [21] for a variety of color spaces, they produce a set of
segmentation hierarchies by region merging, which are
used to produce a set of object proposals. While we also
take advantage of different hierarchies to gain diversity,
we leverage multiscale information rather than different
3
color spaces.
Recently, two works proposed to train a cascade of
classiﬁers to learn which sets of regions should be
merged to form objects. Ren and Shankhnarovich [29]
produce full region hierarchies by iteratively merging
pairs of regions and adapting the classiﬁers to different
scales. Weiss and Taskar [30] specialize the classiﬁers also
to size and class of the annotated instances to produce
object proposals.
Malisiewicz and Efros [4] took one of the ﬁrst steps
towards combinatorial grouping, by running multiple
segmenters with different parameters and merging up
to three adjacent regions. In [8], another step was taken
by considering hierarchical segmentations at three different scales and combining pairs and triplets of adjacent
regions from the two coarser scales to produce object
proposals.
The most recent wave of object proposal algorithms


key: chroma:document
string_value:
regions from the two coarser scales to produce object
proposals.
The most recent wave of object proposal algorithms
is represented by [13], [14], and [15], which all keep the
quality of the seminal proposal works while improving
the speed considerably. Kr ¨ahenb ¨uhl and Koltun [13]
ﬁnd object proposal by identifying critical level sets
in geodesic distance transforms, based on seeds placed
in learnt places in the image. Rantalankila et al. [14]
perform a global and local search in the space of sets
of superpixels. Humayun et al. [15] reuse a graph to
perform many parametric min-cuts over different seeds
in order to speed the process up.
A substantial difference between our approach and
previous work is that, instead of relying on precomputed hierarchies or superpixels, we propose a uniﬁed approach that produces and groups high-quality
multiscale regions. With respect to the combinatorial approaches of [4], [8], our main contribution is to develop
efﬁcient algorithms to explore a much larger combinatorial space by taking into account a set of object examples,
increasing thus the likelihood of having complete objects
in the pool of proposals. Our approach has therefore


key: chroma:document
string_value:
increasing thus the likelihood of having complete objects
in the pool of proposals. Our approach has therefore
the ﬂexibility to adapt to speciﬁc applications and types
of objects, and can produce proposals at any trade-off
between their number and their accuracy.
3 T HESEGMENTATION ALGORITHM
Consider a segmentation of the image into regions that
partition its domain S=fSigi. A segmentation hierarchy
is a family of partitions fS;S1;:::;SLgsuch that: (1)
Sis the ﬁnest set of superpixels , (2)SLis the complete
domain, and (3) regions from coarse levels are unions of
regions from ﬁne levels. A hierarchy where each level Si
is assigned a real-valued index ican be represented by a
dendrogram, a region tree where the height of each node
is its index. Furthermore, it can also be represented as an
ultrametric contour map (UCM), an image obtained by
weighting the boundary of each pair of adjacent regions
in the hierarchy by the index at which they are merged
[31], [32]. This representation uniﬁes the problems of
contour detection and hierarchical image segmentation:
*
##
#


key: chroma:document
string_value:
[31], [32]. This representation uniﬁes the problems of
contour detection and hierarchical image segmentation:
*
##
#
Fig. 3. Duality between a UCM and a region tree :
Schematic view of the dual representation of a segmentation hierarchy as a region dendrogram and as an
ultrametric contour map.
a threshold at level iin the UCM produces the segmentationSi.
Figure 3 schematizes these concepts. First, the lower
left corner shows the probability of boundary of a UCM.
One of the main properties of a UCM is that when we
threshold the contour strength at a certain value, we
obtain a closed boundary map, and thus a partition.
Thresholding at different i, therefore, we obtain the
so-called merging-sequence partitions (left column in
Figure 3); named after the fact that a step in this sequence
corresponds to merging the set of regions sharing the
boundary of strength exactly i.
For instance, the boundary between the wheels and
the ﬂoor has strength 1, thus thresholding the contour
above1makes the wheels merge with the ﬂoor. If we
represent the regions in a partition as nodes of a graph,
we can then represent the result of merging them as their


key: chroma:document
string_value:
represent the regions in a partition as nodes of a graph,
we can then represent the result of merging them as their
parent in a tree. The result of sweeping all ivalues
can therefore be represented as a region tree, whose root
is the region representing the whole image (right part
of Figure 3). Given that each merging is associated with
a contour strength, the region tree is in fact a region
dendogram.
As an example, in the gPb-ucm algorithm of [20],
brightness, color and texture gradients at three ﬁxed disk
sizes are ﬁrst computed. These local contour cues are
globalized using spectral graph-partitioning, resulting in
the gPb contour detector. Hierarchical segmentation is
then performed by iteratively merging adjacent regions
based on the average gPb strength on their common
boundary. This algorithm produces therefore a tree of
regions at multiple levels of homogeneity in brightness,
color and texture, and the boundary strength of its UCM
can be interpreted as a measure of contrast.
Coarse-to-ﬁne is a powerful processing strategy in
computer vision. We exploit it in two different ways
to develop an efﬁcient, scalable and high-performance
segmentation algorithm: (1) To speed-up spectral graph


key: chroma:document
string_value:
to develop an efﬁcient, scalable and high-performance
segmentation algorithm: (1) To speed-up spectral graph
partitioning and (2) To create aligned segmentation hierarchies.
4
3.1 Fast Downsampled Eigenvector Computation
The normalized cuts criterion is a key globalization
mechanism of recent high-performance contour detectors such as [20], [22]. Although powerful, such spectral
graph partitioning has a signiﬁcant computational cost
and memory footprint that limit its scalability. In this
section, we present an efﬁcient normalized cuts approximation which in practice preserves full performance for
contour detection, while having low memory requirements and providing a 20speed-up.
Given a symmetric afﬁnity matrix A, we would like
to compute the ksmallest eigenvectors of the Laplacian
ofA. Directly computing such eigenvectors can be very
costly even with sophisticated solvers, due to the large
size ofA. We therefore present a technique for efﬁciently
approximating the eigenvector computation by taking
advantage of the multiscale nature of our problem: A
models afﬁnities between pixels in an image, and images
naturally lend themselves to multiscale or pyramid-like


key: chroma:document
string_value:
models afﬁnities between pixels in an image, and images
naturally lend themselves to multiscale or pyramid-like
representations and algorithms.
Our algorithm is inspired by two observations: 1) if
Ais bistochastic (the rows and columns of Asum to 1)
then the eigenvectors of the Laplacian Aare equal to
the eigenvectors of the Laplacian of A2, and 2) because
of the scale-similar nature of images, the eigenvectors
of a “downsampled” version of Ain which every other
pixel has been removed should be similar to the eigenvectors of A. Let us deﬁne pixel decimate (A), which
takes an afﬁnity matrix Aand returns the set of indices
of rows/columns in Acorresponding to a decimated
version of the image from which Awas constructed. That
is, ifi=pixel decimate (A), thenA[i;i]is a decimated
matrix in which alternating rows and columns of the imagehave been removed. Computing the eigenvectors of
A[i;i]works poorly, as decimation disconnects pixels in
the afﬁnity matrix, but the eigenvectors of the decimated


key: chroma:document
string_value:
A[i;i]works poorly, as decimation disconnects pixels in
the afﬁnity matrix, but the eigenvectors of the decimated
squared afﬁnity matrix A2[i;i]are similar to those of A,
because by squaring the matrix before decimation we
intuitively allow each pixel to propagate information to
all of its neighbors in the graph, maintaining connections even after decimation. Our algorithm works by
efﬁciently computing A2[i;i]asA[:;i]TA[:;i]1(the naive
approach of ﬁrst squaring Aand then decimating it is
prohibitively expensive), computing the eigenvectors of
A2[i;i], and then “upsampling” those eigenvectors back
to the space of the original image by pre-multiplying
byA[:;i]. This squaring-and-decimation procedure can
be applied recursively several times, each application
improving efﬁciency while slightly sacriﬁcing accuracy.
Pseudocode for our algorithm, which we call
“DNCuts” (Downsampled Normalized Cuts) is given in


key: chroma:document
string_value:
Pseudocode for our algorithm, which we call
“DNCuts” (Downsampled Normalized Cuts) is given in
Algorithm 1, where Ais our afﬁnity matrix and dis
the number of times that our squaring-and-decimation
operation is applied. Our algorithm repeatedly applies
our joint squaring-and-decimation procedure, computes
1. The Matlab-like notation A[:; i]indicates keeping the columns of
matrix Awhose indices are in the set i.Algorithm 1 dncuts (A;d;k )
1:A0 A
2:fors= [1;2;:::;d ]do
3:is pixel decimate (As1)
4:Bs As1[ :; is]
5:Cs diag (Bs~1)1Bs
6:As CT
sBs
7:end for
8:Xd ncuts (Ad;k)
9:fors= [d;d1;:::; 1]do
10:Xs1 CsXs
11:end for
12:return whiten (X0)
Fig. 4. Example of segmentation projection . In order to
“snap” the boundaries of a segmentation R(left) to those


key: chroma:document
string_value:
Fig. 4. Example of segmentation projection . In order to
“snap” the boundaries of a segmentation R(left) to those
of a segmentation S(middle), since they do not align, we
compute(R;S)(right) by assigning to each segment in
Sits mode among the labels of R.
the smallest keigenvectors of the ﬁnal “downsampled” matrix Adby using a standard sparse eigensolver
ncuts (Ad;k), and repeatedly “upsamples” those eigenvectors. Because our Ais not bistochastic and decimation
is not an orthonormal operation, we must do some
normalization throughout the algorithm (line 5) and
whiten the resulting eigenvectors (line 10). We found that
values ofd= 2 ord= 3 worked well in practice. Larger
values ofdyielded little speed improvement (as much of
the cost is spent downsampling A0) and start negatively
affecting accuracy. Our technique is similar to Nystrom’s
method for computing the eigenvectors of a subset of A,
but our squaring-and-decimation procedure means that
we do not depend on long-range connections between
pixels.
3.2 Aligning Segmentation Hierarchies


key: chroma:document
string_value:
but our squaring-and-decimation procedure means that
we do not depend on long-range connections between
pixels.
3.2 Aligning Segmentation Hierarchies
In order to leverage multi-scale information, our approach combines segmentation hierarchies computed
independently at multiple image resolutions. However, since subsampling an image removes details and
smooths away boundaries, the resulting UCMs are misaligned, as illustrated in the second panel of Fig. 2. In this
section, we propose an algorithm to align an arbitrary
segmentation hierarchy to a target segmentation and,
in Sect. 5, we show its effectiveness for multi-scale
segmentation.
The basic operation is to “snap” the boundaries of a
segmentationR=fRigito a segmentation S=fSjgj, as
illustrated in Fig. 4. For this purpose, we deﬁne L(Sj),
5
the new label of a region Sj2S, as the majority label of
its pixels inR:
L(Sj) = arg max
ijSj\Rij
jSjj(1)
We call the segmentation deﬁned by this new labeling of
all the regions of Stheprojection ofRontoSand denote
it by(R;S).


key: chroma:document
string_value:
We call the segmentation deﬁned by this new labeling of
all the regions of Stheprojection ofRontoSand denote
it by(R;S).
In order to project an UCM onto a target segmentation
S, which we denote (UCM;S), we project in turn each
of the levels of the hierarchy onto S. Note that, since
all the levels are projected onto the same segmentation,
the family of projections is by construction a hierarchy of segmentations. This procedure is summarized in
pseudo-code in Algorithm 2.
Algorithm 2 UCM Rescaling and Alignment
Require: An UCM with a set of levels [t1;:::;tK]
Require: A target segmentation S
1:UCM 0
2:fort= [t1;:::;tK]do
3:S sampleHierarchy (UCM;t)
4:S rescaleSegmentation (S;S)
5:S (S;S)
6:contours extractBoundary (S)
7: UCM max(UCM;tcontours )
8:end for
9:return UCM
Observe that the routines sampleHierarchy and
extractBoundary can be computed efﬁciently because


key: chroma:document
string_value:
8:end for
9:return UCM
Observe that the routines sampleHierarchy and
extractBoundary can be computed efﬁciently because
they involve only thresholding operations and connected
components labeling. The complexity is thus dominated
by rescaleSegmentation in Step 4, a nearest neighbor
interpolation, and the projection in Step 5, which are
computedKtimes.
3.3 Multiscale Hierarchical Segmentation
Single-scale segmentation : We consider as input
the following local contour cues: (1) brightness, color and
texture differences in half-disks of three sizes [33], (2)
sparse coding on patches [22], and (3) structured forest
contours [34]. We globalize the contour cues independently using our fast eigenvector gradients of Sect. 3.1,
combine global and local cues linearly, and construct
an UCM based on the mean contour strength. We tried
learning weights using gradient ascent on the F-measure
on the training set [20], but evaluating the ﬁnal hierarchies rather than open contours. We observed that this
objective favors the quality of contours at the expense of
regions and obtained better overall results by optimizing
the Segmentation Covering metric [20].


key: chroma:document
string_value:
objective favors the quality of contours at the expense of
regions and obtained better overall results by optimizing
the Segmentation Covering metric [20].
Hierarchy Alignment : We construct a multiresolution pyramid with Nscales by subsampling / supersampling the original image and applying our singlescale segmenter. In order to preserve thin structures
and details, we declare as set of possible boundarylocations the ﬁnest superpixels in the highest-resolution.
Then, applying recursively Algorithm 2, we project each
coarser UCM onto the next ﬁner scale until aligning it
to the highest resolution superpixels.
Multiscale Hierarchy : After alignment, we have a
ﬁxed set of boundary locations, and Nstrengths for
each of them, coming from the different scales. We
formulate this problem as binary boundary classiﬁcation
and train a classiﬁer that combines these Nfeatures into
a single probability of boundary estimation. We experimented with several learning strategies for combining
UCM strengths: (a) Uniform weights transformed into
probabilities with Platt’s method. (b) SVMs and logistic
regression, with both linear and additive kernels. (c)


key: chroma:document
string_value:
probabilities with Platt’s method. (b) SVMs and logistic
regression, with both linear and additive kernels. (c)
Random Forests. (d) The same algorithm as for singlescale. We found the results with all learning methods
surprisingly similar, in agreement with the observation
reported by [33]. This particular learning problem, with
only a handful of dimensions and millions of data points,
is relatively easy and performance is mainly driven by
our already high-performing and well calibrated features. We therefore use the simplest option (a).
4 E XPERIMENTS ON THE BSDS500
We conduct extensive experiments on the BSDS500 [35],
using the standard evaluation metrics and following the
best practice rules of that dataset. We also report results
with a recent evaluation metric Fop[36], [37], PrecisionRecall for objects and parts, using the publicly-available
code.
Single-scale Segmentation : Table 1-top shows the
performance of our single-scale segmenter for different
types of input contours on the validation set of the
BSDS500. We obtain high-quality hierarchies for all the
cues considered, showing the generality of our approach.
Furthermore, when using them jointly (row ’Comb.’
in top panel), our segmenter outperforms the versions
with individual cues, suggesting its ability to leverage


key: chroma:document
string_value:
Furthermore, when using them jointly (row ’Comb.’
in top panel), our segmenter outperforms the versions
with individual cues, suggesting its ability to leverage
diversiﬁed inputs. In terms of efﬁciency, our fast normalized cuts algorithm provides an average 20 speedup over [20], starting from the same local cues, with
no signiﬁcant loss in accuracy and with a low memory
footprint.
Multiscale Segmentation : Table 1-bottom evaluates
our full approach in the same experimental conditions as
the upper panel. We observe a consistent improvement
in performance in all the metrics for all the inputs, which
validates our architecture for multiscale segmentation.
We experimented with the range of scales and found
N=f0:5;1;2gadequate for our purposes. A ﬁner
sampling or a wider range of scales did not provide
noticeable improvements. We tested also two degraded
versions of our system (not shown in the table). For
the ﬁrst one, we resized contours to the original image
resolution, created UCMs and combined them with the
same method as our ﬁnal system. For the second one, we
transformed per-scale UCMs to the original resolution,
6
Boundary Region


key: chroma:document
string_value:
same method as our ﬁnal system. For the second one, we
transformed per-scale UCMs to the original resolution,
6
Boundary Region
Fb Fop SC PRI VI
Input ODS OIS ODS OIS ODS OIS ODS OIS ODS OISSingle-ScalePb [33] 0.702 0.733 0.334 0.370 0.577 0.636 0.801 0.847 1.692 1.490
SC [22] 0.697 0.725 0.264 0.306 0.540 0.607 0.777 0.835 1.824 1.659
SF [34] 0.719 0.737 0.338 0.399 0.582 0.651 0.803 0.851 1.608 1.432
Comb. 0.719 0.750 0.358 0.403 0.602 0.655 0.809 0.855 1.619 1.405MultiscalePb [33] 0.713 0.745 0.350 0.389 0.598 0.656 0.807 0.856 1.601 1.418
SC [22] 0.705 0.734 0.331 0.384 0.579 0.647 0.799 0.851 1.637 1.460


key: chroma:document
string_value:
SC [22] 0.705 0.734 0.331 0.384 0.579 0.647 0.799 0.851 1.637 1.460
SF [34] 0.725 0.744 0.370 0.420 0.600 0.660 0.810 0.854 1.557 1.390
Comb. 0.725 0.757 0.371 0.408 0.611 0.670 0.813 0.862 1.548 1.367
TABLE 1
BSDS500 val set. Control experiments for single-scale
(top) and multiscale (bottom) hierarchical segmentation
with different input contour detectors
but omitted the strength transfer to the ﬁnest superpixels
before combining them. The ﬁrst ablated version produces interpolation artifacts and smooths away details,
while the second one suffers from misalignment. Both
fail to improve performance over the single-scale result,
which provides additional empirical support for our
multiscale approach. We also observed a small degradation in performance when forcing the input contour
detector to use only the original image resolution, which
indicates the advantages of considering multiscale information at all stages of processing.
Since there are no drastic changes in our results when
taking as input the different individual cues or their combination, in the sequel we use the version with structured


key: chroma:document
string_value:
Since there are no drastic changes in our results when
taking as input the different individual cues or their combination, in the sequel we use the version with structured
forests for efﬁciency reasons, which we denote MCGUCM-Our .
Comparison with state-of-the-art. : Figure 5 compares our multiscale hierarchical segmenter MCG ( )
and our single-scale hierarchical segmenter SCG ( )
on the BSDS500 test set against all the methods for
which there is publicly available code. We also compare
to the recent ISCRA [29] hierarchies ( ), provided
precomputed by the authors. We obtain consistently the
best results to date on BSDS500 for all operating regimes,
both in terms of boundary and region quality.
Note that the range of object scales in the BSDS500
is limited, which translates into modest absolute gains
from MCG ( ) with respect to SCG ( ) in terms
of boundary evaluation (left-hand plot), but more signiﬁcant improvements in terms of objects and parts
(right-hand plot). We will also observe more substantial
improvements with respect to gPb-UCM ( ) when we
move to PASCAL, SBD, and COCO in Section 6 (e.g. see
Fig. 9).


key: chroma:document
string_value:
move to PASCAL, SBD, and COCO in Section 6 (e.g. see
Fig. 9).
Ground-Truth Hierarchy : In order to gain further
insights, we transfer the strength of each ground-truth
segmentation to our highest-resolution superpixels SN
and construct a combined hierarchy. This approximation to the semantic hierarchy, Ground-Truth Hierarchy
(GTH) in Fig. 5, is an upper-bound for our approach as
both share the same boundary locations and the only
difference is their strength. Since the strength of GTHis proportional to the number of subjects marking it, it
provides naturally the correct semantic ordering, where
outer object boundaries are stronger than internal parts.
Recently, Maire et al. [38] developed an annotation
tool where the user encodes explicitly the “perceptual
strength” of each contour. Our approach provides an
alternative where the semantic hierarchy is reconstructed
by sampling ﬂat annotations from multiple subjects.
5 O BJECT PROPOSAL GENERATION
The image segmentation algorithm presented in the
previous sections builds on low-level features, so its
regions are unlikely to represent accurately complete
objects with heterogeneous parts. In this context, object
proposal techniques create a set of hypotheses, possibly
overlapping, which are more likely to represent full


key: chroma:document
string_value:
objects with heterogeneous parts. In this context, object
proposal techniques create a set of hypotheses, possibly
overlapping, which are more likely to represent full
object instances.
Our approach to object proposal generation is to
combinatorially look for sets of regions from our segmentation hierarchies that merged together are likely
to represent complete objects. In this section, we ﬁrst
describe the efﬁcient computation of certain region descriptors on a segmentation tree. Then, we describe how
we use these techniques to efﬁciently explore the sets of
merged regions from the hierarchy. Finally, we explain
how we train the parameters of our algorithm for object
proposals and how we rank the candidates by their
probability of representing an object.
Fast computation of descriptors : Let us assume, for
instance, we want to compute the area of all regions
in the hierarchy. Intuitively, working strictly on the
merging-sequence partitions, we would need to scan all
pixels in all partitions. On the other hand, working on
the region tree allows us to scan the image only once to
compute the area of the leaves, and then propagate the
area to all parents as the addition of the areas of their
children.
As a drawback, the algorithms become intricate in
terms of coding and necessary data structures. Take, for


key: chroma:document
string_value:
area to all parents as the addition of the areas of their
children.
As a drawback, the algorithms become intricate in
terms of coding and necessary data structures. Take, for
instance, the computation of the neighbors of a certain
region, which is trivial via scanning the partition on the
merging sequence (look for region labels in the adjacent
boundary pixels), but need tailored data structures and
algorithms in the region tree.
Formally, let us assume the image has ppixels, and we
build a hierarchy based on ssuperpixels (leaves of the
tree), andmmergings (different UCM strength values).
The cost of computing the area on all regions using the
merging-sequence partitions will be the cost of scanning
all pixels in these partitions, thus p(m+1). In contrast,
the cost using the region tree will involve scanning the
image once, and then propagating the area values, so
p+m, which is notably faster.
We built tailored algorithms and data structures to
compute the bounding box, perimeter, and neighbors of
a region using the region tree representation.
7
Boundaries
0 0:10:20:30:40:50:60:70:80:9 100:10:20:30:40:50:60:70:80:91
RecallPrecision


key: chroma:document
string_value:
RecallPrecision
Human [0.81-0.21] ISCRA [0.72]
GTH [0.96] NCuts [0.63]
MCG-Our [0.74] EGB [0.61]
SCG-Our [0.74] MShift [0.60]
gPb-UCM [0.73] Quadtree [0.41]Objects and Parts
0 0:10:20:30:40:50:60:70:80:9 100:10:20:30:40:50:60:70:80:91
RecallPrecisionHuman [0.56-0.05]
GTH [0.64]
MCG-Our [0.38]
SCG-Our [0.35]
ISCRA [0.35]
gPb-UCM [0.35]
MShift [0.23]
NCuts [0.21]
EGB [0.16]
Quadtree [0.06]
Fig. 5. BSDS500 test set. Precision-Recall curves for boundaries [35] (left) and for objects and parts [36] (right). The
marker on each curve is placed on the Optimal Dataset Scale (ODS), and its F measure is presented in brackets in the


key: chroma:document
string_value:
marker on each curve is placed on the Optimal Dataset Scale (ODS), and its F measure is presented in brackets in the
legend. The isolated red asterisks refer to the human performance assessed on the same image (one human partition
against the rest of human annotations on the same image) and on a different image (one human partition against the
human partitions of a different, randomly selected, image).
Combinatorial Grouping of Proposals : We can
cast object segmentation as selecting some regions in
the hierarchy, or in other words, as a combinatorial
optimization problem on the hierarchy. To illustrate this
principle, Figure 6(a) shows the simpliﬁed representation
of the hierarchy in Figure 3. Figure 6(b) and (c) show two
object proposals, and their representation in the region
hierarchy.
R4
R5
R1R2R3R6
R9
R7
R1R2R3R4R8
R5R6
(a)
R9
R7
R1R2R3R4R8
R5R6
(b)
R9
R7
R1R2R3R4R8
R5R6
(c)


key: chroma:document
string_value:
R5R6
(b)
R9
R7
R1R2R3R4R8
R5R6
(c)
Fig. 6. Object segmentation as combinatorial optimization : Examples of objects (b), (c), formed by selecting regions from a hierarchy (a).
Since hierarchies are built taking only low-level features into account, and do not use semantic information,
objects will usually not be optimally represented using a
single region in the hierarchy. As an example, Figure 6(c)
shows the optimum representation of the car, consisting
of three regions.
A sensible approach to create object proposals is therefore to explore the set of n-tuples of regions. The main
idea behind MCG is to explore this set efﬁciently, taking
advantage of the region tree representation, via the fastcomputation of region neighbors.
The whole set of tuples, however, is huge, and so it
is not feasible to explore it exhaustively. Our approach
ranks the proposals using the height of their regions in
the tree (UCM strength) and explores the tree from the
top, but only down to a certain threshold. To speed the
process up, we design a top-down algorithm to compute
the region neighbors and thus only compute them down
to a certain depth in the tree.


key: chroma:document
string_value:
process up, we design a top-down algorithm to compute
the region neighbors and thus only compute them down
to a certain depth in the tree.
To further improve the results, we not only consider
then-tuples from the resulting MCG-UCM-Our hierarchy, but also the rest of hierarchies computed at different
scales. As we will show in the experiments, diversity
signiﬁcantly improves the proposal results.
Parameter Learning via Pareto Front Optimization :
MCG takes a set of diverse hierarchies and computes the
n-tuples up to a certain UCM strength. We can interpret
then-tuples from each hierarchy as a ranked list of Ni
proposals that are put together to create the ﬁnal set of
Npproposals.
At training time, we would like to ﬁnd, for different values of Np, the number of proposals from each
ranked list Nisuch that the joint pool of Npproposals
has the best achievable quality. We frame this learning
problem as a Pareto front optimization [39], [40] with
two conﬂicting objective functions: number of proposals
and achievable quality. At test time, we select a working
point on the Pareto front, represented by the
Ni


key: chroma:document
string_value:
and achievable quality. At test time, we select a working
point on the Pareto front, represented by the
Ni	
values, based either on the number of proposals Npwe
can handle or on the minimum achievable quality our
8
application needs, and we combine the Nitop proposals
from each hierarchy list.
Formally, assuming Rranked lists Li, an exhaustive
learning algorithm would consider all possible values
of theR-tuplefN1;:::;NRg, whereNi2f0;:::;jLijg;
adding up toQR
1jLijparameterizations to try, which is
intractable in our setting.
Figure 7 illustrates the learning process. To reduce the
dimensionality of the search space, we start by selecting
two ranked lists L1,L2(green curves) and we sample
the list atSlevels of number of proposals (green dots).
We then scan the full S2different parameterizations to
combine the proposals from both (blue dots). In other
words, we analyze the sets of proposals created by
combining the top N1fromL1(green dots) and the top
N2fromL2.
1011021031040.30.60.9
Number of proposalsQuality
1011021031040.30.60.9
Number of proposalsQuality


key: chroma:document
string_value:
N2fromL2.
1011021031040.30.60.9
Number of proposalsQuality
1011021031040.30.60.9
Number of proposalsQuality
1011021031040.30.60.9
Number of proposalsQualityRanked lists of object proposals
L 1 L RPareto front
reduction of parameters
{N 1 · · ·  N R}
Fig. 7. Pareto front learning : Training the combinatorial
generation of proposals using the Pareto front
The key step of the optimization consists in discarding
those parameterizations whose quality point is not in
the Pareto front (red curve). (i.e., those parameterizations
that can be substituted by another with better quality
with the same number of proposals, or by one with the
same quality with less proposals.) We sample the Pareto
front toSpoints and we iterate the process until all the
ranked lists are combined.
Each point in the ﬁnal Pareto front corresponds to a
particular parameterization fN1;:::;NRg. At train time,
we choose a point on this curve, either at a given
number of proposals Ncor at the achievable quality
we are interested in (black triangle) and store the parameters
N1;:::;NR	
. At test time, we combine the


key: chroma:document
string_value:
we are interested in (black triangle) and store the parameters
N1;:::;NR	
. At test time, we combine the
N1;:::;NR	
top proposals from each ranked list. The
number of sampled conﬁgurations using the proposed
algorithm is (R1)S2, that is, we have reduced an
exponential problem ( SR) to a quadratic one.
Regressed Ranking of Proposals : To further reduce
the number of proposals, we train a regressor from
low-level features, as in [18]. Since the proposals are
all formed by a set of regions from a reduced set of
hierarchies, we focus on features that can be computed
efﬁciently in a bottom-up fashion, as explained previously.
We compute the following features:
Size and location : Area and perimeter of the candidate; area, position, and aspect ratio of the bounding
box; and the area balance between the regions in thecandidate.
Shape : Perimeter (and sum of contour strength)
divided by the squared root of the area; and area of the
region divided by that of the bounding box.
Contours : Sum of contour strength at the boundaries, mean contour strength at the boundaries; minimum and maximum UCM threshold of appearance and


key: chroma:document
string_value:
Contours : Sum of contour strength at the boundaries, mean contour strength at the boundaries; minimum and maximum UCM threshold of appearance and
disappearance of the regions forming the candidate.
We train a Random Forest using these features to regress
the object overlap with the ground truth, and diversify
the ranking based on Maximum Marginal Relevance
measures [18]. We tune the random forest learning on
half training set and validating on the other half. For
the ﬁnal results, we train on the training set and evaluate
our proposals on the validation set of PASCAL 2012.
6 E XPERIMENTS ON PASCAL VOC, SBD,
AND COCO
This section presents our large-scale empirical validation
of the object proposal algorithm described in the previous section. We perform experiments in three annotated
databases, with a variety of measures that demonstrate
the state-of-the-art performance of our algorithm.
Datasets and Evaluation Measures : We conduct experiments in the following three annotated datasets: the
segmentation challenge of PASCAL 2012 Visual Object
Classes (SegVOC12) [41], the Berkeley Semantic Boundaries Dataset (SBD) [42], and the Microsoft Common
Objects in Context (COCO) [43]. They all consist of
images with annotated objects of different categories.


key: chroma:document
string_value:
Objects in Context (COCO) [43]. They all consist of
images with annotated objects of different categories.
Table 2 summarizes the number of images and object
instances in each database.
Number of Number of Number of
Classes Images Objects
SegVOC12 20 2 913 9 847
SBD 20 12 031 32 172
COCO 80 123 287 910 983
TABLE 2
Sizes of the databases
Regarding the performance metrics, we measure the
achievable quality with respect to the number of proposals, that is, the quality we would have if an oracle
selected the best proposal among the pool. This aligns
with the fact that object proposals are a preprocessing
step for other algorithms that will represent and classify
them. We want, therefore, the achievable quality within
the proposals to be as high as possible, while reducing
the number of proposals to make the ﬁnal system as fast
as possible.
As a measure of quality of a speciﬁc proposal with
respect to an annotated object, we consider the Jaccard
indexJ, also known as overlap or intersection over
union; which is deﬁned as the size of the intersection
of the two pixel sets over the size of their union.
9
To compute the overall quality for the whole database,


key: chroma:document
string_value:
of the two pixel sets over the size of their union.
9
To compute the overall quality for the whole database,
we ﬁrst select the best proposal for each annotated
instance with respect to J. The Jaccard index at instance
level (Ji) is then deﬁned as the mean best overlap for all
the ground-truth instances in the database, also known
as Best Spatial Support score (BSS) [4] or Average Best
Overlap (ABO) [12].
Computing the mean of the best overlap on all objects,
as done by Ji, hides the distribution of quality among
different objects. As an example, Ji= 0:5can mean
that the algorithm covers half the objects perfectly and
completely misses the other half, or can also mean that
all the objects are covered exactly at J= 0:5. This
information might be useful to decide which algorithm
to use. Computing a histogram of the best overlap would
provide very rich information, but then the resulting
plot would be 3D (number of proposals, bins, and bin
counts). Alternatively, we propose to plot different percentiles of the histogram.
Interestingly, a certain percentile of the histogram
of best overlaps consists in computing the number of
objects whose best overlap is above a certain Jaccard


key: chroma:document
string_value:
Interestingly, a certain percentile of the histogram
of best overlaps consists in computing the number of
objects whose best overlap is above a certain Jaccard
threshold, which can be interpreted as the best achievable recall of the technique over a certain threshold. We
compute the recall at three different thresholds: J= 0:5,
J=0:7, andJ=0:85.
Learning Strategy Evaluation : We ﬁrst estimate the
loss in performance due to not sweeping all the possible
values offN1;:::;NRgin the combination of proposal
lists via the proposed greedy strategy. To do so, we will
compare this strategy with the full combination on a
reduced problem to make the latter feasible. Speciﬁcally,
we combine the 4 ranked lists coming from the singletons at all scales, instead of the full 16 lists coming from
singletons, pairs, triplets, and 4-tuples. We also limit the
search to 20 000 proposals, further speeding the process
up.
In this situation, the mean loss in achievable quality
along the full curve of parameterization is Ji= 0:0002 ,
with a maximum loss of Ji=0:004(0:74%). In exchange,
our proposed learning strategy on the full 16 ranked lists
takes about 20 seconds to compute on the training set of


key: chroma:document
string_value:
our proposed learning strategy on the full 16 ranked lists
takes about 20 seconds to compute on the training set of
SegVOC12, while the singleton-limited full combination
takes 4 days (the full combination would take months).
Combinatorial Grouping : We now evaluate the
Pareto front optimization strategy in the training set of
SegVOC12. As before, we extract the lists of proposals
from the three scales and the multiscale hierarchy, for
singletons, pairs, triplets, and 4-tuples of regions, leading
to 16 lists, ranked by the minimum UCM strength of the
regions forming each proposal.
Figure 8 shows the Pareto front evolution of Jiwith
respect to the number of proposals for up to 1, 2, 3, and
4 regions per proposal (4, 8, 12, and 16 lists, respectively)
at training and testing time on SegVOC12. As baselines,
we plot the raw singletons from MCG-UCM-Our, gPbUCM, and Quadtree; as well as the uniform combination
of scales.103104105020406080100
Number of proposalsRegion distribution percentageScale 2.0
Scale 1.0
Scale 0.5
Multi-Scale
Fig. 10. Region distribution learnt by the Pareto front
optimization on SegVOC12.


key: chroma:document
string_value:
Scale 1.0
Scale 0.5
Multi-Scale
Fig. 10. Region distribution learnt by the Pareto front
optimization on SegVOC12.
The improvement of considering the combination of
all 1-region proposals ( ) from the 3 scales and the
MCG-UCM-Our with respect to the raw MCG-UCM-Our
( ) is signiﬁcant, which corroborates the gain in diversity obtained from hierarchies at different scales. In turn,
the addition of 2- and 3-region proposals ( and )
noticeably improves the achievable quality. This shows
that hierarchies do not get full objects in single regions,
which makes sense given that they are built using lowlevel features only. The improvement when adding 4tuples ( ) is marginal at the number of proposals we
are considering. When analyzing the equal distribution
of proposals from the four scales ( ), we see that the
less proposals we consider, the more relevant the Pareto
optimization becomes. At the selected working point, the
gain of the Pareto optimization is 2 points.
Figure 10 shows the distribution of proposals from
each of the scales combined in the Pareto front. We
see that the coarse scale (0.5) is the most picked at
low number of proposals, and the rest come into play
when increasing their number, since one can afford more


key: chroma:document
string_value:
see that the coarse scale (0.5) is the most picked at
low number of proposals, and the rest come into play
when increasing their number, since one can afford more
detailed proposals. The multi-scale hierarchy is the one
with less weight, since it is created from the other three.
Pareto selection and ranking : Back to Figure 8,
the red asterisk ( ) marks the selected conﬁguration
N1;:::;NR	
in the Pareto front (black triangle in Figure 7), which is selected at a practical level of proposals.
The red plus sign ( ) represents the set of proposals after
removing those duplicate proposals whose overlap leads
to a Jaccard higher than 0.95. The proposals at this point
are the ones that are ranked by the learnt regressor ( ).
At test time (right-hand plot), we directly combine the
learnt
N1;:::;NR	
proposals from each ranked list.
Note that the Pareto optimization does not overﬁt, given
the similar result in the training and validation datasets.
We then remove duplicates and rank the results. In this
case, note the difference between the regressed result in


key: chroma:document
string_value:
the similar result in the training and validation datasets.
We then remove duplicates and rank the results. In this
case, note the difference between the regressed result in
the training and validation sets, which reﬂects overﬁtting, but despite this we found it beneﬁcial with respect
to the non-regressed result.
10
Training
1021031041051060:40:50:60:70:80:9
Number of proposalsJaccard index at instance level ( Ji)Validation
1021031041051060:40:50:60:70:80:9
Number of proposalsJaccard index at instance level ( Ji)Pareto up to 4-tuples
Pareto up to triplets
Pareto up to pairs
Pareto only singletons
Raw Ours-multi singl.
Raw gPb-UCM singl.
Raw Quadtree singl.
Equal distribution
Selected conﬁguration
Filtered candidates
Regressed ranking
Fig. 8. Pareto front evaluation. Achievable quality of our proposals for singletons, pairs, triplets, and 4-tuples; and
the raw proposals from the hierarchies on PASCAL SegVOC12 training (left) and validation (right) sets.
PASCAL SegVOC12


key: chroma:document
string_value:
the raw proposals from the hierarchies on PASCAL SegVOC12 training (left) and validation (right) sets.
PASCAL SegVOC12
1021031040:30:40:50:60:70:80:9
Number of proposalsJaccard index at instance level ( Ji)SBD
1021031040:30:40:50:60:70:80:9
Number of proposalsJaccard index at instance level ( Ji)COCO
1021031040:30:40:50:60:70:80:9
Number of proposalsJaccard index at instance level ( Ji)MCG-Our
SCG-Our
CPMC [18]
CI [19]
GOP [13]
GLS [14]
RIGOR [15]
ShSh [17]
SeSe [12]
MCG-UCM-Our
gPb-UCM
Quadtree
Fig. 9. Object Proposals: Jaccard index at instance level. Results on SegVOC12, SBD, and COCO.
In the validation set of SegVOC12, the full set of
proposals (i.e., combining the full 16 lists) would contain
millions of proposals per image. The multiscale combinatorial grouping allows us to reduce the number of


key: chroma:document
string_value:
proposals (i.e., combining the full 16 lists) would contain
millions of proposals per image. The multiscale combinatorial grouping allows us to reduce the number of
proposals to 5 086 with a very high achievable Jiof0:81
(). The regressed ranking ( ) allows us to further
reduce the number of proposals below this point.
Segmented Proposals: Comparison with State of
the Art : We ﬁrst compare our results against those
methods that produce segmented object proposals [13],
[14], [15], [12], [16], [17], [18], [19], using the implementations from the respective authors. We train MCG on
the training set of SegVOC12, and we use the learnt
parameters on the validation sets of SegVOC12, SBD,
and COCO.
Figure 9 shows the achievable quality at instance level
(Ji) of all methods on the validation set of SegVOC12,
SBD, and COCO. We plot the raw regions of MCGUCM-Our, gPb-UCM, and QuadTree as baselines where
available. We also evaluate a faster single-scale version of
MCG ( Single-scale Combinatorial Grouping - SCG ), which
takes the hierarchy at the native scale only and combines


key: chroma:document
string_value:
MCG ( Single-scale Combinatorial Grouping - SCG ), which
takes the hierarchy at the native scale only and combines
up to 4 regions per proposal. This approach decreases
the computational load one order of magnitude whilekeeping competitive results.
MCG proposals ( ) signiﬁcantly outperform the
state-of-the-art at all regimes. The bigger the database is,
the better MCG results are with respect to the rest, which
shows that our techniques better generalize to unseen
images (recall that MCG is trained only in SegVOC12).
As commented on the measures description, Jishows
mean aggregate results, so they can mask the distribution of quality among objects in the database. Figure 11
shows the recall at three different Jaccard levels. First,
these plots further highlight how challenging COCO
is, since we observe a signiﬁcant drop in performance,
more pronounced than when measured by JiandJc.
Another interesting result comes from observing the
evolution of the plots for the three different Jaccard
values. Take for instance the performance of GOP ( )
against MCG-Our ( ) in SBD. While for J=0:5GOP
slightly outperforms MCG, the higher the threshold, the
better MCG. Overall, MCG has specially good results


key: chroma:document
string_value:
slightly outperforms MCG, the higher the threshold, the
better MCG. Overall, MCG has specially good results
at higherJvalues. In other words, if one looks for
proposals of very high accuracy, MCG is the method
with highest recall, at all regimes and in all databases.
In all measures and databases, SCG ( ) obtains very
competitive results, especially if we take into account
11
PASCAL SegVOC12
10210310400:10:20:30:40:50:60:70:80:9
J=0:5
J=0:7
J=0:85
Number of proposalsRecallSBD
10210310400:10:20:30:40:50:60:70:80:9
J= 0:5
J= 0:7
J= 0:85
Number of proposalsRecallCOCO
10210310400:10:20:30:40:50:60:70:80:9
J= 0:5
J= 0:7
J= 0:85
Number of proposalsRecallMCG-Our
SCG-Our
CPMC [18]
CI [19]
GOP [13]
GLS [14]
RIGOR [15]
ShSh [17]
SeSe [12]


key: chroma:document
string_value:
CI [19]
GOP [13]
GLS [14]
RIGOR [15]
ShSh [17]
SeSe [12]
Quadtree
Fig. 11. Segmented Object Proposals: Recall at different Jaccard levels. Percentage of annotated objects for
which there is a proposal whose overlap with the segmented ground-truth shapes (not boxes) is above J= 0:5,
J= 0:7, andJ= 0:85, for different number of proposals per image. Results on SegVOC12, SBD, and COCO.
PASCAL SegVOC12
10210310400:10:20:30:40:50:60:70:80:9
J= 0:5
J= 0:7
J= 0:85
Number of proposalsRecallSBD
10210310400:10:20:30:40:50:60:70:80:9
J= 0:5
J= 0:7
J= 0:85
Number of proposalsRecallCOCO
10210310400:10:20:30:40:50:60:70:80:9
J= 0:5
J= 0:7
J= 0:85
Number of proposalsRecallMCG-Our
SCG-Our
CPMC [18]


key: chroma:document
string_value:
J= 0:5
J= 0:7
J= 0:85
Number of proposalsRecallMCG-Our
SCG-Our
CPMC [18]
CI [19]
GOP [13]
GLS [14]
RIGOR [15]
ShSh [17]
SeSe [12]
RP [25]
EB [26]
BING [27]
Obj [16]
Quadtree
Fig. 12. Bounding-Box Proposals: Recall at different Jaccard levels. Percentage of annotated objects for which
there is a bounding box proposal whose overlap with the ground-truth boxes is above J= 0:5,J= 0:7, andJ= 0:85,
for different number of proposals per image. Results on SegVOC12, SBD, and COCO.
that it is 7faster than MCG, as we will see in next
sections.
The complementarity of MCG with respect to other
proposal techniques, and their combination using the
Pareto front optimization is studied in [44].
Boxes Proposals: Comparison with State of the
Art: Although focused in providing segmented object
proposals, MCG may also be used to provide boxes
proposals, by taking the bounding box of the segmented


key: chroma:document
string_value:
Art: Although focused in providing segmented object
proposals, MCG may also be used to provide boxes
proposals, by taking the bounding box of the segmented
proposals and deduplicating them. We add the state
of the art in boxes proposals [26], [27], [25], and [16]
to the comparison. Figure 12 shows the recall values
of the boxes results when compared to the bounding
boxes of the annotated objects, for three different Jaccard
thresholds.
While many of the techniques speciﬁcally tailored to
boxes proposals are competitive at J= 0:5, their perfor-mance drops signiﬁcantly at higher Jaccard thresholds.
Despite being tailored to segmented proposals, MCG
clearly outperforms the state of the art if you look
for precise localization of the bounding boxes. Again,
SCG is very competitive, especially taking its speed into
account.
MCG and SCG Time per Image : Table 3 shows
the time per image of the main steps of our approach,
from the raw image to the contours, the hierarchical
segmentation, and the proposal generation. All times are
computed using a single core on a Linux machine. Our
full MCG takes about 25 s. per image to compute the


key: chroma:document
string_value:
segmentation, and the proposal generation. All times are
computed using a single core on a Linux machine. Our
full MCG takes about 25 s. per image to compute the
multiscale hierarchies, and 17 s. to generate and rank
the5 038 proposals on the validation set of SegVOC12.
Our single-scale SCG produces a segmentation hierarchy
of better quality than gPb-ucm [20] in less than 3 seconds
and with signiﬁcant less memory footprint.
12
Fig. 13. COCO Qualitative Results : Image, ground truth, multi-scale UCM and best MCG proposals among the 500
best ranked. (More qualitative examples in the supplemental material.)
Contour Hierarchical CandidateTotalDetection Segmentation Generation
MCG 4.61.3 20.55.6 17.09.8 42.214.8
SCG 1.00.3 2.70.7 2.60.4 6.21.1
TABLE 3
Time in seconds per image of MCG and SCG
Table 4 shows the time-per-image results compared to
the rest of state of the art in segmented proposals generation, all run on the same single-core Linux machine.
Proposal
Generation


key: chroma:document
string_value:
the rest of state of the art in segmented proposals generation, all run on the same single-core Linux machine.
Proposal
Generation
MCG (Our) 42.214.8
SCG (Our) 6.21.1
GOP [13] 1.00.3
GLS [14] 7.91.7
SeSe [12] 15.95.2
RIGOR [15] 31.616.0
CPMC [18] 120
CI [19] 120
ShSh [17] 120
TABLE 4
Time comparison for all considered state-of-the-art
techniques that produce segmented proposals. All run
on the same single-core Linux machine.Practical considerations : One of the key aspects of
object proposals is the size of the pool they generate.
Depending on the application, one may need more precision and thus a bigger pool, or one might need speed and
thus a small pool in exchange for some loss of quality.
MCG and SCG provide a ranked set of around 5 000 and
2 000 proposals, respectively, and one can take the N
ﬁrst in case the speciﬁc application needs a smaller pool.
From a practical point of view, this means that one does


key: chroma:document
string_value:
ﬁrst in case the speciﬁc application needs a smaller pool.
From a practical point of view, this means that one does
not need to re-parameterize them for the speciﬁc needs
of a certain application.
In contrast, the techniques that do not provide a
ranking of the proposals, need to be re-parameterized
to adapt them to a different number of proposals, which
is not desirable in practice.
On top of that, the results show that MCG and SCG
have outstanding generalization power to unseen images
(recall that the results for SBD and COCO have been
obtained using the learnt parameters on SegVOC12),
meaning that MCG and SCG offer the best chance to
obtain competitive results in an unseen database without
need to re-train.
Figure 13 shows some qualitative results on COCO.
7 C ONCLUSIONS
We proposed Multiscale Combinatorial Grouping
(MCG), a uniﬁed approach for bottom-up segmentation
and object proposal generation. Our approach produces
13
state-of-the-art contours, hierarchical regions, and object
proposals. At its core are a fast eigenvector computation
for normalized-cut segmentation and an efﬁcient
algorithm for combinatorial merging of hierarchical


key: chroma:document
string_value:
proposals. At its core are a fast eigenvector computation
for normalized-cut segmentation and an efﬁcient
algorithm for combinatorial merging of hierarchical
regions. We also present Single-scale Combinatorial
Grouping (SCG), a speeded up version of our technique
that produces competitive results in under ﬁve seconds
per image.
We perform an extensive validation in BSDS500,
SegVOC12, SBD, and COCO, showing the quality, robustness and scalability of MCG. Recently, an independent study [45], [46] provided further evidence to the
interest of MCG among the current state-of-the-art in
object proposal generation. Moreover, our object candidates have already been employed as integral part of
high performing recognition systems [47].
In order to promote reproducible research on perceptual grouping, all the resources of this project – code,
pre-computed results, and evaluation protocols – are
publicly available2.
Acknowledgements : The last iterations of this work
have been done while Jordi Pont-Tuset has been at Prof.
Luc Van Gool’s Computer Vision Lab (CVL) of ETHZ,
Switzerland. This work has been partly developed in the
framework of the project BIGGRAPH-TEC2013-43935-R


key: chroma:document
string_value:
Switzerland. This work has been partly developed in the
framework of the project BIGGRAPH-TEC2013-43935-R
and the FPU grant AP2008-01164; ﬁnanced by the Spanish Ministerio de Econom´ ıa y Competitividad , and the European Regional Development Fund (ERDF). This work
was partially supported by ONR MURI N000141010933.
REFERENCES
[1] P . Viola and M. Jones, “Robust real-time face detection,” IJCV ,
vol. 57, no. 2, 2004.
[2] N. Dalal and B. Triggs, “Histograms of oriented gradients for
human detection,” in CVPR , 2005.
[3] P . Felzenszwalb, R. Girshick, D. McAllester, and D. Ramanan,
“Object detection with discriminatively trained part based models,” TP AMI , vol. 32, no. 9, 2010.
[4] T. Malisiewicz and A. A. Efros, “Improving spatial support for
objects via multiple segmentations,” in BMVC , 2007.
[5] C. Gu, J. Lim, P . Arbelaez, and J. Malik, “Recognition using


key: chroma:document
string_value:
[5] C. Gu, J. Lim, P . Arbelaez, and J. Malik, “Recognition using
regions,” in CVPR , 2009.
[6] J. Carreira, F. Li, and C. Sminchisescu, “Object recognition by
sequential ﬁgure-ground ranking,” IJCV , vol. 98, no. 3, pp. 243–
262, 2012.
[7] A. Ion, J. Carreira, and C. Sminchisescu, “Probabilistic joint image
segmentation and labeling by ﬁgure-ground composition,” IJCV ,
vol. 107, no. 1, pp. 40–57, 2014.
[8] P . Arbel ´aez, B. Hariharan, C. Gu, S. Gupta, L. Bourdev, and
J. Malik, “Semantic segmentation using regions and parts,” in
CVPR , 2012.
[9] J. Carreira, R. Caseiro, J. Batista, and C. Sminchisescu, “Semantic
segmentation with second-order pooling,” in ECCV , 2012.


key: chroma:document
string_value:
segmentation with second-order pooling,” in ECCV , 2012.
[10] R. Girshick, J. Donahue, T. Darrell, and J. Malik, “Rich feature
hierarchies for accurate object detection and semantic segmentation,” in CVPR , 2014.
[11] N. Zhang, J. Donahue, R. Girshick, and T. Darrell, “Part-based
r-cnns for ﬁne-grained category detection,” in ECCV , 2014.
[12] J. R. R. Uijlings, K. E. A. van de Sande, T. Gevers, and A. W. M.
Smeulders, “Selective search for object recognition,” IJCV , vol.
104, no. 2, pp. 154–171, 2013.
2.www.eecs.berkeley.edu/Research/Projects/CS/vision/grouping/mcg/[13] P . Kr ¨ahenb ¨uhl and V . Koltun, “Geodesic object proposals,” in
ECCV , 2014.
[14] P . Rantalankila, J. Kannala, and E. Rahtu, “Generating object


key: chroma:document
string_value:
ECCV , 2014.
[14] P . Rantalankila, J. Kannala, and E. Rahtu, “Generating object
segmentation proposals using global and local search,” in CVPR ,
2014.
[15] A. Humayun, F. Li, and J. M. Rehg, “RIGOR: Recycling Inference
in Graph Cuts for generating Object Regions,” in CVPR , 2014.
[16] B. Alexe, T. Deselaers, and V . Ferrari, “Measuring the objectness
of image windows,” TP AMI , vol. 34, pp. 2189–2202, 2012.
[17] J. Kim and K. Grauman, “Shape sharing for object segmentation,”
inECCV , 2012.
[18] J. Carreira and C. Sminchisescu, “CPMC: Automatic object
segmentation using constrained parametric min-cuts,” TP AMI ,
vol. 34, no. 7, pp. 1312–1328, 2012.
[19] I. Endres and D. Hoiem, “Category-independent object proposals
with diverse ranking,” TP AMI , vol. 36, no. 2, pp. 222–234, 2014.


key: chroma:document
string_value:
with diverse ranking,” TP AMI , vol. 36, no. 2, pp. 222–234, 2014.
[20] P . Arbel ´aez, M. Maire, C. C. Fowlkes, and J. Malik, “Contour
detection and hierarchical image segmentation,” TP AMI , vol. 33,
no. 5, pp. 898–916, 2011.
[21] P . F. Felzenszwalb and D. P . Huttenlocher, “Efﬁcient graph-based
image segmentation,” IJCV , vol. 59, p. 2004, 2004.
[22] X. Ren and L. Bo, “Discriminatively trained sparse code gradients
for contour detection,” in NIPS , 2012.
[23] C. J. Taylor, “Towards fast and accurate segmentation,” CVPR ,
2013.
[24] M. Maire and S. X. Yu, “Progressive multigrid eigensolvers for
multiscale spectral segmentation,” ICCV , 2013.


key: chroma:document
string_value:
multiscale spectral segmentation,” ICCV , 2013.
[25] S. Man ´en, M. Guillaumin, and L. Van Gool, “Prime Object Proposals with Randomized Prim’s Algorithm,” in ICCV , 2013.
[26] C. L. Zitnick and P . Doll ´ar, “Edge boxes: Locating object proposals
from edges,” in ECCV , 2014.
[27] M.-M. Cheng, Z. Zhang, W.-Y. Lin, and P . H. S. Torr, “BING:
Binarized normed gradients for objectness estimation at 300fps,”
inCVPR , 2014.
[28] D. Hoiem, A. Efros, and M. Hebert, “Recovering occlusion boundaries from an image,” IJCV , vol. 91, no. 3, pp. 328–346, 2011.
[29] Z. Ren and G. Shakhnarovich, “Image segmentation by cascaded
region agglomeration,” in CVPR , 2013.
[30] D. Weiss and B. Taskar, “Scalpel: Segmentation cascades with


key: chroma:document
string_value:
region agglomeration,” in CVPR , 2013.
[30] D. Weiss and B. Taskar, “Scalpel: Segmentation cascades with
localized priors and efﬁcient learning,” in CVPR , 2013.
[31] L. Najman and M. Schmitt, “Geodesic saliency of watershed
contours and hierarchical segmentation,” TP AMI , vol. 18, no. 12,
pp. 1163–1173, 1996.
[32] P . Arbel ´aez, “Boundary extraction in natural images using ultrametric contour maps,” in POCV , June 2006.
[33] D. Martin, C. Fowlkes, and J. Malik, “Learning to detect natural
image boundaries using local brightness, color and texture cues,”
TP AMI , vol. 26, no. 5, pp. 530–549, 2004.
[34] P . Doll ´ar and C. Zitnick, “Structured forests for fast edge detection,” ICCV , 2013.
[35] http://www.eecs.berkeley.edu/Research/
Projects/CS/vision/grouping/resources.html.


key: chroma:document
string_value:
[35] http://www.eecs.berkeley.edu/Research/
Projects/CS/vision/grouping/resources.html.
[36] J. Pont-Tuset and F. Marques, “Supervised evaluation of image
segmentation and object proposal techniques,” P AMI , 2015.
[37] ——, “Measures and meta-measures for the supervised evaluation
of image segmentation,” in CVPR , 2013.
[38] M. Maire, S. X. Yu, and P . Perona, “Hierarchical scene annotation,”
inBMVC , 2013.
[39] M. Everingham, H. Muller, and B. Thomas, “Evaluating image
segmentation algorithms using the pareto front,” in ECCV , 2006.
[40] M. Ehrgott, Multicriteria optimization . Springer, 2005.
[41] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn,
and A. Zisserman, “The PASCAL Visual Object Classes
Challenge 2012 (VOC2012) Results,” http://www.pascalnetwork.org/challenges/VOC/voc2012/workshop/index.html.


key: chroma:document
string_value:
Challenge 2012 (VOC2012) Results,” http://www.pascalnetwork.org/challenges/VOC/voc2012/workshop/index.html.
[42] B. Hariharan, P . Arbelaez, L. Bourdev, S. Maji, and J. Malik,
“Semantic contours from inverse detectors,” in ICCV , 2011.
[43] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P . Perona, D. Ramanan,
P . Doll ´ar, and C. Zitnick, “Microsoft COCO: Common Objects in
Context,” in ECCV , 2014.
[44] J. Pont-Tuset and L. V . Gool, “Boosting object proposals: From
pascal to COCO,” in ICCV , 2015.
[45] J. Hosang, R. Benenson, and B. Schiele, “How good are detection
proposals, really?” in BMVC , 2014.
[46] J. Hosang, R. Benenson, P . Doll ´ar, and B. Schiele, “What makes
for effective detection proposals?” P AMI , 2015.
14


key: chroma:document
string_value:
for effective detection proposals?” P AMI , 2015.
14
[47] B. Hariharan, P . Arbel ´aez, R. Girshick, and J. Malik, “Simultaneous detection and segmentation,” in ECCV , 2014.
Jordi Pont-Tuset is a post-doctoral researcher
at ETHZ, Switzerland, in Prof. Luc Van Gool’s
computer vision group (2015). He received the
degree in Mathematics in 2008, the degree in
Electrical Engineering in 2008, the M.Sc. in
Research on Information and Communication
Technologies in 2010, and the Ph.D with honors
in 2014; all from the Universitat Polit `ecnica de
Catalunya, BarcelonaTech (UPC). He worked at
Disney Research, Z ¨urich (2014).
Pablo Arbel ´aezreceived a PhD with honors in
Applied Mathematics from the Universit ´e ParisDauphine in 2005. He was a Research Scientist
with the Computer Vision Group at UC Berkeley from 2007 to 2014. He currently holds a
faculty position at Universidad de los Andes in
Colombia. His research interests are in computer vision, where he has worked on a number
of problems, including perceptual grouping, object recognition and the analysis of biomedical
images.


key: chroma:document
string_value:
Colombia. His research interests are in computer vision, where he has worked on a number
of problems, including perceptual grouping, object recognition and the analysis of biomedical
images.
Jonathan T. Barron is a senior research scientist at Google, working on computer vision
and computational photography. He received a
PhD in Computer Science from the University
of California, Berkeley in 2013, where he was
advised by Jitendra Malik, and he received a
Honours BSc in Computer Science from the
University of Toronto in 2007. His research interests include computer vision, machine learning,
computational photography, shape reconstruction, and biological image analysis. He received
a National Science Foundation Graduate Research Fellowship in 2009,
and the C.V. Ramamoorthy Distinguished Research Award in 2013.
Ferran Marques received the degree in Electrical Engineering and the Ph.D. from the Universitat Polit `ecnica de Catalunya, BarcelonaT ech (UPC), where he is currently Professor at
the department of Signal Theory and Communications. In the term 2002-2004, he served
as President of the European Association for
Signal Processing (EURASIP). He has served
as Associate Editor of the IEEE Transactions
on Image Processing (2009-2012) and as Area
Editor for Signal Processing: Image Communication, Elsevier (2010-2014). In 2011, he received the Jaume Vicens


key: chroma:document
string_value:
on Image Processing (2009-2012) and as Area
Editor for Signal Processing: Image Communication, Elsevier (2010-2014). In 2011, he received the Jaume Vicens
Vives distinction for University Teaching Quality. Currently, he serves
as Dean of the Electrical Engineering School (ETSETB-TelecomBCN)
at UPC. He has published over 150 conference and journal papers, 2
books, and holds 4 international patents.
Jitendra Malik is Arthur J. Chick Professor in
the Department of Electrical Engineering and
Computer Science at the University of California
at Berkeley, where he also holds appointments
in vision science and cognitive science. He received the PhD degree in Computer Science
from Stanford University in 1985. In January
1986, he joined UC Berkeley as a faculty member in the EECS department where he served as
Chair of the Computer Science Division during
2002-2006, and of the Department of EECS during 2004-2006. Jitendra Malik’s group has worked on computer vision,
computational modeling of biological vision, computer graphics and
machine learning. Several well-known concepts and algorithms arose in
this work, such as anisotropic diffusion, normalized cuts, high dynamic
range imaging, shape contexts and poselets. According to Google
Scholar, ten of his papers have received more than a thousand citations
each. He has graduated 33 PhD students. Jitendra was awarded the


key: chroma:document
string_value:
Scholar, ten of his papers have received more than a thousand citations
each. He has graduated 33 PhD students. Jitendra was awarded the
Longuet-Higgins Award for “A Contribution that has Stood the Test of
Time” twice, in 2007 and 2008. He is a Fellow of the IEEE and the
ACM, a member of the National Academy of Engineering, and a fellow
of the American Academy of Arts and Sciences. He received the PAMI
Distinguished Researcher Award in computer vision in 2013 and the
K.S. Fu prize in 2014.


key: title
string_value:
High-for-Low and Low-for-High: Efficient Boundary Detection from Deep Object Features and its Applications to High-Level Vision


key: chroma:document
string_value:
High-for-Low and Low-for-High:
Efﬁcient Boundary Detection from Deep Object Features
and its Applications to High-Level Vision
Gedas Bertasius
University of Pennsylvania
gberta@seas.upenn.eduJianbo Shi
University of Pennsylvania
jshi@seas.upenn.eduLorenzo Torresani
Dartmouth College
lt@dartmouth.edu
Abstract
Most of the current boundary detection systems rely exclusively on low-level features, such as color and texture.
However, perception studies suggest that humans employ
object-level reasoning when judging if a particular pixel
is a boundary. Inspired by this observation, in this work
we show how to predict boundaries by exploiting objectlevel features from a pretrained object-classiﬁcation network. Our method can be viewed as a High-for-Low approach where high-level object features inform the low-level
boundary detection process. Our model achieves state-ofthe-art performance on an established boundary detection
benchmark and it is efﬁcient to run.
Additionally, we show that due to the semantic nature of


key: chroma:document
string_value:
benchmark and it is efﬁcient to run.
Additionally, we show that due to the semantic nature of
our boundaries we can use them to aid a number of highlevel vision tasks. We demonstrate that by using our boundaries we improve the performance of state-of-the-art methods on the problems of semantic boundary labeling, semantic segmentation and object proposal generation. We can
view this process as a Low-for-High scheme, where lowlevel boundaries aid high-level vision tasks.
Thus, our contributions include a boundary detection
system that is accurate, efﬁcient, generalizes well to multiple datasets, and is also shown to improve existing stateof-the-art high-level vision methods on three distinct tasks.
1. Introduction
In the vision community, boundary detection has always
been considered a low-level problem. However, psychological studies suggest that when a human observer perceives
boundaries, object level reasoning is used [13, 25, 17]. Despite these ﬁndings, most of the boundary detection methods rely exclusively on low-level color and gradient features. In this work, we present a method that uses objectlevel features to detect boundaries. We argue that using
object-level information to predict boundaries is more sim-Low-Level Task High-Level Tasks
BD SBL SS OP
ODS MF AP PI-IOU MR


key: chroma:document
string_value:
object-level information to predict boundaries is more sim-Low-Level Task High-Level Tasks
BD SBL SS OP
ODS MF AP PI-IOU MR
SotA 0.76 [27] 28.0 [11] 19.9 [11] 45.8 [5] 0.88 [31]
HFL 0.77 62.5 54.6 48.8 0.90
Table 1: Summary of results achieved by our proposed
method (H FL) and state-of-the-art methods (SotA). We provide results on four vision tasks: Boundary Detection (BD),
Semantic Boundary Labeling (SBL), Semantic Segmentation (SS), and Object Proposal (OP). The evaluation metrics
include ODS F-score for BD task, max F-score (MF) and
average precision (AP) for SBL task, per image intersection over union (PI-IOU) for SS task, and max recall (MR)
for OP task. Our method produces better results in each of
these tasks according to these metrics.
ilar to how humans reason. Our boundary detection scheme
can be viewed as a High-for-Low approach where we use
high-level object features as cues for a low-level boundary
detection process. Throughout the rest of the paper, we refer to our proposed boundaries as High-for-Low boundaries


key: chroma:document
string_value:
high-level object features as cues for a low-level boundary
detection process. Throughout the rest of the paper, we refer to our proposed boundaries as High-for-Low boundaries
(HFL).
We present an efﬁcient deep network that uses objectlevel information to predict the boundaries. Our proposed
architecture reuses features from the sixteen convolutional
layers of the network of Simonyan et al. [29], which we
refer to as VGG net. The VGG net has been trained for
object classiﬁcation, and therefore, reusing its features allows our method to utilize high-level object information to
predict H FL boundaries. In the experimental section, we
demonstrate that using object-level features produces semantically meaningful boundaries and also achieves above
state-of-the-art boundary detection accuracy.
Additionally, we demonstrate that we can successfully
apply our H FL boundaries to a number of high-level vision
tasks. We show that by using H FL boundaries we improve
the results of three existing state-of-the-art methods on the
tasks of semantic boundary labeling, semantic segmenta1arXiv:1504.06201v3  [cs.CV]  21 Sep 2015
tion and object proposal generation. Therefore, using H FL
boundaries to boost the results in high level vision tasks


key: chroma:document
string_value:
tion and object proposal generation. Therefore, using H FL
boundaries to boost the results in high level vision tasks
can be viewed as a Low-for-High scheme, where boundaries
serve as low-level cues to aid high-level vision tasks.
We present the summarized results for the boundary detection and the three mentioned high-level vision tasks in
Table 1. Speciﬁcally, we compare our proposed method and
an appropriate state-of-the-art method for that task. As the
results indicate, we achieve better results in each of the tasks
for each presented evaluation metric. We present more detailed results for each of these tasks in the later sections.
In summary, our contributions are as follows. First, we
show that using object-level features for boundary detection produces perceptually informative boundaries that outperform prior state-of-the-art boundary detection methods.
Second, we demonstrate that we can use H FL boundaries
to enhance the performance on the high-level vision tasks
of semantic boundary labeling, semantic segmentation and
object proposal. Finally, our method can detect boundaries
in near-real time. Thus, we present a boundary detection
system that is accurate, efﬁcient, and is also applicable to
high level vision tasks.
2. Related Work
Most of the contour detection methods predict boundaries based purely on color, text, or other low-level features.


key: chroma:document
string_value:
high level vision tasks.
2. Related Work
Most of the contour detection methods predict boundaries based purely on color, text, or other low-level features.
We can divide these methods into three broad categories:
spectral methods, supervised discriminative methods and
deep learning based methods.
Spectral methods formulate contour detection problem
as an eigenvalue problem. The solution to this problem is
then used to reason about the boundaries. The most successful approaches in this genre are the MCG detector [2], gPb
detector [1], PMI detector [14], and Normalized Cuts [28].
Some of the notable discriminative boundary detection
methods include sketch tokens (ST) [18], structured edges
(SE) [6] and sparse code gradients (SCG) [23]. While SCG
use supervised SVM learning [4], the latter two methods
rely on a random forest classiﬁer and models boundary detection as a classiﬁcation task.
Recently there have been attempts to apply deep learning
to the task of boundary detection. SCT [20] is a sparse coding approach that reconstructs an image using a learned dictionary and then detect boundaries. Both N4ﬁelds [10] and


key: chroma:document
string_value:
DeepNet [16] approaches use Convolutional Neural Networks (CNNs) to predict edges. N4ﬁelds rely on dictionary learning and the use of the Nearest Neighbor algorithm
within a CNN framework while DeepNet uses a traditional
CNN architecture to predict contours.
The most similar to our approach is DeepEdge [3],
which uses a multi-scale bifurcated network to perform contour detection using object-level features. However, we
show that our method achieves better results even without
1100x1100
512 
500x375...
1024 
Input Image
5504 Convolutional     Feature Maps 500x375Upsampled     Image
5504-Dimensional    Feature Vectors  Predicted  Boundaries
|{z}
Feature Interpolation & Concatenation
1100x1100x64- Candidate Contour Points
35x35x512
Shared Weights for all     Candidate Points
|{z}
Figure 1: An illustration of our architecture (best viewed in
color). First we extract a set of candidate contour points.
Then we upsample the image and feed it through 16convolutional layers pretrained for object classiﬁcation. For each
candidate point, we ﬁnd its correspondence in each of the
feature maps and perform feature interpolation. This yields


key: chroma:document
string_value:
candidate point, we ﬁnd its correspondence in each of the
feature maps and perform feature interpolation. This yields
a5504 -dimensional feature vector for each candidate point.
We feed each of these vectors to two fully connected layers
and store the predictions to produce a ﬁnal boundary map.
the complicated multi-scale and bifurcated architecture of
DeepEdge. Additionally, unlike DeepEdge, our system can
run in near-real time.
In comparison to prior approaches, we offer several contributions. First, we propose to use object-level information to predict boundaries. We argue that such an approach
leads to semantic boundaries, which are more consistent
with humans reasoning. Second, we avoid feature engineering by learning boundaries from human-annotated data.
Finally, we demonstrate excellent results for both low-level
and high-level vision tasks. For the boundary detection task,
our proposed H FL boundaries outperform all of the prior
methods according to both F-score metrics. Additionally,
we show that because H FL boundaries are based on objectlevel features, they can be used to improve performance in
the high-level vision tasks of semantic boundary labeling,
semantic segmentation, and object proposal generation.
3. Boundary Detection
In this section, we describe our proposed architecture
and the speciﬁc details on how we predict H FL boundaries


key: chroma:document
string_value:
3. Boundary Detection
In this section, we describe our proposed architecture
and the speciﬁc details on how we predict H FL boundaries
using our method. The detailed illustration of our architecture is presented in Figure 1.
Figure 2: A visualization of selected convolutional feature maps from VGG network (resized to the input image dimension).
Because VGG was optimized for an object classiﬁcation task, it produces high activation values on objects and their parts.
3.1. Selection of Candidate Contour Points
We ﬁrst extract a set of candidate contour points with
a high recall. Due to its efﬁciency and high recall performance, we use the SE edge detector [6]. In practice, we
could eliminate this step and simply try to predict boundaries at every pixel. However, selecting a set of initial candidate contour points, greatly reduces the computational cost.
Since our goal is to build a boundary detector that is both accurate and efﬁcient, we use these candidate points to speed
up the computation of our method.
3.2. Object-Level Features
After selecting candidate contour points, we up-sample
the original input image to a larger dimension (for example
11001100 ). The up-sampling is done to minimize the loss
of information due to the input shrinkage caused by pooling


key: chroma:document
string_value:
the original input image to a larger dimension (for example
11001100 ). The up-sampling is done to minimize the loss
of information due to the input shrinkage caused by pooling
at the different layers. Afterwards, we feed the up-sampled
image through 16convolutional layers of the VGG net.
We use the VGG net as our model because it has been
trained to recognize a large number of object classes (the
1000 categories of the ImageNet dataset [24]) and thus encodes object-level features that apply to many classes. To
preserve speciﬁc location information we utilize only the
16convolutional layers of the VGG net. We don’t use fully
connected layers because they don’t preserve spatial information, which is crucial for accurate boundary detection.
We visualize some of the selected convolutional maps in
Figure 2. Note the high activation values around the various objects in the images, which conﬁrms our hypothesis
that the VGG net encodes object speciﬁc information in its
convolutional feature maps.
3.3. Feature Interpolation
Similarly to [26, 12, 19], we perform feature interpolation in deep layers. After the up-sampled image passes


key: chroma:document
string_value:
3.3. Feature Interpolation
Similarly to [26, 12, 19], we perform feature interpolation in deep layers. After the up-sampled image passes
through all 16convolutional layers, for each selected candidate contour point we ﬁnd its corresponding point in the
feature maps. Due to the dimension differences in convo-lutional maps these correspondences are not exact. Thus
we perform feature interpolation by ﬁnding the four nearest
points and averaging their activation values. This is done
in each of the 5504 feature maps. Thus, this results in a
5504 -dimensional vector for each candidate point.
We note that the interpolation of convolutional feature
maps is the crucial component that enables our system to
predict the boundaries efﬁciently. Without feature interpolation, our method would need to independently process
the candidate edge points by analyzing a small image patch
around each point, as for example done in DeepEdge [3]
which feeds one patch at a time through a deep network.
However, when the number of candidate points is large
(e.g., DeepEdge considers about 15K points at each of 4 different scales), their patches overlap signiﬁcantly and thus a
large amount of computation is wasted by recalculating ﬁlter response values over the same pixels. Instead, we can


key: chroma:document
string_value:
large amount of computation is wasted by recalculating ﬁlter response values over the same pixels. Instead, we can
compute the features for all candidate points with a single
pass through the network by performing deep convolution
over the entire image (i.e., feeding the entire image rather
than one patch at a time) and then by interpolating the convolutional feature maps at the location of each candidate
edge point so as to produce its feature descriptor. Thanks to
this speedup, our method has a runtime of 1:2seconds (using a K40 GPU), which is better than the runtimes of prior
deep-learning based edge detection methods [27, 10, 16, 3].
3.4. Learning to Predict Boundaries
After performing feature interpolation, we feed the
5504 -dimensional feature vectors corresponding to each of
the candidate contour points to two fully connected layers
that are optimized to the human agreement criterion. To be
more precise, we deﬁne our prediction objective as a fraction of human annotators agreeing on the presence of the
boundary at a particular pixel. Therefore, a learning objective aims at mimicking the judgement of the human labelers.
Finally, to detect H FL boundaries, we accumulate the
predictions from the fully connected layers for each of the
candidate points and produce a boundary probability map
as illustrated in Figure 1.


key: chroma:document
string_value:
Finally, to detect H FL boundaries, we accumulate the
predictions from the fully connected layers for each of the
candidate points and produce a boundary probability map
as illustrated in Figure 1.
3.5. Implementation Details
In this section, we describe the details behind the training
procedure of our model. We use the Caffe library [15] to
implement our network architecture.
In the training stage, we freeze the weights in all of the
convolutional layers. To learn the weights in the two fully
connected layers we train our model to optimize the least
squares error of the regression criterion that we described
in the previous subsection. To enforce regularization we set
a dropout rate of 0:5in the fully connected layers.
Our training dataset includes 80Kpoints from the
BSDS500 dataset [22]. As described in the previous subsection, the labels represent the fraction of human annotators agreeing on the boundary presence. We divide the label space into four quartiles, and select an equal number of
samples for each quartile to balance the training dataset. In
addition to the training dataset, we also sample a hold-out
dataset of size 40;000. We use this for the hard-positive
mining [21] in order to reduce the number of false-negative
predictions.


key: chroma:document
string_value:
dataset of size 40;000. We use this for the hard-positive
mining [21] in order to reduce the number of false-negative
predictions.
For the ﬁrst 25epochs we train the network on the original80;000training samples. After the ﬁrst 25epochs, we
test the network on the hold-out dataset and detect false negative predictions made by our network. We then augment
the original 80;000 training samples with the false negatives and the same number of randomly selected true negatives. For the remaining 25epochs, we train the network on
this augmented dataset.
3.6. Boundary Detection Results
In this section, we present our results on the BSDS500
dataset [22], which is the most established benchmark for
boundary detection. The quality of the predicted boundaries is evaluated using three standard measures: ﬁxed contour threshold (ODS), per-image best threshold (OIS), and
average precision (AP).
We compare our approach to the state-of-the-art based
on two different sets of BSDS500 ground truth boundaries.
First, we evaluate the accuracy by matching each of the
predicted boundary pixels with the ground truth boundaries
that were annotated by anyof the human annotators. This


key: chroma:document
string_value:
First, we evaluate the accuracy by matching each of the
predicted boundary pixels with the ground truth boundaries
that were annotated by anyof the human annotators. This
set of ground truth boundaries is referred to as “any”. We
present the results for “any” ground truth boundaries in the
lower half of Table 2. As indicated by the results, H FL
boundaries outperform all the prior methods according to
both F-score measures.
Recently, there has been some criticism raised about
the procedure for boundary detection evaluation on the
BSDS500 dataset. One issue with the BSDS500 dataset
involves the so called “orphan” boundaries: the bound-Consensus GT ODS OIS AP FPS
SCG [23] 0.6 0.64 0.56 1/280
DeepNet [16] 0.61 0.64 0.61 1=5z
PMI [14] 0.61 0.68 0.56 1/900
DeepEdge [3] 0.62 0.64 0.64 1=1000z
N4-ﬁelds [10] 0.64 0.67 0.64 1=6z
HFL 0.65 0.68 0.67 5=6z
Any GT ODS OIS AP FPS


key: chroma:document
string_value:
HFL 0.65 0.68 0.67 5=6z
Any GT ODS OIS AP FPS
SE [6] 0.75 0.77 0.80 2.5
MCG [2] 0.75 0.78 0.76 1/24
N4-ﬁelds [10] 0.75 0.77 0.78 1=6z
DeepEdge [3] 0.75 0.77 0.81 1=1000z
MSC [30] 0.76 0.78 0.79 DeepContour [27] 0.76 0.77 0.8 1=30z
HFL 0.77 0.79 0.8 5=6z
Table 2: Boundary detection results on BSDS500 benchmark. Upper half of the table illustrates the results for “consensus” ground-truth criterion while the lower half of the
table depicts the results for “any” ground-truth criterion. In
both cases, our method outperforms all prior methods according to both ODS (optimal dataset scale) and OIS (optimal image scale) metrics. We also report the run-time of
our method (zGPU time) in the FPS column (frames per
second), which shows that our algorithm is faster than prior
approaches based on deep learning [27, 10, 16, 3].


key: chroma:document
string_value:
second), which shows that our algorithm is faster than prior
approaches based on deep learning [27, 10, 16, 3].
Figure 5: Qualitative results on the BSDS benchmark. The
ﬁrst column of images represent input images. The second
column illustrates SE [6], while the third column depicts
HFL boundaries. Notice that SE boundaries are predicted
with low conﬁdence if there is no signiﬁcant change in color
between the object and the background. Instead, because
our model is deﬁned in terms of object-level features, it can
predict object boundaries with high conﬁdence even if there
is no signiﬁcant color variation in the scene.
aries that are marked by only one or two human annotators. These “orphan” boundaries comprise around 30% of
BSDS500 dataset but most of them are considered uninformative. However, the standard evaluation benchmark rewards the methods that predict these boundaries. To resolve
500 1000 1500 2000 2500 3000 3500 4000 4500 5000 5500024x 10−3Weight Magnitude
Feature Map Number (Sorted in the Order from Early to Deep Layers)Figure 6: We train a linear regression model and visualize
its weight magnitudes in order to understand which features
are used most heavily in the boundary prediction (this linear


key: chroma:document
string_value:
its weight magnitudes in order to understand which features
are used most heavily in the boundary prediction (this linear
regression is used only for the visualization purposes and
not for the accuracy analysis). Note how heavily weighted
features lie in the deepest layers of the network, i.e., the layers that are most closely associated with object information.
this issue we also evaluate our H FL boundaries on the so
called “consensus” set of ground truth boundaries. These
“consensus” boundaries involve only boundaries that are
marked by allof the human annotators and hence, are considered perceptually meaningful. In the upper half of Table 2, we present the results achieved by our method on the
“consensus” set of the ground truth boundaries. Our H FL
boundaries outperform or tie all the prior methods in each
of the three evaluation metrics, thus suggesting that H FL
boundaries are similar to the boundaries that humans annotated. We also report the runtimes in Table 2 and note that
our method runs faster than previous deep-learning based
edge detection systems [27, 10, 16, 3].
Our proposed model computes a highly nonlinear function of the 5504-dimensional feature vector of each candidate point. Thus, it is difﬁcult to assess which features are
used most heavily by our edge predictor. However, we can


key: chroma:document
string_value:
used most heavily by our edge predictor. However, we can
gain a better insight by replacing the nonlinear function with
a simple linear model. In Fig. 6 we show the weight magnitudes of a simple linear regression model (we stress that
this linear model is used only for feature visualization purposes). From this Figure, we observe that many important
features are located in the deepest layers of the VGG network. As shown in [7], these layers encode high-level object information, which conﬁrms our hypothesis that highlevel information is useful for boundary detection.
Finally, we present some qualitative results achieved by
our method in Figure 5. These examples illustrate the effective advantage that H FL boundaries provide over another
state-of-the-art edge detection system, the SE system [6].
Speciﬁcally, observe the parts of the image where there is a
boundary that separates an object from the background but
where the color change is pretty small. Notice that because
the SE boundary detection is based on low-level color and
texture features, it captures these boundaries with very low
conﬁdence. In comparison, because H FL boundaries rely
on object-level features, it detects these boundaries with
high conﬁdence.4. High-Level Vision Applications
In this section, we describe our proposed Low-for-High


key: chroma:document
string_value:
high conﬁdence.4. High-Level Vision Applications
In this section, we describe our proposed Low-for-High
pipeline: using low-level boundaries to aid a number of
high-level vision tasks. We focus on the tasks of semantic
boundary labeling, semantic segmentation and object proposal generation. We show that using H FL boundaries improves the performance of state-of-the-art methods in each
of these high-level vision tasks.
4.1. Semantic Boundary Labeling
The task of semantic boundary labeling requires not only
to predict the boundaries but also to associate a speciﬁc
object class to each of the boundaries. This implies that
given our predicted boundaries we also need to label them
with object class information. We approach this problem
by adopting the ideas from the recent work on Fully Convolutional Networks (FCN) [19]. Given an input image,
we concurrently feed it to our boundary-predicting network
(described in Section 3), and also through the FCN that was
pretrained for 20Pascal VOC classes and the background
class. While our proposed network produces H FL boundaries, the FCN model predicts class probabilities for each
of the pixels. We can then merge the two output maps as
follows. For a given boundary point we consider a 99


key: chroma:document
string_value:
of the pixels. We can then merge the two output maps as
follows. For a given boundary point we consider a 99
grid around that point from each of the 21FCN object-class
probability maps. We calculate the maximum value inside
each grid, and then label the boundary at a given pixel with
the object-class that corresponds to the maximum probability across these 21maps. We apply this procedure for each
of the boundary points, in order to associate object-class
labels to the boundaries. Note that we consider the grids
around the boundary pixel because the output of the FCN
has a poor localization, and considering the grids rather than
individual pixels leads to higher accuracy.
We can also merge H FL boundaries with the state-ofthe-art DeepLab-CRF segmentation [5] to obtain higher accuracy. We do this in a similar fashion as just described.
First, around a given boundary point we extract a 99
grid from the DeepLab-CRF segmentation. We then compute the mode value in the grid (excluding the background
class), and use the object-class corresponding to the mode
value as a label for the given boundary point. We do this for
each of the boundary points. By merging H FL boundaries
and the output of FCN or DeepLab-CRF, we get semantic


key: chroma:document
string_value:
each of the boundary points. By merging H FL boundaries
and the output of FCN or DeepLab-CRF, we get semantic
boundaries that are highly localized and also contain objectspeciﬁc information.
4.1.1 Semantic Boundary Labeling Results
In this section, we present semantic boundary labeling results on the SBD dataset [11], which includes ground truth
boundaries that are also labeled with one of 20Pascal VOC
classes. The boundary detection accuracy for each class is
Method (Metric) aero bike bird boat bottle bus car cat chair cow table dog horse mbike person plant sheep sofa train tv mean
InvDet (MF) 42.6 49.5 15.7 16.8 36.7 43.0 40.8 22.6 18.1 26.6 10.2 18.0 35.2 29.4 48.2 14.3 26.8 11.2 22.2 32.0 28.0
HFL-FC8 (MF) 71.6 59.6 68.0 54.1 57.2 68.0 58.8 69.3 43.3 65.8 33.3 67.9 67.5 62.2 69.0 43.8 68.5 33.9 57.7 54.8 58.7


key: chroma:document
string_value:
HFL-CRF (MF) 73.9 61.4 74.6 57.2 58.8 70.4 61.6 71.9 46.5 72.3 36.2 71.1 73.0 68.1 70.3 44.4 73.2 42.6 62.4 60.1 62.5
InvDet (AP) 38.4 29.6 9.6 9.9 24.2 33.6 31.3 17.3 10.7 16.4 3.7 12.1 28.5 20.4 45.7 7.6 16.1 5.7 14.6 22.7 19.9
HFL-FC8 (AP) 66.0 50.7 58.9 40.6 47.1 62.9 51.0 59.0 25.6 54.6 15.3 57.8 57.3 55.9 62.2 27.5 55.6 18.0 50.1 40.6 47.8
HFL-CRF (AP) 71.2 55.2 69.3 45.7 48.9 71.1 56.8 65.7 29.1 65.9 17.7 64.5 68.3 64.7 65.9 29.1 66.5 25.7 60.0 49.8 54.6


key: chroma:document
string_value:
Table 3: Results of semantic boundary labeling on the SBD benchmark using the Max F-score (MF) and Average Precision
(AP) metrics. Our method (H FL) outperforms Inverse Detectors [11] for all 20categories according to both metrics. Note
that using the CRF output to label the boundaries produces better results than using the outputs from the FC8 layer of FCN.
evaluated using the maximum F-score (MF), and average
precision (AP) measures.
Labeling boundaries with the semantic object information is a novel and still relatively unexplored problem.
Therefore, we found only one other approach (Inverse Detectors) that tried to tackle this problem [11]. The basic idea behind Inverse Detectors consists of several steps.
First, generic boundaries in the image are detected. Then,
a number of object proposal boxes are generated. These
two sources of information are then used to construct the
features. Finally, a separate classiﬁer is used to label the
boundaries with the object-speciﬁc information.
Table 3 shows that our approach signiﬁcantly outperforms Inverse Detectors according to both the maximum Fscore and the average precision metrics for all twenty categories. As described in Section 4.1 we evaluate the two


key: chroma:document
string_value:
variants of our method. Denoted by H FL-FC8 is the variant for which we label H FL boundaries with the outputs
from the last layer (FC8) of the pretrained FCN. We denote
with H FL-CRF the result of labeling our boundaries with
the output from the DeepLab-CRF [5]. Among these two
variants, we show that the latter one produces better results.
This is expected since the CRF framework enforces spatial
coherence in the semantic segments.
In Figure 7, we present some of the qualitative results
produced by our method. We note that even with multiple
objects in the image, our method successfully recognizes
and localizes boundaries of each of the classes.
4.2. Semantic Segmentation
For the semantic segmentation task, we propose to
enhance the DeepLab-CRF [5] with our predicted H FL
boundaries. DeepLab-CRF is a system comprised of a Fully
Convolutional Network (described in Section 4.1) and a
dense CRF applied on top of FCN predictions.
Speciﬁcally, in the CRF, the authors propose to use
a Gaussian kernel and a bilateral term including position
and color terms as the CRF features (see [5]). While in
most cases the proposed scheme works well, DeepLab-CRF


key: chroma:document
string_value:
a Gaussian kernel and a bilateral term including position
and color terms as the CRF features (see [5]). While in
most cases the proposed scheme works well, DeepLab-CRF
sometimes produces segmentations that are not spatially coherent, particularly for images containing small object reFigure 7: A visualization of the predicted semantic boundary labels. Images in the ﬁrst column are input examples.
Columns two and three show semantic H FL boundaries of
different object classes. Note that even with multiple objects appearing simultaneously, our method outputs precise
semantic boundaries.
gions.
We propose to address this issue by adding features
based on our predicted H FL boundaries in the CRF framework. Note that we use predicted boundaries from Section 3
and not the boundaries labeled with the object information
that we obtained in Section 4.1. We use the Normalized
Cut [28] framework to generate our features.
First, we construct a pixel-wise afﬁnity matrix Wusing
our H FL boundaries. We measure the similarity between
two pixels as:
Wij= exp (max
p2ijfM(p)2
2g)
whereWijrepresents the similarity between pixels iand
j,pdenotes the boundary point along the line segment ij
connecting pixels iandj,Mdepicts the magnitude of the


key: chroma:document
string_value:
j,pdenotes the boundary point along the line segment ij
connecting pixels iandj,Mdepicts the magnitude of the
boundary at pixel p, anddenotes the smoothness parameter, which is usually set to 14% of the maximum boundary
value in the image.
The intuitive idea is that two pixels are similar (i.e.
Wij= 1) if there is no boundary crossing the line connecting these two pixels (i.e. M(p) = 08p2ij) or if the
boundary strength is low. We note that it is not necessary
to build a full afﬁnity matrix W. We build a sparse afﬁnMetric Method (Dataset) aero bike bird boat bottle bus car cat chair cow table dog horse mbike person plant sheep sofa train tv mean
PP-IOUDL-CRF (VOC) 78.6 41.1 83.5 75.3 72.9 83.1 76.6 80.8 37.8 72.1 66.5 64.7 65.8 75.7 80.5 34.4 75.9 47.4 86.6 77.9 68.9


key: chroma:document
string_value:
DL-CRF+H FL (VOC) 77.9 41.2 83.1 74.4 73.2 85.5 76.1 80.6 35.7 71.0 66.6 64.3 65.9 75.2 80.2 32.8 75.2 47.0 87.1 77.9 68.5
DL-CRF (SBD) 74.2 68.0 81.9 64.6 71.8 86.3 78.3 84.3 41.6 78.0 49.9 82.0 78.5 77.1 80.1 54.3 75.6 49.8 79.5 70.1 71.4
DL-CRF+H FL (SBD) 75.1 69.2 81.6 64.8 71.3 86.4 78.1 84.1 41.2 77.8 50.4 81.6 78.2 78.5 80.7 53.8 74.9 49.1 79.5 70.4 71.4
PI-IOUDL-CRF (VOC) 46.1 28.0 48.5 54.5 45.5 57.6 34.1 47.3 19.5 61.4 41.6 42.5 34.4 61.8 62.1 22.1 50.5 41.0 61.2 31.9 44.6


key: chroma:document
string_value:
DL-CRF+H FL (VOC) 47.5 27.6 50.4 63.5 47.7 57.9 38.7 47.2 21.1 57.3 41.2 43.7 36.0 66.4 61.1 21.3 53.9 42.1 70.9 34.6 46.5
DL-CRF (SBD) 59.4 36.5 58.0 38.6 32.0 58.1 44.7 59.6 25.8 51.8 28.1 59.0 46.9 50.3 61.8 22.2 45.9 33.4 62.1 41.0 45.8
DL-CRF+H FL (SBD) 63.4 42.5 58.4 41.3 32.5 61.2 45.7 61.4 28.4 55.5 31.5 61.4 51.8 54.6 62.1 24.9 52.6 34.2 67.1 45.1 48.8
Table 4: Semantic segmentation results on the SBD and VOC 2007 datasets. We measure the results according to PP-IOU
(per pixel) and PI-IOU (per image) evaluation metrics. We denote the original DeepLab-CRF system and our proposed


key: chroma:document
string_value:
(per pixel) and PI-IOU (per image) evaluation metrics. We denote the original DeepLab-CRF system and our proposed
modiﬁcation as DL-CRF and DL-CRF+H FL, respectively. According to the PP-IOU metric, our proposed features (DLCRF+H FL) yield almost equivalent results as the original DeepLab-CRF system. However, based on PI-IOU metric, our
proposed features improve the mean accuracy by 3%and1:9%on SBD and VOC 2007 datasets respectively.
ity matrix connecting every pair of pixels iandjthat have
distance 5or less from each other.
After building a boundary-based afﬁnity matrix Wwe
setDii=P
i6=jWijand compute eigenvectors vof the
generalized eigenvalue system:
(DW)v=Dv
We then resize the eigenvectors vto the original image
dimensions, and use them as additional features to the CRF
part of DeepLab-CRF system. In our experiments, we use
the16eigenvectors corresponding to the smallest eigenvalues, which results in 16extra feature channels.
Note that the eigenvectors contain soft segmentation information. Because H FL boundaries predict object-level


key: chroma:document
string_value:
Note that the eigenvectors contain soft segmentation information. Because H FL boundaries predict object-level
contours with high conﬁdence, the eigenvectors often capture regions corresponding to objects. We visualize a few
selected eigenvectors in Figure 8. In the experimental section, we demonstrate that our proposed features make the
output produced by DeepLab-CRF more spatially coherent
and improve the segmentation accuracy according to one of
the metrics.
We also note that our proposed features are applicable to any generic method that incorporates CRF. For instance, even if DeepLab-CRF used an improved DeepLab
network architecture, our features would still be beneﬁcial
because they contribute directly to the CRF part and not the
DeepLab network part of the system.
4.2.1 Semantic Segmentation Results
In this section, we present semantic segmentation results
on the SBD [11] and also Pascal VOC 2007 [8] datasets,
which both provide ground truth segmentations for 20Pascal VOC classes. We evaluate the results in terms of two
metrics. The ﬁrst metric measures the accuracy in terms of
pixel intersection-over-union averaged per pixel (PP-IOU)
across the 20 classes. According to this metric, the accuracy
is computed on a per pixel basis. As a result, the images that


key: chroma:document
string_value:
across the 20 classes. According to this metric, the accuracy
is computed on a per pixel basis. As a result, the images that
contain large object regions are given more importance.
Figure 8: In this ﬁgure, the ﬁrst column depicts an input
image while the second and third columns illustrate two selected eigenvectors for that image. The eigenvectors contain
soft segmentation information. Because H FL boundaries
capture object-level boundaries, the resulting eigenvectors
primarily segment regions corresponding to the objects.
We observe that while DeepLab-CRF works well on the
images containing large object regions, it produces spatially
disjoint outputs for the images with smaller and object regions (see Figure 9). This issue is often being overlooked,
because according to the PP-IOU metric, the images with
large object regions are given more importance and thus
contribute more to the accuracy. However, certain applications may require accurate segmentation of small objects.
Therefore, in addition to PP-IOU, we also consider the PIIOU metric (pixel intersection-over-union averaged per image across the 20 classes), which gives equal weight to each
of the images.
For both of the metrics we compare the semantic segmentation results of a pure DeepLab-CRF [5] and also a


key: chroma:document
string_value:
of the images.
For both of the metrics we compare the semantic segmentation results of a pure DeepLab-CRF [5] and also a
modiﬁcation of DeepLab-CRF with our proposed features
added to the CRF framework. We present the results for
both of the metrics in Table 4.
Based on these results, we observe that according to
the ﬁrst metric (PP-IOU), our proposed features yield almost equivalent results as the original DeepLab-CRF system. However, according to the second metric (PI-IOU) our
features yield an average improvement of 3%and1:9%in
Figure 9: An illustration of the more challenging semantic
segmentation examples. The ﬁrst column depicts the predictions achieved by DeepLab-CRF, while the second column illustrates the results after adding our proposed features to the CRF framework. The last column represents
ground truth segmentations. Notice how our proposed features render the predicted semantic segments more spatially
coherent and overall more accurate.
SBD and VOC 2007 datasets respectively.
We also visualize the qualitative results produced by both
approaches in Figure 9. Notice how our proposed features
make the segmentations look smoother relative to the segmentations produced by the original DeepLab-CRF system.
Once again, we want to stress that our H FL features


key: chroma:document
string_value:
make the segmentations look smoother relative to the segmentations produced by the original DeepLab-CRF system.
Once again, we want to stress that our H FL features
are applicable to any method that uses the CRF. Therefore,
based on the results presented in this section, we believe that
our proposed features could be beneﬁcial in a wide array of
problems that involve the use of the CRF framework.
4.3. Object Proposals
Finally, we show that our method produces object-level
boundaries that can be successfully exploited in an object
proposal scheme. Speciﬁcally we adopt the EdgeBoxes approach [31], which can be applied to any generic boundaries to produce a list of object proposal boxes. The original EdgeBoxes method uses SE boundaries to generate the
boxes. However, SE boundaries are predicted using lowlevel color and texture features, rather than object-level
features. Thus, here we validate the hypothesis that the
EdgeBoxes proposals can be improved by replacing the SE
boundaries with our H FL boundaries.
4.3.1 Object Proposal Results
In this section, we present object proposal results on the
Pascal VOC 2012 dataset [9]. We evaluate the quality of
bounding-box proposals according to three metrics: area
under the curve (AUC), the number of proposals needed to


key: chroma:document
string_value:
Pascal VOC 2012 dataset [9]. We evaluate the quality of
bounding-box proposals according to three metrics: area
under the curve (AUC), the number of proposals needed to
reach recall of 75%, and the maximum recall over 5000 object bounding-boxes. Additionally, we compute the accuracy for each of the metrics for three different intersectionMethodIoU 0.65 IoU 0.7 IoU 0.75
AUC N@75% Recall AUC N@75% Recall AUC N@75% Recall
SE 0:52 413 0.93 0:47 658 0.88 0.41 inf 0:75
HFL 0.53 365 0.95 0.48 583 0.9 0.41 2685 0.77
Table 5: Comparison of object proposal results. We
compare the quality of object proposals using Structured
Edges [6] and H FL boundaries. We evaluate the performance for three different IOU values and demonstrate that
using H FL boundaries produces better results for each evaluation metric and for each IOU value.
over union (IOU) values: 0:65;0:7, and 0:75. We present
these results in Table 5. As described in Section 4.3, we use
EdgeBoxes [31], a package that uses generic boundaries, to
generate object proposals. We compare the quality of the


key: chroma:document
string_value:
EdgeBoxes [31], a package that uses generic boundaries, to
generate object proposals. We compare the quality of the
generated object proposals when using SE boundaries and
HFL boundaries. We demonstrate that for each IOU value
and for each of the three evaluation metrics, H FL boundaries produce better or equivalent results. This conﬁrms our
hypothesis that H FL boundaries can be used effectively for
high-level vision tasks such as generating object proposals.
5. Conclusions
In this work, we presented an efﬁcient architecture that
uses object-level information to predict semantically meaningful boundaries. Most prior edge detection methods rely
exclusively on low-level features, such as color or texture, to
detect the boundaries. However, perception studies suggest
that humans employ object-level reasoning when deciding
whether a given pixel is a boundary [13, 25, 17]. Thus,
we propose a system that focuses on the semantic objectlevel cues rather than low level image information to detect
the boundaries. For this reason we refer to our boundary
detection scheme as a High-for-Low approach, where highlevel object features inform the low-level boundary detection process. In this paper we demonstrated that our proposed method produces boundaries that accurately separate
objects and the background in the image and also achieve
higher F-score compared to any prior work.


key: chroma:document
string_value:
objects and the background in the image and also achieve
higher F-score compared to any prior work.
Additionally, we showed that, because H FL boundaries
are based on object-level features, they can be employed to
aid a number of high level vision tasks in a Low-for-High
fashion. We use our boundaries to boost the accuracy of
state-of-the-art methods on the high-level vision tasks of semantic boundary labeling, semantic segmentation, and object proposals generation. We show that using H FL boundaries leads to better results in each of these tasks.
To conclude, our boundary detection method is accurate,
efﬁcient, applicable to a variety of datasets, and also useful
for multiple high-level vision tasks. We plan to release the
source code for H FL upon the publication of the paper .
6. Acknowledgements
We thank Mohammad Haris Baig for the suggestions and
help with the software. This research was funded in part by
NSF award CNS-1205521.
References
[1] P. Arbelaez, M. Maire, C. Fowlkes, and J. Malik. Contour
detection and hierarchical image segmentation. IEEE Trans.
Pattern Anal. Mach. Intell. , 33(5):898–916, May 2011. 2


key: chroma:document
string_value:
detection and hierarchical image segmentation. IEEE Trans.
Pattern Anal. Mach. Intell. , 33(5):898–916, May 2011. 2
[2] P. Arbelaez, J. Pont-Tuset, J. Barron, F. Marqués, and J. Malik. Multiscale combinatorial grouping. In Computer Vision
and Pattern Recognition (CVPR) , 2014. 2, 4
[3] G. Bertasius, J. Shi, and L. Torresani. Deepedge: A multiscale bifurcated deep network for top-down contour detection. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , June 2015. 2, 3, 4, 5
[4] C. J. C. Burges. A tutorial on support vector machines for
pattern recognition. Data Mining and Knowledge Discovery ,
2:121–167, 1998. 2
[5] L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L.
Yuille. Semantic image segmentation with deep convolutional nets and fully connected crfs. CoRR , abs/1412.7062,
2014. 1, 5, 6, 7


key: chroma:document
string_value:
2014. 1, 5, 6, 7
[6] P. Dollár and C. L. Zitnick. Fast edge detection using structured forests. PAMI , 2015. 2, 3, 4, 5, 8
[7] J. Donahue, Y . Jia, O. Vinyals, J. Hoffman, N. Zhang,
E. Tzeng, and T. Darrell. Decaf: A deep convolutional
activation feature for generic visual recognition. CoRR ,
abs/1310.1531, 2013. 5
[8] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn,
and A. Zisserman. The PASCAL Visual Object Classes
Challenge 2007 (VOC2007) Results. http://www.pascalnetwork.org/challenges/VOC/voc2007/workshop/index.html.
7
[9] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn,
and A. Zisserman. The PASCAL Visual Object Classes
Challenge 2012 (VOC2012) Results. http://www.pascalnetwork.org/challenges/VOC/voc2012/workshop/index.html.
8


key: chroma:document
string_value:
Challenge 2012 (VOC2012) Results. http://www.pascalnetwork.org/challenges/VOC/voc2012/workshop/index.html.
8
[10] Y . Ganin and V . S. Lempitsky. N4-ﬁelds: Neural network
nearest neighbor ﬁelds for image transforms. ACCV , 2014.
2, 3, 4, 5
[11] B. Hariharan, P. Arbelaez, L. Bourdev, S. Maji, and J. Malik.
Semantic contours from inverse detectors. In International
Conference on Computer Vision (ICCV) , 2011. 1, 5, 6, 7
[12] B. Hariharan, P. A. Arbeláez, R. B. Girshick, and J. Malik.
Hypercolumns for object segmentation and ﬁne-grained localization. CoRR , abs/1411.5752, 2014. 3
[13] P.-J. Hsieh, E. Vul, and N. Kanwisher. Recognition alters the
spatial pattern of fmri activation in early retinotopic cortex.
Journal of Neurophysiology , 103(3):1501–1507, 2010. 1, 8


key: chroma:document
string_value:
spatial pattern of fmri activation in early retinotopic cortex.
Journal of Neurophysiology , 103(3):1501–1507, 2010. 1, 8
[14] P. Isola, D. Zoran, D. Krishnan, and E. H. Adelson. Crisp
boundary detection using pointwise mutual information. In
ECCV , 2014. 2, 4[15] Y . Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell. Caffe: Convolutional architecture for fast feature embedding. arXiv preprint
arXiv:1408.5093 , 2014. 4
[16] J. J. Kivinen, C. K. Williams, N. Heess, and D. Technologies. Visual boundary prediction: A deep neural prediction
network and quality dissection. AISTATS , 1(2):9, 2014. 2, 3,
4, 5
[17] Z. Kourtzi and N. Kanwisher. Representation of perceived
object shape by the human lateral occipital complex. Science, 293:1506–1509, 2001. 1, 8


key: chroma:document
string_value:
object shape by the human lateral occipital complex. Science, 293:1506–1509, 2001. 1, 8
[18] J. Lim, C. L. Zitnick, and P. Dollár. Sketch tokens: A learned
mid-level representation for contour and object detection. In
CVPR , 2013. 2
[19] J. Long, E. Shelhamer, and T. Darrell. Fully convolutional
networks for semantic segmentation. CVPR (to appear) ,
Nov. 2015. 3, 5
[20] M. Maire, S. X. Yu, and P. Perona. Reconstructive sparse
code transfer for contour detection and semantic labeling. In
Asian Conference on Computer Vision (ACCV) , 2014. 2
[21] T. Malisiewicz, A. Gupta, and A. A. Efros. Ensemble of
exemplar-svms for object detection and beyond. In ICCV ,
2011. 4
[22] D. Martin, C. Fowlkes, D. Tal, and J. Malik. A database
of human segmented natural images and its application to
evaluating segmentation algorithms and measuring ecological statistics. In Proc. 8th Int’l Conf. Computer Vision , volume 2, pages 416–423, July 2001. 4


key: chroma:document
string_value:
evaluating segmentation algorithms and measuring ecological statistics. In Proc. 8th Int’l Conf. Computer Vision , volume 2, pages 416–423, July 2001. 4
[23] X. Ren and L. Bo. Discriminatively Trained Sparse Code
Gradients for Contour Detection. In Advances in Neural Information Processing Systems , December 2012. 2, 4
[24] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh,
S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein,
A. C. Berg, and L. Fei-Fei. ImageNet Large Scale Visual
Recognition Challenge, 2014. 3
[25] J. L. Sanguinetti, J. J. Allen, and M. A. Peterson. The ground
side of an object perceived as shapeless yet processed for
semantics. Psychological science , page 0956797613502814,
2013. 1, 8
[26] P. Sermanet, D. Eigen, X. Zhang, M. Mathieu, R. Fergus,
and Y . LeCun. Overfeat: Integrated recognition, localization and detection using convolutional networks. CoRR ,
abs/1312.6229, 2013. 3


key: chroma:document
string_value:
and Y . LeCun. Overfeat: Integrated recognition, localization and detection using convolutional networks. CoRR ,
abs/1312.6229, 2013. 3
[27] W. Shen, X. Wang, Y . Wang, X. Bai, and Z. Zhang. Deepcontour: A deep convolutional feature learned by positivesharing loss for contour detection. June 2015. 1, 3, 4, 5
[28] J. Shi and J. Malik. Normalized cuts and image segmentation. IEEE Transactions on Pattern Analysis and Machine
Intelligence , 22:888–905, 1997. 2, 6
[29] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. CoRR ,
abs/1409.1556, 2014. 1
[30] A. Sironi, V . Lepetit, and P. Fua. Multiscale centerline detection by learning a scale-space distance transform. June 2014.
4
[31] C. L. Zitnick and P. Dollár. Edge boxes: Locating object
proposals from edges. In ECCV , 2014. 1, 8


key: chroma:document
string_value:
Semantic Segmentation with Boundary Neural Fields
Gedas Bertasius
University of Pennsylvania
gberta@seas.upenn.eduJianbo Shi
University of Pennsylvania
jshi@seas.upenn.eduLorenzo Torresani
Dartmouth College
lt@dartmouth.edu
Abstract
The state-of-the-art in semantic segmentation is currently represented by fully convolutional networks (FCNs).
However, FCNs use large receptive ﬁelds and many pooling layers, both of which cause blurring and low spatial
resolution in the deep layers. As a result FCNs tend to produce segmentations that are poorly localized around object
boundaries. Prior work has attempted to address this issue
in post-processing steps, for example using a color-based
CRF on top of the FCN predictions. However, these approaches require additional parameters and low-level features that are difﬁcult to tune and integrate into the original
network architecture. Additionally, most CRFs use colorbased pixel afﬁnities, which are not well suited for semantic
segmentation and lead to spatially disjoint predictions.
To overcome these problems, we introduce a Boundary


key: chroma:document
string_value:
segmentation and lead to spatially disjoint predictions.
To overcome these problems, we introduce a Boundary
Neural Field (BNF), which is a global energy model integrating FCN predictions with boundary cues. The boundary information is used to enhance semantic segment coherence and to improve object localization. Speciﬁcally, we
ﬁrst show that the convolutional ﬁlters of semantic FCNs
provide good features for boundary detection. We then employ the predicted boundaries to deﬁne pairwise potentials
in our energy. Finally, we show that our energy decomposes semantic segmentation into multiple binary problems,
which can be relaxed for efﬁcient global optimization. We
report extensive experiments demonstrating that minimization of our global boundary-based energy yields results superior to prior globalization methods, both quantitatively
as well as qualitatively.
1. Introduction
The recent introduction of fully convolutional networks
(FCNs) [22] has led to signiﬁcant quantitative improvements on the task of semantic segmentation. However, despite their empirical success, FCNs suffer from some limitations. Large receptive ﬁelds in the convolutional layers and
the presence of pooling layers lead to blurring and segmentation predictions at a signiﬁcantly lower resolution than the


key: chroma:document
string_value:
the presence of pooling layers lead to blurring and segmentation predictions at a signiﬁcantly lower resolution than the
Figure 1: Examples illustrating shortcomings of prior semantic segmentation methods: the second column shows
results obtained with a FCN [22], while the third column
shows the output of a Dense-CRF applied to FCN predictions [19, 7]. Segments produced by FCN are blob-like and
are poorly localized around object boundaries. Dense-CRF
produces spatially disjoint object segments due to the use of
a color-based pixel afﬁnity function that is unable to measure semantic similarity between pixels.
original image. As a result, their predicted segments tend to
be blobby and lack ﬁne object boundary details. We report
in Fig. 1 some examples illustrating typical poor localization of objects in the outputs of FCNs.
Recently, Chen at al. [7] addressed this issue by applying a Dense-CRF post-processing step [19] on top of coarse
FCN segmentations. However, such an approach introduces
several problems of its own. First, the Dense-CRF adds new
parameters that are difﬁcult to tune and integrate into the
original network architecture. Additionally, most methods


key: chroma:document
string_value:
parameters that are difﬁcult to tune and integrate into the
original network architecture. Additionally, most methods
based on CRFs or MRFs use low-level pixel afﬁnity functions, such as those based on color. These low-level afﬁnities often fail to capture semantic relationships between objects and lead to poor segmentation results (see last column
in Fig. 1).
We propose to address these shortcomings by means of
a Boundary Neural Field (BNF), an architecture that employs a single semantic segmentation FCN to predict semantic boundaries and then use them to produce semantic
segmentation maps via a global optimization. We demonstrate that even though the semantic segmentation FCN has
not been optimized to detect boundaries, it provides good
1arXiv:1511.02674v2  [cs.CV]  24 May 2016
...
Convolutional Feature Maps RGB Input Image
Segmentation Mask in       Softmax Layer
   Minimize Global Energy
Boundaries
|{z}...
wn()++()w1
Xi✓i(zi)+Xij✓ij(zi,zj)
Linear Combination of Feature MapsFigure 2: The architecture of our system (best viewed in
color). We employ a semantic segmentation FCN [7] for
two purposes: 1) to obtain semantic segmentation unaries


key: chroma:document
string_value:
color). We employ a semantic segmentation FCN [7] for
two purposes: 1) to obtain semantic segmentation unaries
for our global energy; 2) to compute object boundaries.
Speciﬁcally, we deﬁne semantic boundaries as a linear combination of these feature maps (with a sigmoid function applied on top of the sum) and learn individual weights corresponding to each convolutional feature map. We integrate
this boundary information in the form of pairwise potentials
(pixel afﬁnities) for our energy model.
features for boundary detection. Speciﬁcally, the contributions of our work are as follows:
We show that semantic boundaries can be expressed
as a linear combination of interpolated convolutional
feature maps inside an FCN. We introduce a boundary
detection method that exploits this intuition to predict
object boundaries with accuracy superior to the statethe-of-art.
We demonstrate that boundary-based pixel afﬁnities
are better suited for semantic segmentation than the
commonly used color afﬁnity functions.
Finally, we introduce a new global energy that decomposes semantic segmentation into multiple binary
problems and relaxes the integrality constraint. We
show that minimizing our proposed energy yields better qualitative and quantitative results relative to traditional globalization models such as MRFs or CRFs.


key: chroma:document
string_value:
problems and relaxes the integrality constraint. We
show that minimizing our proposed energy yields better qualitative and quantitative results relative to traditional globalization models such as MRFs or CRFs.
2. Related Work
Boundary Detection. Spectral methods comprise one
of the most prominent categories for boundary detection.
In a typical spectral framework, one formulates a generalized eigenvalue system to solve a low-level pixel groupingproblem. The resulting eigenvectors are then used to predict the boundaries. Some of the most notable approaches
in this genre are MCG [2], gPb [1], PMI [17], and Normalized Cuts [29]. A weakness of spectral approaches is that
they tend to be slow as they perform a global inference over
the entire image.
To address this issue, recent approaches cast boundary
detection as a classiﬁcation problem and predict the boundaries in a local manner with high efﬁciency. The most notable examples in this genre include sketch tokens (ST) [20]
and structured edges (SE) [9], which employ fast random
forests. However, many of these methods are based on
hand-constructed features, which are difﬁcult to tune.


key: chroma:document
string_value:
forests. However, many of these methods are based on
hand-constructed features, which are difﬁcult to tune.
The issue of hand-constructed features have been recently addressed by several approaches based on deep learning, such as N4ﬁelds [11], DeepNet [18], DeepContour [27], DeepEdge [3], H FL [4] and HED [33]. All of
these methods use CNNs in some way to predict the boundaries. Whereas DeepNet and DeepContour optimize ordinary CNNs to a boundary based optimization criterion from
scratch, DeepEdge and H FL employ pretrained models to
compute boundaries. The most recent of these methods is
HED [33], which shows the beneﬁt of deeply supervised
learning for boundary detection.
In comparison to prior deep learning approaches, our
method offers several contributions. First, we exploit the inherent relationship between boundary detection and semantic segmentation to predict semantic boundaries. Specifically, we show that even though the semantic FCN has
not been explicitly trained to predict boundaries, the convolutional ﬁlters inside the FCN provide good features for
boundary detection. Additionally, unlike DeepEdge [3] and
HFL [4], our method does not require a pre-processing step
to select candidate contour points, as we predict boundaries


key: chroma:document
string_value:
HFL [4], our method does not require a pre-processing step
to select candidate contour points, as we predict boundaries
on all pixels in the image. We demonstrate that our approach allows us to achieve state-of-the-art boundary detection results according to both F-score and Average Precision metrics. Additionally, due to the semantic nature of
our boundaries, we can successfully use them as pairwise
potentials for semantic segmentation in order to improve
object localization and recover ﬁne structural details, typically lost by pure FCN-based approaches.
Semantic Segmentation. We can group most semantic segmentation methods into three broad categories. The
ﬁrst category can be described as “two-stage” approaches,
where an image is ﬁrst segmented and then each segment
is classiﬁed as belonging to a certain object class. Some
of the most notable methods that belong to this genre include [24, 6, 12, 14].
The primary weakness of the above methods is that they
are unable to recover from errors made by the segmentation
algorithm. Several recent papers [15, 10] address this issue
by proposing to use deep per-pixel CNN features and then
classify each pixel as belonging to a certain class. While
these approaches partially address the incorrect segmentation problem, they perform predictions independently on


key: chroma:document
string_value:
classify each pixel as belonging to a certain class. While
these approaches partially address the incorrect segmentation problem, they perform predictions independently on
each pixel. This leads to extremely local predictions, where
the relationships between pixels are not exploited in any
way, and thus the resulting segmentations may be spatially
disjoint.
The third and ﬁnal group of semantic segmentation
methods can be viewed as front-to-end schemes where segmentation maps are predicted directly from raw pixels without any intermediate steps. One of the earliest examples of
such methods is the FCN introduced in [22]. This approach
gave rise to a number of subsequent related approaches
which have improved various aspects of the original semantic segmentation [7, 34, 8, 16, 21]. There have also been attempts at integrating the CRF mechanism into the network
architecture [7, 34]. Finally, it has been shown that semantic
segmentation can also be improved using additional training
data in the form of bounding boxes [8].
Our BNF offers several contributions over prior work. To
the best of our knowledge, we are the ﬁrst to present a model
that exploits the relationship between boundary detection
and semantic segmentation within a FCN framework . We
introduce pairwise pixel afﬁnities computed from semantic boundaries inside an FCN, and use these boundaries to


key: chroma:document
string_value:
and semantic segmentation within a FCN framework . We
introduce pairwise pixel afﬁnities computed from semantic boundaries inside an FCN, and use these boundaries to
predict the segmentations in a global fashion. Unlike [21],
which requires a large number of additional parameters to
learn for the pairwise potentials, our global model only
needs5Kextra parameters, which is about 3orders of
magnitudes less than the number of parameters in a typical deep convolutional network (e.g. VGG [30]). We empirically show that our proposed boundary-based afﬁnities
are better suited for semantic segmentation than color-based
afﬁnities. Additionally, unlike in [7, 34, 21], the solution to
our proposed global energy can be obtained in closed-form,
which makes global inference easier. Finally we demonstrate that our method produces better results than traditional globalization models such as CRFs or MRFs.
3. Boundary Neural Fields
In this section, we describe Boundary Neural Fields.
Similarly to traditional globalization methods, Boundary
Neural Fields are deﬁned by an energy including unary
and pairwise potentials. Minimization of the global energy yields the semantic segmentation. BNFs build both
unary and pairwise potentials from the input RGB image


key: chroma:document
string_value:
and pairwise potentials. Minimization of the global energy yields the semantic segmentation. BNFs build both
unary and pairwise potentials from the input RGB image
and then combine them in a global manner. More precisely,
the coarse segmentations predicted by a semantic FCN are
used to deﬁne the unary potentials of our BNF. Next, we
show that the convolutional feature maps of the FCN can
be used to accurately predict semantic boundaries. These
boundaries are then used to build pairwise pixel afﬁnities,
which are used as pairwise potentials by the BNF. Finally,we introduce a global energy function, which minimizes the
energy corresponding to the unary and pairwise terms and
improves the initial FCN segmentation. The detailed illustration of our architecture is presented in Figure 2. We now
explain each of these steps in more detail.
3.1. FCN Unary Potentials
To predict semantic unary potentials we employ the
DeepLab model [7], which is a fully convolutional adaptation of the VGG network [30]. The FCN consists of 16convolutional layers and 3fully convolutional layers. There are
more recent FCN-based methods that have demonstrated
even better semantic segmentation results [8, 34, 16, 21].


key: chroma:document
string_value:
more recent FCN-based methods that have demonstrated
even better semantic segmentation results [8, 34, 16, 21].
Although these more advanced architectures could be integrated into our framework to improve our unary potentials, in this work we focus on two aspects orthogonal to
this prior work: 1) demonstrating that our boundary-based
afﬁnity function is better suited for semantic segmentation
than the common color-based afﬁnities and 2) showing that
our proposed global energy achieves better qualitative and
quantitative semantic segmentation results in comparison to
prior globalization models.
3.2. Boundary Pairwise Potentials
In this section, we describe our approach for building
pairwise pixel afﬁnities using semantic boundaries. The basic idea behind our boundary detection approach is to express semantic boundaries as a function of convolutional
feature maps inside the FCN. Due to the close relationship
between the tasks of semantic segmentation and boundary
detection, we hypothesize that convolutional feature maps
from the semantic segmentation FCN can be employed as
features for boundary detection.
3.2.1 Learning to Predict Semantic Boundaries.
We propose to express semantic boundaries as a linear combination of interpolated FCN feature maps with a non-linear
function applied on top of this sum. We note that interpolation of feature maps has been successfully used in prior


key: chroma:document
string_value:
function applied on top of this sum. We note that interpolation of feature maps has been successfully used in prior
work (see e.g. [15]) in order to obtain dense pixel-level features from the low-resolution outputs of deep convolutional
layers. Here we adopt interpolation to produce pixel-level
boundary predictions. There are several advantages to our
proposed formulation. First, because we express boundaries
as a linear combination of feature maps, we only need to
learn a small number of parameters, corresponding to the
individual weight values of each feature map in the FCN.
This amounts to5Klearning parameters, which is much
smaller than the number of parameters in the entire network
(15M). In comparison, DeepEdge [3] and HFL [4] need
17M and 6M additional parameters to predict boundaries.
Furthermore, expressing semantic boundaries as a linear
combination of FCN feature maps allows us to efﬁciently
predict boundary probabilities for all pixels in the image
(we resize the FCN feature maps to the original image dimensions). This eliminates the need to select candidate
boundary points in a pre-processing stage, which was instead required in prior boundary detection work [3, 4].
Our boundary prediction pipeline can be described as


key: chroma:document
string_value:
boundary points in a pre-processing stage, which was instead required in prior boundary detection work [3, 4].
Our boundary prediction pipeline can be described as
follows. First we use use SBD segmentations [13] to optimize our FCN for semantic segmentation task. We then
treat FCN convolutional maps as features for the boundary detection task and use the boundary annotations from
BSDS 500 dataset [23] to learn the weights for each feature
map. BSDS 500 dataset contains 200training, 100validation, 200testing images, and ground truth annotations by 5
human labelers for each of these images.
To learn the weights corresponding to each convolutional
feature map we ﬁrst sample 80Kpoints from the dataset.
We deﬁne the target labels for each point as the fraction of
human annotators agreeing on that point being a boundary.
To ﬁx the issue of label imbalance (there are many more
non-boundaries than boundaries), we divide the label space
into four quartiles, and select an equal number of samples
for each quartile to balance the training dataset. Given these
sampled points, we then deﬁne our features as the values in
the interpolated convolutional feature maps corresponding
to these points. To predict semantic boundaries we weigh
each convolutional feature map by its weight, sum them up


key: chroma:document
string_value:
the interpolated convolutional feature maps corresponding
to these points. To predict semantic boundaries we weigh
each convolutional feature map by its weight, sum them up
and apply a sigmoid function on top of it. We obtain the
weights corresponding to each convolutional feature map by
minimizing the cross-entropy loss using a stochastic batch
gradient descent for 50epochs. To obtain crisper boundaries at test-time we post-process the boundary probabilities
using non-maximum suppression.
To give some intuition on how FCN feature maps contribute to boundary detection, in Fig. 3 we visualize the feature maps corresponding to the highest weight magnitudes.
It is clear that many of these maps contain highly localized
boundary information.
Boundary Detection Results Before discussing how
boundary information is integrated in our energy for semantic segmentation, here we present experimental results assessing the accuracy of our boundary detection scheme. We
tested our boundary detector on the BSDS500 dataset [23],
which is the standard benchmark for boundary detection.
The quality of the predicted boundaries is evaluated using
three standard measures: ﬁxed contour threshold (ODS),
per-image best threshold (OIS), and average precision (AP).
In Table 1 we show that our algorithm outperforms all
prior methods according to both F-score measures and the


key: chroma:document
string_value:
per-image best threshold (OIS), and average precision (AP).
In Table 1 we show that our algorithm outperforms all
prior methods according to both F-score measures and the
Average Precision metric. In Fig. 4, we also visualize our
predicted boundaries. The second column shows the pixellevel softmax output computed from the linear combination of feature maps, while the third column depicts our ﬁFigure 3: An input image and convolutional feature maps
corresponding to the largest weight magnitude values. Intuitively these are the feature maps that contribute most heavily to the task of boundary detection.
Method ODS OIS AP
SCG [25] 0.739 0.758 0.773
SE [9] 0.746 0.767 0.803
MCG [2] 0.747 0.779 0.759
N4-ﬁelds [11] 0.753 0.769 0.784
DeepEdge [3] 0.753 0.772 0.807
DeepContour [27] 0.756 0.773 0.797
HFL [4] 0.767 0.788 0.795
HED [33] 0.782 0.804 0.833
BNF 0.788 0.807 0.851


key: chroma:document
string_value:
HED [33] 0.782 0.804 0.833
BNF 0.788 0.807 0.851
Table 1: Boundary detection results on BSDS500 benchmark. Our proposed method outperforms all prior algorithms according to all three evaluation metrics.
nal boundaries after applying a non-maximum suppression
post-processing step.
We note that our predicted boundaries achieve highconﬁdence predictions around objects. This is important as
we employ these boundaries to improve semantic segmentation results, as discussed in the next subsection.
3.2.2 Constructing Pairwise Pixel Afﬁnities.
We can use the predicted boundaries to build pairwise pixel
afﬁnities. Intuitively, we declare two pixels as similar (i.e.,
likely to belong to the same segment) if there is no boundary crossing the straight path between these two pixels.
Conversely, two pixels are dissimilar if there is a boundary crossing their connecting path. The larger the boundary
magnitude of the crossed path, the more dissimilar the two
pixels should be, since a strong boundary is likely to mark
the separation of two distinct segments. Similarly to [1], we
encode this intuition with a following formulation:
wsb
ij= exp (Mij
sb) (1)


key: chroma:document
string_value:
encode this intuition with a following formulation:
wsb
ij= exp (Mij
sb) (1)
Figure 4: A ﬁgure illustrating our boundary detection results. In the second column, we visualize the raw probability output of our boundary detector. In the third column,
we present the ﬁnal boundary maps after non-maximum
suppression. While most prior methods predict the boundaries where the sharpest change in color occurs, our method
captures semantic object-level boundaries, which we subsequently use to aid semantic segmentation.
whereMijdenotes the maximum boundary value that
crosses the straight line path between pixels iandj,sbdepicts the smoothing parameter and wsb
ijdenotes the semantic
boundary-based afﬁnity between pixels iandj.
Similarly, we want to exploit high-level object information in the network to deﬁne another type of pixel similarity.
Speciﬁcally, we use object class probabilities from the softmax (SM) layer to achieve this goal. Intuitively, if pixels i
andjhave different hard segmentation labels from the softmax layer, we set their similarity ( wsm
ij) to0. Otherwise,
we compute their similarity using the following equation:
wsm
ij= exp (Dij
sm) (2)


key: chroma:document
string_value:
ij) to0. Otherwise,
we compute their similarity using the following equation:
wsm
ij= exp (Dij
sm) (2)
whereDijdenotes the difference in softmax output values corresponding to the most likely object class for pixels
iandj, andsmis a smoothing parameter. Then we can
write the ﬁnal afﬁnity measure as:
wij= exp (wsm
ij)wsb
ij (3)
We exponentiate the term corresponding to the objectlevel afﬁnity because our boundary-based afﬁnity may be
too aggressive in declaring two pixels as dissimilar. To address this issue, we increase the importance of the objectlevel afﬁnity in (3) using the exponential function. However, in the experimental results section, we demonstrate
that most of the beneﬁt from modeling pairwise potentials
comes from wsb
ijrather thanwsm
ij.We then use this pairwise pixel afﬁnity measure to build
a global afﬁnity matrix Wthat encodes relationships between pixels in the entire image. For a given pixel, we
sample10% of points in the neighborhood of radius 20
around that pixel, and store the resulting afﬁnities into W.


key: chroma:document
string_value:
sample10% of points in the neighborhood of radius 20
around that pixel, and store the resulting afﬁnities into W.
3.3. Global Inference
The last step in our proposed method is to combine semantic boundary information with the coarse segmentation
from the FCN softmax layer to produce an improved segmentation. We do this by introducing a global energy function that utilizes the afﬁnity matrix constructed in the previous section along with the segmentation from the FCN
softmax layer. Using this energy, we perform a global inference to get segmentations that are well localized around the
object boundaries and that are also spatially smooth.
Typical globalization models such as MRFs [31],
CRFs [19] or Graph Cuts [5] produce a discrete label assignment for the segmentation problem by jointly modeling a multi-label distribution and solving a non-convex optimization. The common problem in doing so is that the
optimization procedure may get stuck in local optima.
We introduce a new global energy function, which overcomes this issue and achieves better segmentation in comparison to prior globalization models. Similarly to prior
globalization approaches, our goal is to minimize the energy
corresponding to the sum of unary and pairwise potentials.
However, the key difference in our approach comes from the


key: chroma:document
string_value:
globalization approaches, our goal is to minimize the energy
corresponding to the sum of unary and pairwise potentials.
However, the key difference in our approach comes from the
relaxation of some of the constraints. Speciﬁcally, instead
of modeling our problem as a joint multi-label distribution,
we propose to decompose it into multiple binary problems,
which can be solved concurrently. This decomposition can
be viewed as assigning pixels to foreground and background
labels for each of the different object classes. Additionally,
we relax the integrality constraint. Both of these relaxations
make our problem more manageable and allow us to formulate a global energy function that is differentiable, and has
a closed form solution.
In [35], the authors introduce the idea of learning
with global and local consistency in the context of semisupervised problems. Inspired by this work, we incorporate
some of these ideas in the context of semantic segmentation. Before deﬁning our proposed global energy function,
we introduce some relevant notation.
For the purpose of illustration, suppose that we only have
two classes: foreground and background. Then we can denote an optimal continuous solution to such a segmentation
problem with variable z. To denote similarity between pixelsiandjwe usewij. Then,diindicates the degree of a


key: chroma:document
string_value:
problem with variable z. To denote similarity between pixelsiandjwe usewij. Then,diindicates the degree of a
pixeli. In graph theory, the degree of a node denotes the
number of edges incident to that node. Thus, we set the degree of a pixel to di=Pn
j=1wijfor alljexcepti6=j.
Finally, with fiwe denote an initial segmentation probaInput
 Softmax
 Dense-CRF
 BNF Boundaries
 BNF Segmentation
Figure 5: A ﬁgure illustrating semantic segmentation results. Images in columns two and three represent FCN softmax and
Dense-CRF predictions, respectively. Note that all methods use the same FCN unary potentials. Additionally, observe that
unlike FCN and Dense-CRF, our methods predicts segmentation that are both well localized around object boundaries and
that are also spatially smooth.
bility, which in our case is obtained from the FCN softmax
layer.
Using this notation, we can then formulate our global
inference objective as:
z= argmin
z
2X
idi(zifi
di)2+1
2X
ijwij(zizj)2(4)
This energy consists of two different terms. Similar to


key: chroma:document
string_value:
idi(zifi
di)2+1
2X
ijwij(zizj)2(4)
This energy consists of two different terms. Similar to
the general globalization framework, our ﬁrst term encodes
the unary energy while the second term includes the pairwise energy. We now explain the intuition behind each of
these terms. The unary term attempts to ﬁnd a segmentation
assignment ( zi) that deviates little from the initial candidate
segmentation computed from the softmax layer (denoted by
fi). Theziin the unary term is weighted by the degree di
of the pixel in order to produce larger unary costs for pixels
that have many similar pixels within the neighborhood. Instead, the pairwise term ensures that pixels that are similar
should be assigned similar zvalues. To balance the energies of the two terms we introduce a parameter and set it
to0:025throughout all our experiments.
We can also express the same global energy function in
matrix notation:
z= argmin
z
2D(zD1f)T(zD1f)+1
2zT(DW)z
(5)
where zis an1vector containing an optimal continuous assignment for all npixels, Dis a diagonal degree


key: chroma:document
string_value:
2zT(DW)z
(5)
where zis an1vector containing an optimal continuous assignment for all npixels, Dis a diagonal degree
matrix, and Wis thennpixel afﬁnity matrix. Finally, f
denotes an1vector containing the probabilities from the
softmax layer corresponding to a particular object class.
An advantage of our energy is that it is differentiable. Ifwe denote the above energy as E(z)then the derivative of
this energy can be written as follows:
@E(z)
@z=D(zD1f) + (DW)z=0 (6)
With simple algebraic manipulations we can then obtain
a closed form solution to this optimization:
z= (DW)1f (7)
where=1
1+and=
1+. In the general case where
we havekobject classes we can write the solution as:
Z= (DW)1F (8)
where Znow depicts a nkmatrix containing assignments for all kobject classes, while Fdenotesnkmatrix
with object class probabilities from softmax layer. Due to


key: chroma:document
string_value:
where Znow depicts a nkmatrix containing assignments for all kobject classes, while Fdenotesnkmatrix
with object class probabilities from softmax layer. Due to
the large size of DWit is impractical to invert it. However, if we consider an image as a graph where each pixel
denotes a vertex in the graph, we can observe that the term
DWin our optimization is equivalent to a Laplacian matrix of such graph. Since we know that a Laplacian matrix is
positive semi-deﬁnite, we can use the preconditioned conjugate gradient method [28] to solve the system in Eq. (9).
Alternatively, because our deﬁned global energy in Eq. (5)
is differentiable, we can efﬁciently solve this optimization
problem using stochastic gradient descent. We choose the
former option and solve the following system:
(DW)z=f (9)
To obtain the ﬁnal discrete segmentation, for each pixel
we assign the object class that corresponds to the largest
column value in the row of Z(note that each row in Zrepresents a single pixel in the image, and each column in Z


key: chroma:document
string_value:
we assign the object class that corresponds to the largest
column value in the row of Z(note that each row in Zrepresents a single pixel in the image, and each column in Z
Metric Inference Method RGB Afﬁnity BNF Afﬁnity
PP-IOUBelief Propagation [31] 75.4 75.6
ICM 74.2 75.8
TRWS [32] 75.9 76.7
QPBO [26] 76.9 77.2
BNF 74.6 77.6
PI-IOUBelief Propagation [31] 45.9 46.2
ICM 45.7 48.8
TRWS [32] 51.5 52.0
QPBO [26] 55.3 57.2
BNF 53.0 58.5
Table 2: We compare semantic segmentation results
when using a color-based pixel afﬁnity and our proposed
boundary-based afﬁnity. We note that our proposed afﬁnity
improves the performance of all globalization techniques.
Note that all of the inference methods use the same FCN
unary potentials . This suggests that for every method our


key: chroma:document
string_value:
improves the performance of all globalization techniques.
Note that all of the inference methods use the same FCN
unary potentials . This suggests that for every method our
boundary-based afﬁnity is more beneﬁcial for semantic segmentation than the color-based afﬁnity.
represents one of the object classes). In the experimental
section, we show that this solution produces better quantitative and qualitative results in comparison to commonly used
globalization techniques.
4. Experimental Results
In this section we present quantitative and qualitative results for semantic segmentation on the SBD [13] dataset,
which contains objects and their per-pixel annotations for
20Pascal VOC classes. We evaluate semantic segmentation results using two evaluation metrics. The ﬁrst metric
measures accuracy based on pixel intersection-over-union
averaged per pixels (PP-IOU) across the 20 classes. According to this metric, the accuracy is computed on a perpixel basis. As a result, the images that contain large object regions are given more importance. However, for certain applications we may need to accurately segment small
objects. Therefore, similar to [4] we also consider the PIIOU metric (pixel intersection-over-union averaged per image across the 20 classes), which gives equal weight to each
of the images.


key: chroma:document
string_value:
of the images.
We compare Boundary Neural Fields with other commonly used global inference methods. These methods include Belief Propagation [31], Iterated Conditional Mode
(ICM), Graph Cuts [5], and Dense-CRF [19]. Note that in
all of our evaluations we use the same FCN unary potentials
for every model.
Our evaluations provide evidence for three conclusions:
In Subsection 4.1, we show that our boundary-based
pixel afﬁnities are better suited for semantic segmentation than the traditional color-based afﬁnities.
In Subsection 4.2, we demonstrate that our global minimization leads to better results than those achieved byother inference schemes.
In Fig. 5, we qualitatively compare the outputs of FCN
and Dense-CRF to our predicted segmentations. This
comparison shows that the BNF segments are better
localized around the object boundaries and that they
are also spatially smooth.
4.1. Comparing Afﬁnity Functions for Semantic
Segmentation
In Table 2, we consider two global models. Both models use the same unary potentials obtained from the FCN
softmax layer. However, the ﬁrst model uses the popular
color-based pairwise afﬁnities, while the second employs


key: chroma:document
string_value:
softmax layer. However, the ﬁrst model uses the popular
color-based pairwise afﬁnities, while the second employs
our boundary-based afﬁnities. Each of these two models
is optimized using several inference strategies. The table
shows that using our boundary based-afﬁnity function improves the results of all global inference methods according to both evaluation metrics. Note that we cannot include Dense-CRF [19] in this comparison because it employs an efﬁcient message-passing technique and integrating our afﬁnities into this technique is a non-trivial task.
However, we compare our method with Dense-CRF in Subsection 4.2.
The results in Table 2 suggest that our semantic boundary based pixel afﬁnity function yields better semantic segmentation results compared to the commonly-used color
based afﬁnities. We note that we also compared the results
of our inference technique using other edge detectors, notably UCM [1] and H FL [4]. In comparison to UCM edges,
we observed that our boundaries provide 1:0%and6:0%
according to both evaluation metrics respectively. When
comparing our boundaries with H FL method, we observed
similar segmentation performance, which suggests that our
method works best with the high quality semantic boundaries.


key: chroma:document
string_value:
comparing our boundaries with H FL method, we observed
similar segmentation performance, which suggests that our
method works best with the high quality semantic boundaries.
4.2. Comparing Inference Methods for Semantic
Segmentation
Additionally, we also present semantic segmentation results for both of the metrics (PP-IOU and PI-IOU) in Table 3. In this comparison, all the techniques use the same
FCN unary potentials. Additionally, all inference methods
except Dense-CRF use our afﬁnity measure (since the previous analysis suggested that our afﬁnities yield better performance). We use BNF-SB to denote the variant of our
method that uses only semantic boundary based afﬁnities.
Additionally, we use BNF-SB-SM to indicate the version
of our method that uses both boundary and softmax -based
afﬁnities (see Eq. (3)).
Based on these results, we observe that our proposed
technique outperforms all the other globalization methods
according to both metrics, by 0:3%and1:3%respectively.
Metric Method aero bike bird boat bottle bus car cat chair cow table dog horse mbike person plant sheep sofa train tv mean


key: chroma:document
string_value:
Metric Method aero bike bird boat bottle bus car cat chair cow table dog horse mbike person plant sheep sofa train tv mean
PP-IOUFCN-Softmax 80.7 71.6 80.7 71.3 72.9 88.1 81.8 86.6 47.4 82.9 57.9 83.9 79.6 80.4 81.0 64.7 78.2 54.5 80.9 69.9 74.8
Belief Propagation [31] 81.4 72.2 82.4 72.2 74.3 88.8 82.4 87.2 48.4 83.8 58.4 84.6 80.5 80.9 81.5 65.1 79.5 55.5 81.5 71.2 75.6
ICM 81.7 72.2 82.8 72.1 75.3 89.6 83.4 87.7 46.3 83.3 58.4 84.6 80.6 81.4 81.5 65.8 79.5 56.0 80.7 74.1 75.8


key: chroma:document
string_value:
TRWS [32] 81.6 70.9 83.8 72.0 75.1 89.5 82.5 88.0 51.7 86.6 61.9 85.8 83.3 80.8 81.1 65.3 81.5 58.8 77.6 75.9 76.7
Graph Cuts [5] 82.5 72.4 84.6 73.3 77.2 89.7 83.3 88.8 49.3 84.0 60.3 85.4 82.2 81.2 81.9 66.7 79.8 58.0 82.3 74.9 76.9
QPBO [26] 82.6 72.3 84.7 73.1 76.7 89.9 83.6 89.3 49.7 85.0 61.1 86.2 82.9 81.3 82.3 67.1 80.5 58.8 82.2 75.1 77.2
Dense-CRF [19] 83.4 71.5 84.9 72.6 76.2 89.5 83.3 89.1 50.4 86.7 61.0 86.8 83.5 81.8 82.3 66.9 82.2 58.2 81.9 75.1 77.3


key: chroma:document
string_value:
BNF-SB 81.9 72.5 84.9 73.3 76.0 90.3 83.1 89.2 51.2 86.7 61.5 86.6 83.2 81.3 81.9 66.2 81.7 58.6 81.6 75.8 77.4
BNF-SB-SM 82.2 73.1 85.1 73.8 76.7 90.6 83.4 89.5 51.3 86.7 61.4 86.8 83.3 81.7 82.3 67.7 81.9 58.4 82.4 75.4 77.6
PI-IOUFCN-Softmax 56.9 35.1 47.8 41.1 27.4 51.1 43.4 52.7 22.2 43.1 29.2 54.2 40.5 45.6 59.1 24.2 43.6 24.8 55.9 37.2 41.8
Belief Propagation [31] 68.0 38.6 52.9 45.8 31.9 55.9 47.2 58.2 24.6 49.9 31.7 60.2 44.9 50.1 62.4 25.2 49.9 27.6 62.3 42.2 46.2


key: chroma:document
string_value:
ICM 65.3 40.9 56.4 45.3 33.7 58.9 49.5 61.9 25.8 53.5 33.2 62.1 48.0 53.2 63.4 24.1 54.8 34.0 63.7 47.7 48.8
TRWS [32] 67.5 40.7 60.3 46.3 35.6 63.4 49.6 69.3 29.7 58.9 37.8 67.4 57.3 53.8 64.1 26.3 62.0 36.9 63.1 49.9 52.0
Graph Cuts [5] 72.1 47.8 64.5 50.8 36.0 70.8 51.4 71.6 31.7 65.8 34.4 71.8 62.0 59.4 64.8 29.0 60.9 38.7 70.3 51.6 55.3
QPBO [26] 71.6 46.8 65.6 49.6 38.0 72.6 52.7 76.7 32.5 69.6 38.9 74.4 61.4 61.0 66.2 30.3 68.7 41.4 72.2 52.8 57.2


key: chroma:document
string_value:
Dense-CRF [19] 68.0 39.5 58.0 45.0 33.4 62.8 47.7 66.0 29.4 60.9 36.0 68.5 54.6 51.4 63.7 28.3 57.6 37.1 65.9 48.2 51.1
BNF-SB 71.6 48.1 67.2 52.3 37.8 79.5 52.9 80.8 33.3 71.5 39.5 75.1 65.7 63.4 65.1 31.1 67.5 39.6 73.2 54.7 58.5
BNF-SB-SM 72.0 48.9 66.5 52.9 39.1 79.0 53.4 78.6 32.9 72.2 39.4 74.6 65.9 64.2 65.8 31.7 66.9 39.0 73.1 53.9 58.5
Table 3: Semantic segmentation results on the SBD dataset according to PP-IOU (per pixel) and PI-IOU (per image) evaluation metrics. We use BNF-SB to denote the variant of our method that uses only semantic boundary based afﬁnities.


key: chroma:document
string_value:
Additionally, we use BNF-SB-SM to indicate our method that uses boundary and softmax based afﬁnities (See Eq. (3)). We
observe that our proposed globalization method outperforms other globalization techniques according to both metrics by
at least 0:3%and1:3%respectively. Note that in this experiment, all of the inference methods use the same FCN unary
potentials . Additionally, for each method except Dense-CRF (it is challenging to incorporate boundary based afﬁnities into
the Dense-CRF framework) we use our boundary based afﬁnities, since those lead to better results.
Additionally, these results indicate that most beneﬁt comes
from the semantic boundary afﬁnity term rather than the
softmax afﬁnity term.
In Fig. 5, we also present qualitative semantic segmentation results. Note that, compared to the segmentation output from the softmax layer, our segmentation is much better localized around the object boundaries. Additionally,
in comparison to Dense-CRF predictions, our method produces segmentations that are much spatially smoother.
4.3. Semantic Boundary Classiﬁcation
We can also label our boundaries with a speciﬁc object


key: chroma:document
string_value:
4.3. Semantic Boundary Classiﬁcation
We can also label our boundaries with a speciﬁc object
class, using the same classiﬁcation strategy as in the H FL
system [4]. Since the SBD dataset provides annotations for
semantic boundary classiﬁcation, we can test our results
against the state-of-the-art H FL [4] method for this task.
Due to the space limitation, we do not include full results for
each category. However, we observe that our produced results achieve mean Max F-Score of 54:5%(averaged across
all20classes) whereas H FL method obtains 51:7%.
5. Conclusions
In this work we introduced a Boundary Neural Field
(BNF), an architecture that employs a semantic segmentation FCN to predict semantic boundaries and then uses the
predicted boundaries and the FCN output to produce an improved semantic segmentation maps a global optimization.
We showed that our predicted boundaries are better suited
for semantic segmentation than the commonly used low-level color based afﬁnities. Additionally, we introduced a
global energy function that decomposes semantic segmentation into multiple binary problems and relaxes an integrality constraint. We demonstrated that the minimization
of this global energy allows us to predict segmentations
that are better localized around the object boundaries and


key: chroma:document
string_value:
of this global energy allows us to predict segmentations
that are better localized around the object boundaries and
that are spatially smoother compared to the segmentations
achieved by prior methods. We made the code of our globalization technique available at http://www.seas.upenn.
edu/~gberta/publications.html .
The main goal of this work was to show the effectiveness of boundary-based afﬁnities for semantic segmentation. However, due to differentiability of our global energy,
it may be possible to add more parameters inside the BNFs
and learn them in a front-to-end fashion. We believe that
optimizing the entire architecture jointly could capture the
inherent relationship between semantic segmentation and
boundary detection even better and further improve the performance of BNFs. We will investigate this possibility in
our future work.
6. Acknowledgements
This research was funded in part by NSF award CNS1205521.
References
[1] Pablo Arbelaez, Michael Maire, Charless Fowlkes, and Jitendra Malik. Contour detection and hierarchical image segmentation. IEEE
Trans. Pattern Anal. Mach. Intell. , 33(5):898–916, May 2011. 2, 4, 7


key: chroma:document
string_value:
Trans. Pattern Anal. Mach. Intell. , 33(5):898–916, May 2011. 2, 4, 7
[2] Pablo Arbelaez, J. Pont-Tuset, Jon Barron, F. Marqués, and Jitendra
Malik. Multiscale combinatorial grouping. In Computer Vision and
Pattern Recognition (CVPR) , 2014. 2, 4
[3] Gedas Bertasius, Jianbo Shi, and Lorenzo Torresani. Deepedge: A
multi-scale bifurcated deep network for top-down contour detection.
InThe IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , June 2015. 2, 3, 4
[4] Gedas Bertasius, Jianbo Shi, and Lorenzo Torresani. High-for-low
and low-for-high: Efﬁcient boundary detection from deep object features and its applications to high-level vision. In The IEEE International Conference on Computer Vision (ICCV) , December 2015. 2,
3, 4, 7, 8
[5] Yuri Boykov, Olga Veksler, and Ramin Zabih. Fast approximate energy minimization via graph cuts. IEEE Trans. Pattern Anal. Mach.
Intell. , 23(11):1222–1239, November 2001. 5, 7, 8


key: chroma:document
string_value:
Intell. , 23(11):1222–1239, November 2001. 5, 7, 8
[6] João Carreira, Rui Caseiro, Jorge Batista, and Cristian Sminchisescu.
Semantic segmentation with second-order pooling. In Proceedings
of the 12th European Conference on Computer Vision - Volume Part
VII, ECCV’12, pages 430–443, Berlin, Heidelberg, 2012. SpringerVerlag. 2
[7] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin
Murphy, and Alan L. Yuille. Semantic image segmentation with deep
convolutional nets and fully. In ICLR , 2015. 1, 2, 3
[8] Jifeng Dai, Kaiming He, and Jian Sun. Boxsup: Exploiting bounding boxes to supervise convolutional networks for semantic segmentation. In The IEEE International Conference on Computer Vision
(ICCV) , December 2015. 3
[9] Piotr Dollár and C. Lawrence Zitnick. Fast edge detection using
structured forests. PAMI , 2015. 2, 4


key: chroma:document
string_value:
[9] Piotr Dollár and C. Lawrence Zitnick. Fast edge detection using
structured forests. PAMI , 2015. 2, 4
[10] Clement Farabet, Camille Couprie, Laurent Najman, and Yann LeCun. Learning hierarchical features for scene labeling. IEEE Transactions on Pattern Analysis and Machine Intelligence , August 2013.
2
[11] Yaroslav Ganin and Victor S. Lempitsky. N4-ﬁelds: Neural network
nearest neighbor ﬁelds for image transforms. ACCV , 2014. 2, 4
[12] Saurabh Gupta, Ross Girshick, Pablo Arbeláez, and Jitendra Malik.
Learning rich features from RGB-D images for object detection and
segmentation. In Proceedings of the European Conference on Computer Vision (ECCV) , 2014. 2
[13] Bharath Hariharan, Pablo Arbelaez, Lubomir Bourdev, Subhransu
Maji, and Jitendra Malik. Semantic contours from inverse detectors.
InInternational Conference on Computer Vision (ICCV) , 2011. 4, 7


key: chroma:document
string_value:
Maji, and Jitendra Malik. Semantic contours from inverse detectors.
InInternational Conference on Computer Vision (ICCV) , 2011. 4, 7
[14] Bharath Hariharan, Pablo Arbeláez, Ross Girshick, and Jitendra Malik. Simultaneous detection and segmentation. In European Conference on Computer Vision (ECCV) , 2014. 2
[15] Bharath Hariharan, Pablo Andrés Arbeláez, Ross B. Girshick, and
Jitendra Malik. Hypercolumns for object segmentation and ﬁnegrained localization. In The IEEE Conference on Computer Vision
and Pattern Recognition (CVPR) , June 2015. 2, 3
[16] Seunghoon Hong, Hyeonwoo Noh, and Bohyung Han. Decoupled
deep neural network for semi-supervised semantic segmentation. In
NIPS) , December 2015. 3
[17] Phillip Isola, Daniel Zoran, Dilip Krishnan, and Edward H. Adelson.
Crisp boundary detection using pointwise mutual information. In
ECCV , 2014. 2
[18] Jyri J Kivinen, Christopher KI Williams, and Nicolas Heess. Visual
boundary prediction: A deep neural prediction network and quality


key: chroma:document
string_value:
[18] Jyri J Kivinen, Christopher KI Williams, and Nicolas Heess. Visual
boundary prediction: A deep neural prediction network and quality
dissection. AISTATS , 1(2):9, 2014. 2
[19] Philipp Krähenbühl and Vladlen Koltun. Efﬁcient inference in fully
connected crfs with gaussian edge potentials. In J. Shawe-Taylor,
R.S. Zemel, P.L. Bartlett, F. Pereira, and K.Q. Weinberger, editors,
Advances in Neural Information Processing Systems 24 , pages 109–
117. Curran Associates, Inc., 2011. 1, 5, 7, 8[20] Joseph Lim, C. Lawrence Zitnick, and Piotr Dollár. Sketch tokens:
A learned mid-level representation for contour and object detection.
InCVPR , 2013. 2
[21] Guosheng Lin, Chunhua Shen, Ian D. Reid, and Anton van den Hengel. Efﬁcient piecewise training of deep structured models for semantic segmentation. CoRR , abs/1504.01013, 2015. 3
[22] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In The IEEE Conference


key: chroma:document
string_value:
[22] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In The IEEE Conference
on Computer Vision and Pattern Recognition (CVPR) , June 2015. 1,
3
[23] D. Martin, C. Fowlkes, D. Tal, and J. Malik. A database of human
segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In Proc. 8th
Int’l Conf. Computer Vision , volume 2, pages 416–423, July 2001. 4
[24] Mohammadreza Mostajabi, Payman Yadollahpour, and Gregory
Shakhnarovich. Feedforward semantic segmentation with zoom-out
features. CoRR , abs/1412.0774, 2014. 2
[25] X. Ren and L. Bo. Discriminatively Trained Sparse Code Gradients
for Contour Detection. In Advances in Neural Information Processing Systems , December 2012. 4
[26] Carsten Rother, Vladimir Kolmogorov, Victor Lempitsky, and Martin
Szummer. Optimizing binary mrfs via extended roof duality. In Proc
Comp. Vision Pattern Recogn. (CVPR) , June 2007. 7, 8
[27] Wei Shen, Xinggang Wang, Yan Wang, Xiang Bai, and Zhijiang


key: chroma:document
string_value:
Comp. Vision Pattern Recogn. (CVPR) , June 2007. 7, 8
[27] Wei Shen, Xinggang Wang, Yan Wang, Xiang Bai, and Zhijiang
Zhang. Deepcontour: A deep convolutional feature learned by
positive-sharing loss for contour detection. June 2015. 2, 4
[28] Jonathan R Shewchuk. An introduction to the conjugate gradient
method without the agonizing pain. Technical report, Pittsburgh, PA,
USA, 1994. 6
[29] Jianbo Shi and Jitendra Malik. Normalized cuts and image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence , 22:888–905, 1997. 2
[30] K. Simonyan and A. Zisserman. Very deep convolutional networks
for large-scale image recognition. CoRR , abs/1409.1556, 2014. 3
[31] Marshall F. Tappen and William T. Freeman. Comparison of graph
cuts with belief propagation for stereo, using identical mrf parameters. In Proceedings of the Ninth IEEE International Conference on
Computer Vision - Volume 2 , ICCV ’03, pages 900–, Washington,
DC, USA, 2003. IEEE Computer Society. 5, 7, 8


key: chroma:document
string_value:
Computer Vision - Volume 2 , ICCV ’03, pages 900–, Washington,
DC, USA, 2003. IEEE Computer Society. 5, 7, 8
[32] Martin Wainwright, Tommi Jaakkola, and Alan Willsky. Map estimation via agreement on (hyper)trees: Message-passing and linear
programming approaches. IEEE Transactions on Information Theory, 51:3697–3717, 2002. 7, 8
[33] Saining Xie and Zhuowen Tu. Holistically-nested edge detection.
InThe IEEE International Conference on Computer Vision (ICCV) ,
December 2015. 2, 4
[34] Shuai Zheng, Sadeep Jayasumana, Bernardino Romera-Paredes, Vibhav Vineet, Zhizhong Su, Dalong Du, Chang Huang, and Philip Torr.
Conditional random ﬁelds as recurrent neural networks. In International Conference on Computer Vision (ICCV) , 2015. 3
[35] Dengyong Zhou, Olivier Bousquet, Thomas N. Lal, Jason Weston,
and Bernhard Schölkopf. Learning with local and global consistency.
In S. Thrun, L.K. Saul, and B. Schölkopf, editors, Advances in Neural Information Processing Systems 16 , pages 321–328. MIT Press,


key: chroma:document
string_value:
Unitary Evolution Recurrent Neural Networks
Martin ArjovskyMARJOVSKY @DC.UBA.AR
Amar ShahAS793@ CAM .AC.UK
Yoshua Bengio
Universidad de Buenos Aires, University of Cambridge,
Universit ´e de Montr ´eal. Yoshua Bengio is a CIFAR Senior Fellow.
Indicates ﬁrst authors. Ordering determined by coin ﬂip.
Abstract
Recurrent neural networks (RNNs) are notoriously difﬁcult to train. When the eigenvalues
of the hidden to hidden weight matrix deviate
from absolute value 1, optimization becomes difﬁcult due to the well studied issue of vanishingandexploding gradients, especially when trying to learn long-term dependencies. To circumvent this problem, we propose a new architecture
that learns a unitary weight matrix, with eigenvalues of absolute value exactly 1. The challenge we address is that of parametrizing unitary matrices in a way that does not require expensive computations (such as eigendecomposition) after each weight update. We construct an
expressive unitary weight matrix by composing
several structured matrices that act as building
blocks with parameters to be learned. Optimization with this parameterization becomes feasible


key: chroma:document
string_value:
expressive unitary weight matrix by composing
several structured matrices that act as building
blocks with parameters to be learned. Optimization with this parameterization becomes feasible
only when considering hidden states in the complex domain. We demonstrate the potential of
this architecture by achieving state of the art results in several hard tasks involving very longterm dependencies.
1. Introduction
Deep Neural Networks have shown remarkably good performance on a wide range of complex data problems including speech recognition (Hinton et al., 2012), image
recognition (Krizhevsky et al., 2012) and natural language
processing (Collobert et al., 2011). However, training very
deep models remains a difﬁcult task. The main issue surrounding the training of deep networks is the vanishing
andexploding gradients problems introduced by HochreProceedings of the 33rdInternational Conference on Machine
Learning , New York, NY , USA, 2016. JMLR: W&CP volume
48. Copyright 2016 by the author(s).iter (1991) and shown by Bengio et al. (1994) to be necessarily arising when trying to learn to reliably store bits
of information in any parametrized dynamical system. If
gradients propagated back through a network vanish, the
credit assignment role of backpropagation is lost, as information about small changes in states in the far past has no


key: chroma:document
string_value:
gradients propagated back through a network vanish, the
credit assignment role of backpropagation is lost, as information about small changes in states in the far past has no
inﬂuence on future states. If gradients explode, gradientbased optimization algorithms struggle to traverse down a
cost surface, because gradient-based optimization assumes
small changes in parameters yield small changes in the objective function. As the number of time steps considered
in the sequence of states grows, the shrinking or expanding
effects associated with the state-to-state transformation at
individual time steps can grow exponentially, yielding respectively vanishing or exploding gradients. See Pascanu
et al. (2010) for a review.
Although the long-term dependencies problem appears
intractable in the absolute (Bengio et al., 1994) for
parametrized dynamical systems, several heuristics have
recently been found to help reduce its effect, such as the
use of self-loops and gating units in the LSTM (Hochreiter & Schmidhuber, 1997) and GRU (Cho et al., 2014) recurrent architectures. Recent work also supports the idea
of using orthogonal weight matrices to assist optimization
(Saxe et al., 2014; Le et al., 2015).
In this paper, we explore the use of orthogonal and unitary


key: chroma:document
string_value:
(Saxe et al., 2014; Le et al., 2015).
In this paper, we explore the use of orthogonal and unitary
matrices in recurrent neural networks. We start in Section 2
by showing a novel bound on the propagated gradients in
recurrent nets when the recurrent matrix is orthogonal. Section 3 discusses the difﬁculties of parameterizing real valued orthogonal matrices and how they can be alleviated by
moving to the complex domain.
We discuss a novel approach to constructing expressive
unitary matrices as the composition of simple unitary matrices which require at most O(nlogn)computation and
O(n)memory, when the state vector has dimension n.
These are unlike general matrices, which require O(n2)
computation and memory. Complex valued representationsarXiv:1511.06464v4  [cs.LG]  25 May 2016
Unitary Evolution Recurrent Neural Networks
have been considered for neural networks in the past, but
with limited success and adoption (Hirose, 2003; Zimmermann et al., 2011). We hope our ﬁndings will change this.
Whilst our model uses complex valued matrices and parameters, all implementation and optimization is possible
with real numbers and has been done in Theano (Bergstra


key: chroma:document
string_value:
Whilst our model uses complex valued matrices and parameters, all implementation and optimization is possible
with real numbers and has been done in Theano (Bergstra
et al., 2010). This along with other implementation details
are discussed in Section 4, and the code used for the experiments is available online. The potential of the developed
model for learning long term dependencies with relatively
few parameters is explored in Section 5. We ﬁnd that the
proposed architecture generally outperforms LSTMs and
previous approaches based on orthogonal initialization.
2. Orthogonal Weights and Bounding the
Long-Term Gradient
A matrix, W, is orthogonal if W>W=WW>=I.
Orthogonal matrices have the property that they preserve
norm (i.e.kWhk2=khk2) and hence repeated iterative
multiplication of a vector by an orthogonal matrix leaves
the norm of the vector unchanged.
LethTandhtbe the hidden unit vectors for hidden layers
Tandtof a neural network with Thidden layers and T
t. IfCis the objective we are trying to minimize, then
thevanishing andexploding gradient problems refer to the
decay or growth of@C
@htas the number of layers, T, grows.
Letbe a pointwise nonlinearity function, and


key: chroma:document
string_value:
decay or growth of@C
@htas the number of layers, T, grows.
Letbe a pointwise nonlinearity function, and
zt+1=Wtht+Vtxt+1
ht+1=(zt+1) (1)
then by the chain rule
@C
@ht=@C
@hT@hT
@ht
=@C
@hTT1Y
k=t@hk+1
@hk=@C
@hTT1Y
k=tDk+1WT
k (2)
where Dk+1=diag(0(zk+1))is the Jacobian matrix of
the pointwise nonlinearity.
In the following we deﬁne the norm of a matrix to refer to
the spectral radius norm (or operator 2-norm) and the norm
of a vector to mean L2-norm. By deﬁnition of the operator norms, for any matrices A;Band vectorvwe have
kAvkk AkkvkandkABkk AkkBk. If the weight
matrices Wkare norm preserving (i.e. orthogonal), thenweprove




@C


key: chroma:document
string_value:
matrices Wkare norm preserving (i.e. orthogonal), thenweprove




@C
@ht



=




@C
@hTT1Y
k=tDk+1WT
k








@C
@hT



T1Y
k=t

Dk+1WT
k

=



@C
@hT



T1Y
k=tkDk+1k: (3)
Since Dkis diagonal,kDkk= max j=1;:::;nj0(z(j)
k)j,
withz(j)
kthej-th pre-activation of the k-th hidden layer.
If the absolute value of the derivative 0can take some
value > 1, then this bound is useless, since k@C
@htk


@C
@hT


Ttwhich grows exponentially in T. We therefore cannot effectively bound@C
@htfor deep networks, resulting potentially in exploding gradients.
In the casej0j<  < 1, equation 3 proves that that@C


key: chroma:document
string_value:
@htfor deep networks, resulting potentially in exploding gradients.
In the casej0j<  < 1, equation 3 proves that that@C
@httends to 0 exponentially fast as Tgrows, resulting in guaranteed vanishing gradients. This argument makes the rectiﬁed linear unit (ReLU) nonlinearity an attractive choice
(Glorot et al., 2011; Nair & Hinton, 2010). Unless all the
activations are killed at one layer, the maximum entry of
Dkis 1, resulting inkDkk= 1 for all layers k. With
ReLU nonlinearities, we thus have




@C
@ht







@C
@hT



T1Y
k=tkDk+1k=



@C
@hT



: (4)
Most notably, this result holds for a network of arbitrary
depth and renders engineering tricks like gradient clipping
unnecessary (Pascanu et al., 2010).
To the best of our knowledge, this analysis is a novel contribution and the ﬁrst time a neural network architecture has
been mathematically proven to avoid exploding gradients.
3. Unitary Evolution RNNs
Unitary matrices generalize orthogonal matrices to the


key: chroma:document
string_value:
been mathematically proven to avoid exploding gradients.
3. Unitary Evolution RNNs
Unitary matrices generalize orthogonal matrices to the
complex domain. A complex valued, norm preserving matrix,U, is called a unitary matrix and is such that UU=
UU=I, where Uis the conjugate transpose of U. Directly parametrizing the set of unitary matrices in such a
way that gradient-based optimization can be applied is not
straightforward because a gradient step will typically yield
a matrix that is not unitary, and projecting on the set of unitary matrices (e.g., by performing an eigendecomposition)
generally costsO(n3)computation when Uisnn.
The most important feature of unitary and orthogonal matrices for our purpose is that they have eigenvalues jwith
Unitary Evolution Recurrent Neural Networks
absolute value 1. The following lemma, proved in (Hoffman & Kunze, 1971), may shed light on a method which
can be used to efﬁciently span a large set of unitary matrices.
Lemma 1. A complex square matrix Wis unitary if and
only if it has an eigendecomposition of the form W=


key: chroma:document
string_value:
Lemma 1. A complex square matrix Wis unitary if and
only if it has an eigendecomposition of the form W=
VDV, wheredenotes the conjugate transpose. Here,
V;D2Cnnare complex matrices, where Vis unitary,
andDis a diagonal such that jDj;jj= 1. Furthermore, W
is a real orthogonal matrix if and only if for every eigenvalue Dj;j=jwith eigenvector vj, there is also a complex conjugate eigenvalue k=jwith corresponding
eigenvectorvk=vj.
Writingj=eiwjwithwj2R, a naive method to learn a
unitary matrix would be to ﬁx a basis of eigenvectors V2
Cnnand set
W=VDV; (5)
where Dis a diagonal such that Dj;j=j.
Lemma 1 informs us how to construct a real orthogonal
matrix, W. We must (i) ensure the columns of Vcome
in complex conjugate pairs, vk=vj, and (ii) tie weights


key: chroma:document
string_value:
matrix, W. We must (i) ensure the columns of Vcome
in complex conjugate pairs, vk=vj, and (ii) tie weights
wk=wjin order to achieve eiwj=eiwk. Most neural network objective functions are differentiable with respect to the weight matrices, and consequently wjmay be
learned by gradient descent.
Unfortunately the above approach has undesirable properties. Fixing Vand learning wrequiresO
n2
memory,
which is unacceptable given that the number of learned parameters isO(n). Further note that calculating Vufor an
arbitrary vector urequiresO(n2)computation. Setting V
to the identity would satisfy the conditions of the lemma,
whilst reducing memory and computation requirements to
O(n), however, Wwould remain diagonal, and have poor
representation capacity.
We propose an alternative strategy to parameterize unitary
matrices. Since the product of unitary matrices is itself a
unitary matrix, we compose several simple, parameteric,
unitary matrices to construct a single, expressive unitary
matrix. The four unitary building blocks considered are
D, a diagonal matrix with Dj;j=eiwj, with parameterswj2R,
R=I2vv


key: chroma:document
string_value:
D, a diagonal matrix with Dj;j=eiwj, with parameterswj2R,
R=I2vv
kvk2, a reﬂection matrix in the complex
vectorv2Cn,
, a ﬁxed random index permutation matrix, and
 F andF1, the Fourier and inverse Fourier transforms.Appealingly, D,Randall permitO(n)storage and
O(n)computation for matrix vector products. FandF1
require no storage and O(nlogn)matrix vector multiplication using the Fast Fourier Transform algorithm. A major
advantage of composing unitary matrices of the form listed
above, is that the number of parameters, memory and computational cost increase almost linearly in the size of the
hidden layer. With such a weight matrix, immensely large
hidden layers are feasible to train, whilst being impossible
in traditional neural networks.
With this in mind, in this work we choose to consider recurrent neural networks with unitary hidden to hidden weight
matrices. Our claim is that the ability to have large hidden
layers where hidden states norms are preserved provides
a powerful tool for modeling long term dependencies in
sequence data. (Bengio et al., 1994) suggest that having
a large memory may be crucial for solving difﬁcult tasks


key: chroma:document
string_value:
sequence data. (Bengio et al., 1994) suggest that having
a large memory may be crucial for solving difﬁcult tasks
with long ranging dependencies: the smaller the state dimension, the more information necessarily has to be eliminated when mapping a long sequence to a ﬁxed-dimension
state.
We call any RNN architecture which uses a unitary hidden
to hidden matrix a unitary evolution RNN (uRNN). After
experimenting with several structures, we settled on the following composition
W=D3R2F1D2R 1FD1: (6)
Whilst each but the permutation matrix is complex, we
parameterize and represent them with real numbers for implementation purposes. When the ﬁnal cost is real and differentiable, we may perform gradient descent optimization
to learn the parameters. (Yang et al., 2015) construct a real
valued, non-orthogonal matrix using a similar parameterization with the motivation of parameter reduction by an
order of magnitude on an industrial sized network. This
combined with earlier work (Le et al., 2010) suggests that it
is possible to create highly expressive matrices by composing simple matrices with few parameters. In the following
section, we explain details on how to implement our model
and illustrate how we bypass the potential difﬁculties of
working in the complex domain.


key: chroma:document
string_value:
section, we explain details on how to implement our model
and illustrate how we bypass the potential difﬁculties of
working in the complex domain.
4. Architecture details
In this section, we describe the nonlinearity we used, how
we incorporate real valued inputs with complex valued hidden units and map from complex hidden states to real outputs.
4.1. Complex hidden units
Our implementation represents all complex numbers using real values in terms of their real and imaginary parts.
Unitary Evolution Recurrent Neural Networks
Under this framework, we sidestep the lack of support
for complex numbers by most deep learning frameworks.
Consider multiplying the complex weight matrix W=
A+iBby the complex hidden vector h=x+iy,
where A;B;x;y are real. It is trivially true that Wh=
(AxBy) +i(Ay+Bx). When we represent v2Cnas
Re(v)>;Im(v)>>2R2n, we compute complex matrix
vector products with real numbers as follows
Re(Wh)
Im(Wh)
=AB
B ARe(h)
Im(h)
: (7)
More generally, let f:Cn!Cnbe any complex function andz=x+iyany complex vector. We may write


key: chroma:document
string_value:
Im(h)
: (7)
More generally, let f:Cn!Cnbe any complex function andz=x+iyany complex vector. We may write
f(z) =(x;y) +i(x;y)where; :Rn!Rn. This
allows us to implement everything using real valued operations, compatible with any any deep learning framework
with automatic differentiation such as Theano.
4.2. Input to Hidden, Nonlinearity, Hidden to Output
As is the case with most recurrent networks, our uRNN follows the same hidden to hidden mapping as equation 1 with
Vt=VandWt=W. Denote the size of the complex
valued hidden states as nh. The input to hidden matrix is
complex valued, V2Cnhnin. We learn the initial hidden
stateh02Cnhas a parameter of the model.
Choosing an appropriate nonlinearity is not trivial in the
complex domain. As discussed in the introduction, using a
ReLU is a natural choice in combination with a norm preserving weight matrix. We ﬁrst experimented with placing
separate ReLU activations on the real and imaginary parts


key: chroma:document
string_value:
ReLU is a natural choice in combination with a norm preserving weight matrix. We ﬁrst experimented with placing
separate ReLU activations on the real and imaginary parts
of the hidden states. However, we found that such a nonlinearity usually performed poorly. Our intuition is that applying separate ReLU nonlinearities to the real and imaginary parts brutally impacts the phase of a complex number,
making it difﬁcult to learn structure.
We speculate that maintaining the phase of hidden states
may be important for storing information across a large
number of time steps, and our experiments supported this
claim. A variation of the ReLU that we name modReLU ,
is what we ﬁnally chose. It is a pointwise nonlinearity,
modReLU (z) :C!C, which affects only the absolute
value of a complex number, deﬁned as
modReLU (z) =(jzj+b)z
jzjifjzj+b0
0 ifjzj+b<0(8)
whereb2Ris a bias parameter of the nonlinearity. For a
nhdimensional hidden space we learn nhnonlinearity bias
parameters, one per dimension. Note that the modReLU
is similar to the ReLU in spirit, in fact more concretely


key: chroma:document
string_value:
parameters, one per dimension. Note that the modReLU
is similar to the ReLU in spirit, in fact more concretely
modReLU (z) =ReLU (jzj+b)z
jzj.To map hidden states to output, we deﬁne a matrix U2
Rno2nh, wherenois the output dimension. We calculate
a linear output as
ot=URe(ht)
Im(ht)
+bo; (9)
wherebo2Rnois the output bias. The linear output is real
valued (ot2Rno) and can be used for prediction and loss
function calculation akin to typical neural networks (e.g. it
may be passed through a softmax which is used for cross
entropy calculation for classiﬁcation tasks).
4.3. Initialization
Due to the stability of the norm preserving operations of
our network, we found that performance was not very sensitive to initialization of parameters. For full disclosure
and reproducibility, we explain our initialization strategy
for each parameter below.
We initialize VandU(the input and output matrices) as in (Glorot & Bengio, 2010),
with weights sampled independently from uniforms,
Uh
p
6pnin+nout;p
6pnin+nouti
.


key: chroma:document
string_value:
with weights sampled independently from uniforms,
Uh
p
6pnin+nout;p
6pnin+nouti
.
The biases,bandboare initialized to 0. This implies
that at initialization, the network is linear with unitary
weights, which seems to help early optimization (Saxe
et al., 2014).
The reﬂection vectors for R1andR2are initialized
coordinate-wise from a uniform U[1;1]. Note that
the reﬂection matrices are invariant to scalar multiplication of the parameter vector, hence the width of the
uniform initialization is unimportant.
The diagonal weights for D1;D2andD3are sampled from a uniform, U[;]. This ensures that the
diagonal entries Dj;jare sampled uniformly over the
complex unit circle.
We initialize h0with a uniform,Uh
q
3
2nh;q
3
2nhi
,
which results in E
kh0k2
= 1. Since the norm of
the hidden units are roughly preserved through unitary
evolution and inputs are typically whitened to have
norm 1, we have hidden states, inputs and linear outputs of the same order of magnitude, which seems to
help optimization.
5. Experiments


key: chroma:document
string_value:
norm 1, we have hidden states, inputs and linear outputs of the same order of magnitude, which seems to
help optimization.
5. Experiments
In this section we explore the performance of our uRNN
in relation to (a) RNN with tanh activations, (b) IRNN (Le
et al., 2015), that is an RNN with ReLU activations and
with the recurrent weight matrix initialized to the identity,
Unitary Evolution Recurrent Neural Networks
Figure 1. Results of the copying memory problem for time lags of 100;200;300;500. The LSTM is able to beat the baseline only for
100times steps. Conversely the uRNN is able to completely solve each time length in very few training iterations, without getting stuck
at the baseline.
and (c) LSTM (Hochreiter & Schmidhuber, 1997) models. We show that the uRNN shines quantitatively when
it comes to modeling long term dependencies and exhibits
qualitatively different learning properties to the other models.
We chose a handful of tasks to evaluate the performance
of the various models. The tasks were especially created
to be be pathologically hard, and have been used as benchmarks for testing the ability of a model to capture long-term
memory (Hochreiter & Schmidhuber, 1997; Le et al., 2015;


key: chroma:document
string_value:
memory (Hochreiter & Schmidhuber, 1997; Le et al., 2015;
Graves et al., 2014; Martens & Sutskever, 2011)
Of the handful of optimization algorithms we tried on the
various models, RMSProp (Tieleman & Hinton, 2012) lead
to fastest convergence and is what we stuck to for all experiments here on in. However, we found the IRNN to be
particularly unstable; it only ran without blowing up with
incredibly low learning rates and gradient clipping. Since
the performance was so poor relative to other models we
compare against, we do not show IRNN curves in the ﬁgures. In each experiment we use a learning rate of 103
and a decay rate of 0:9. For the LSTM and RNN models,
we had to clip gradients at 1 to avoid exploding gradients.Gradient clipping was unnecessary for the uRNN.
5.1. Copying memory problem
Recurrent networks have been known to have trouble remembering information about inputs seen many time steps
previously (Bengio et al., 1994; Pascanu et al., 2010). We
therefore want to test the uRNN’s ability to recall exactly
data seen a long time ago.
Following a similar setup to (Hochreiter & Schmidhuber,


key: chroma:document
string_value:
data seen a long time ago.
Following a similar setup to (Hochreiter & Schmidhuber,
1997), we outline the copy memory task. Consider 10 categories,faig9
i=0. The input takes the form of a T+20 length
vector of categories, where we test over a range of values
ofT. The ﬁrst 10entries are sampled uniformly, independently and with replacement from faig7
i=0, and represent
the sequence which will need to be remembered. The next
T1entries are set to a8, which can be thought of as the
’blank’ category. The next single entry is a9, which represents a delimiter, which should indicate to the algorithm
that it is now required to reproduce the initial 10categories
in the output. The remaining 10entries are set to a8. The
required output sequence consists of T+ 10 repeated entries ofa8, followed by the ﬁrst 10categories of the input
Unitary Evolution Recurrent Neural Networks
Figure 2. Results of the adding problem for T= 100 ;200;400;750. The RNN with tanh is not able to beat the baseline for any time
length. The LSTM and the uRNN show similar performance across time lengths, consistently beating the baseline.


key: chroma:document
string_value:
length. The LSTM and the uRNN show similar performance across time lengths, consistently beating the baseline.
sequence in exactly the same order. The goal is to minimize the average cross entropy of category predictions at
each time step of the sequence. The task amounts to having to remember a categorical sequence of length 10, for T
time steps.
A simple baseline can be established by considering an
optimal strategy when no memory is available, which we
deem the memoryless strategy. The memoryless strategy
would be to predict a8forT+ 10 entries and then predict
each of the ﬁnal 10categories from the set faig7
i=0independently and uniformly at random. The categorical cross
entropy of this strategy is10 log(8)
T+20.
We ran experiments where the RNN with tanh activations,
IRNN, LSTM and uRNN had hidden layers of size 80, 80,
40 and 128 respectively. This equates to roughly 6500 parameters per model. In Figure 1, we see that aside from the
simplest case, both the RNN with tanh and more surprisingly the LSTMs get almost exactly the same cost as the
memoryless strategy. This behaviour is consistent with the
results of (Graves et al., 2014), in which poor performance


key: chroma:document
string_value:
memoryless strategy. This behaviour is consistent with the
results of (Graves et al., 2014), in which poor performance
is reported for the LSTM for a very similar long term memory problem.
The uRNN consistently achieves perfect performance inrelatively few iterations, even when having to recall sequences after 500 time steps. What is remarkable is that
the uRNN does not get stuck at the baseline at all, whilst
the LSTM and RNN do. This behaviour suggests that the
representations learned by the uRNN have qualitatively different properties from both the LSTM and classical RNNs.
5.2. Adding Problem
We closely follow the adding problem deﬁned in (Hochreiter & Schmidhuber, 1997) to explain the task at hand. Each
input consists of two sequences of length T. The ﬁrst sequence, which we denote x, consists of numbers sampled
uniformly at random U[0;1]. The second sequence is an indicator sequence consisting of exactly two entries of 1 and
remaining entries 0. The ﬁrst 1 entry is located uniformly
at random in the ﬁrst half of the sequence, whilst the second 1 entry is located uniformly at random in the second
half. The output is the sum of the two entries of the ﬁrst sequence, corresponding to where the 1 entries are located in


key: chroma:document
string_value:
half. The output is the sum of the two entries of the ﬁrst sequence, corresponding to where the 1 entries are located in
the second sequence. A naive strategy of predicting 1 as the
output regardless of the input sequence gives an expected
mean squared error of 0:167, the variance of the sum of two
independent uniform distributions. This is our baseline to
beat.
Unitary Evolution Recurrent Neural Networks
Figure 3. Results on pixel by pixel MNIST classiﬁcation tasks. The uRNN is able to converge in a fraction of the iterations that the
LSTM requires. The LSTM performs better on MNIST classiﬁcation, but the uRNN outperforms on the more complicated task of
permuted pixels.
We chose to use 128 hidden units for the RNN with tanh,
IRNN and LSTM and 512 for the uRNN. This equates to
roughly 16K parameters for the RNN with tanh and IRNN,
60K for the LSTM and almost 9K for the uRNN. All models were trained using batch sizes of 20 and 50 with the best
results being reported. Our results are shown in Figure 2.
The LSTM and uRNN models are able to convincingly beat
the baseline up to T= 400 time steps. Both models do well


key: chroma:document
string_value:
The LSTM and uRNN models are able to convincingly beat
the baseline up to T= 400 time steps. Both models do well
whenT= 750 , but the mean squared error does not reach
close to 0. The uRNN achieves lower test error, but it’s
curve is more noisy. Despite having vastly more parameters, we monitored the LSTM performance to ensure no
overﬁtting.
The RNN with tanh and IRNN were not able to beat the
baseline for any number of time steps. (Le et al., 2015) report that their RNN solve the problem for T= 150 and the
IRNN forT= 300 , but they require over a million iterations before they start learning. Neither of the two models came close to either the uRNN or the LSTM in performance. The stark difference in our ﬁndings are best explained by our use of RMSprop with signiﬁcantly higher
learning rates ( 103as opposed to 108) than (Le et al.,
2015) use for SGD with momentum.
5.3. Pixel-by-pixel MNIST
In this task, suggested by (Le et al., 2015), algorithms are
fed pixels of MNIST (LeCun et al., 1998) sequentially and


key: chroma:document
string_value:
In this task, suggested by (Le et al., 2015), algorithms are
fed pixels of MNIST (LeCun et al., 1998) sequentially and
required to output a class label at the end. We consider two
tasks: one where pixels are read in order (from left to right,
bottom to top) and one where the pixels are all randomly
permuted using the same randomly generated permutation
matrix. The same model architectures as for the addingproblem were used for this task, except we now use a softmax for category classiﬁcation. We ran the optimization
algorithms until convergence of the mean categorical cross
entropy on test data, and plot test accuracy in Figure 3.
Both the uRNN and LSTM perform applaudably well here.
On the correct unpermuted MNIST pixels, the LSTM performs better, achieving 98.2 % test accurracy versus 95.1%
for the uRNN. However, when we permute the ordering of
the pixels, the uRNN dominates with 91.4% of accuracy in
contrast to the 88% of the LSTM, despite having less than
a quarter of the parameters. This result is state of the art on
this task, beating the IRNN (Le et al., 2015), which reaches
close to 82% after 1 million training iterations. Notice that


key: chroma:document
string_value:
this task, beating the IRNN (Le et al., 2015), which reaches
close to 82% after 1 million training iterations. Notice that
uRNN reaches convergence in less than 20 thousand iterations, while it takes the LSTM from 5 to 10 times as many
to ﬁnish learning.
Permuting the pixels of MNIST images creates many
longer term dependencies across pixels than in the original pixel ordering, where a lot of structure is local. This
makes it necessary for a network to learn and remember
more complicated dependencies across varying time scales.
The results suggest that the uRNN is better able to deal
with such structure over the data, where the LSTM is better
suited to more local sequence structure tasks.
5.4. Exploratory experiments
Norms of hidden state gradients. As discussed in Section 2, key to being able to learn long term dependencies
is in controlling@C
@ht. With this in mind, we explored how
each model propagated gradients, by examining


@C
@ht


as
a function of t. Gradient norms were computed at the beUnitary Evolution Recurrent Neural Networks
Figure 4. From left to right. Norms of the gradients with respect to hidden states i.e.


@C


key: chroma:document
string_value:
Figure 4. From left to right. Norms of the gradients with respect to hidden states i.e.


@C
@ht


at (i) beginning of training, (ii) after 100
iterations. (iii) Norms of the hidden states and (iv) L2distance between hidden states and ﬁnal hidden state. The gradient norms of
uRNNs do not decay as fast as for other models as training progresses. uRNN hidden state norms stay much more consistent over time
than the LSTM. LSTM hidden states stay almost the same after a number of time steps, suggesting that it is not able to use new input
information.
ginning of training and again after 100 iterations of training
on the adding problem. The curves are plotted in Figure 4.
It is clear that at ﬁrst, the uRNN propagates gradients perfectly, while each other model has exponentially vanishing
gradients. After 100 iterations of training, each model experiences vanishing gradients, but the uRNN is best able to
propagate information, having much less decay.
Hidden state saturation. We claim that typical recurrent
architectures saturate, in the sense that after they acquire
some information, it becomes much more difﬁcult to acquire further information pertaining to longer dependencies. We took the uRNN and LSTM models trained on


key: chroma:document
string_value:
some information, it becomes much more difﬁcult to acquire further information pertaining to longer dependencies. We took the uRNN and LSTM models trained on
the adding problem with T= 200 , and computed a forward pass with newly generated data for the adding problem withT= 1000 . In order to show saturation effects,
we plot the norms of the hidden states and the L2distance
between each state and the last in Figure 4.
In our experiments, it is clear that the uRNN does not suffer
as much as other models do. Notice that whilst the norms of
hidden states in the uRNN grow very steadily over time, in
the LSTM they grow very fast, and then stay constant after
about 500time steps. This behaviour may suggest that the
LSTM hidden states saturate in their ability to incorporate
new information, which is vital for modeling long complicated sequences. It is interesting to see that the LSTM
hidden state at t= 500 , is close to that of t= 1000 , whilst
this is far from the case in the uRNN. Again, this suggests
that the LSTM’s capacity to use new information to alter
its hidden state severly degrades with sequence length. The
uRNN does not suffer from this difﬁculty nearly as badly.
A clear example of this phenomenon was observed in the


key: chroma:document
string_value:
uRNN does not suffer from this difﬁculty nearly as badly.
A clear example of this phenomenon was observed in the
adding problem with T= 750 . We found that the Pearson
correlation between the LSTM output prediction and the
ﬁrst of the two uniform samples (whose sum is the target
output) was= 0:991. This suggests that the LSTM learnt
to simply ﬁnd and store the ﬁrst sample, as it was unable
to incorporate any more information by the time it reachedthe second, due to saturation of the hidden states.
6. Discussion
There are a plethora of further ideas that may be explored
from our ﬁndings, both with regards to learning representation and efﬁcient implementation. For example, one hurdle
of modeling long sequences with recurrent networks is the
requirement of storing all hidden state values for the purpose of gradient backpropagation. This can be prohibitive,
since GPU memory is typically a limiting factor of neural
network optimization. However, since our weight matrix is
unitary, its inverse is its conjugate transpose, which is just
as easy to operate with. If further we were to use an invertible nonlinearity function, we would no longer need to store


key: chroma:document
string_value:
as easy to operate with. If further we were to use an invertible nonlinearity function, we would no longer need to store
hidden states, since they can be recomputed in the backward pass. This could have potentially huge implications,
as we would be able to reduce memory usage by an order
ofT, the number of time steps. This would make having
immensely large hidden layers possible, perhaps enabling
vast memory representations.
In this paper we demonstrate state of the art performance
on hard problems requiring long term reasoning and memory. These results are based on a novel parameterization
of unitary matrices which permit efﬁcient matrix computations and parameter optimization. Whilst complex domain modeling has been widely succesful in the signal processing community (e.g. Fourier transforms, wavelets), we
have yet to exploit the power of complex valued representation in the deep learning community. Our hope is that
this work will be a step forward in this direction. We motivate the idea of unitary evolution as a novel way to mitigate
the problems of vanishing and exploding gradients. Empirical evidence suggests that our uRNN is better able to pass
gradient information through long sequences and does not
suffer from saturating hidden states as much as LSTMs,
typical RNNs, or RNNs initialized with the identity weight
matrix (IRNNs).


key: chroma:document
string_value:
suffer from saturating hidden states as much as LSTMs,
typical RNNs, or RNNs initialized with the identity weight
matrix (IRNNs).
Unitary Evolution Recurrent Neural Networks
Acknowledgments : We thank the developers of
Theano (Bergstra et al., 2010) for their great work.
We thank NSERC, Compute Canada, Canada Research
Chairs and CIFAR for their support. We would also like
to thank C ¸ aglar Gulc ¸ehre, David Krueger, Soroush Mehri,
Marcin Moczulski, Mohammad Pezeshki and Saizheng
Zhang for helpful discussions, comments and code sharing.
References
Bengio, Yoshua, Simard, Patrice, and Frasconi, Paolo.
Learning long-term dependencies with gradient descent
is difﬁcult. IEE Transactions on Neural Networks , 5,
1994.
Bergstra, James, Breuleux, Olivier, Bastien, Fr ´ed´eric,
Lamblin, Pascal, Pascanu, Razvan, Desjardins, Guillaume, Turian, Joseph, Warde-Farley, David, and Bengio, Yoshua. Theano: a CPU and GPU math expression


key: chroma:document
string_value:
compiler. Proceedings of the Python for Scientiﬁc Computing Conference (SciPy) , 2010.
Cho, Kyunghyun, van Merri ¨enboer, Bart, Bahdanau,
Dzmitry, and Bengio, Yoshua. On the properties of neural machine translation: Encoder–Decoder approaches.
InEighth Workshop on Syntax, Semantics and Structure
in Statistical Translation , October 2014.
Collobert, Ronan, Weston, Jason, Bottou, L ´eon, Karlen,
Michael, Kavukcuoglu, Koray, and Kuksa, Pavel. Natural language processing (almost) from scratch. Journal
of Machine Learning Research , 12:2493–2537, 2011.
Glorot, Xavier and Bengio, Yoshua. Understanding the
difﬁculty of training deep feedforward neural networks.
International Conference on Artiﬁcial Intelligence and
Statistics (AISTATS) , 2010.
Glorot, Xavier, Bordes, Antoine, and Bengio, Yoshua.
Deep sparse rectiﬁer neural networks. International
Conference on Artiﬁcial Intelligence and Statistics (AISTATS) , 2011.
Graves, Alex, Wayne, Greg, and Danihelka, Ivo. Neural


key: chroma:document
string_value:
Conference on Artiﬁcial Intelligence and Statistics (AISTATS) , 2011.
Graves, Alex, Wayne, Greg, and Danihelka, Ivo. Neural
turing machines. arXiv preprint arXiv:1410.5401 , 2014.
Hinton, Geoffrey, Deng, Li, Yu, Dong, Dahl, George, Mohamed, Abdel-rahman, Jaitly, Navdeep, Senior, Andrew,
Vanhoucke, Vincent, Nguyen, Patrick, Sainath, Tara, and
Kingsbury, Brian. Deep neural networks for acoustic
modeling in speech recognition. Signal Processing Magazine , 2012.
Hirose, Akira. Complex-valued neural networks: theories
and applications , volume 5. World Scientiﬁc Publishing
Company Incorporated, 2003.Hochreiter, S. Untersuchungen zu dynamischen neuronalen Netzen. Diploma thesis, T.U. M ¨unich, 1991.
Hochreiter, Sepp and Schmidhuber, J ¨urgen. Long shortterm memory. Neural Computation , 8(9):1735–1780,
1997.
Hoffman, Kenneth and Kunze, Ray. Linear Algebra . Pearson, second edition, 1971.


key: chroma:document
string_value:
1997.
Hoffman, Kenneth and Kunze, Ray. Linear Algebra . Pearson, second edition, 1971.
Krizhevsky, Alex, Sutskever, Ilya, and Hinton, Geoffrey E.
Imagenet classiﬁcation with deep convolutional neural networks. Neural Information Processing Systems ,
2012.
Le, Quoc, Sarl ´os, Tam ´as, and Smola, Alex. Fastfood - approximating kernel expansions in loglinear time. International Conference on Machine Learning , 2010.
Le, Quoc V ., Navdeep, Jaitly, and Hinton, Geoffrey E. A
simple way to initialize recurrent networks of rectiﬁed
linear units. arXiv preprint arXiv:1504.00941 , 2015.
LeCun, Yann, Bottou, L ´eon, Bengio, Yoshua, and Haffner,
Patrick. Gradient-based learning applied to document
recognition. Proceedings of the IEEE , 1998.
Martens, James and Sutskever, Ilya. Learning recurrent
neural networks with hessian-free optimization. International Conference on Machine Learning , 2011.
Nair, Vinod and Hinton, Geoffrey E. Rectiﬁed linear units
improve restricted boltzmann machines. International
Conference on Machine Learning , 2010.


key: chroma:document
string_value:
Nair, Vinod and Hinton, Geoffrey E. Rectiﬁed linear units
improve restricted boltzmann machines. International
Conference on Machine Learning , 2010.
Pascanu, Razvan, Mikolov, Tomas, and Bengio, Yoshua.
On the difﬁculty of training recurrent neural networks.
International Conference on Machine Learning , 2010.
Saxe, Andrew M., McLelland, James L., and Ganguli,
Surya. Exact solutions to the nonlinear dynamics of
learning in deep linear neural networks. International
Conference in Learning Representations , 2014.
Tieleman, Tijmen and Hinton, Geoffrey. Lecture 6.5rmsprop: Divide the gradient by a running average of
its recent magnitude. Coursera: Neural Networks for
Machine Learning , 2012.
Yang, Zichao, Moczulski, Marcin, Denil, Misha, de Freitas, Nando, Smola, Alex, Song, Le, and Wang, Ziyu.
Deep fried convnets. International Conference on Computer Vision (ICCV) , 2015.
Zimmermann, Hans-Georg, Minin, Alexey, and
Kusherbaeva, Victoria. Comparison of the complex valued and real valued neural networks trained
with gradient descent and random search algorithms. In
ESANN , 2011.


key: chroma:document
string_value:
Higher Order Conditional Random Fields in Deep
Neural Networks
Anurag Arnab, Sadeep Jayasumana, Shuai Zheng, Philip H.S. Torr
University of Oxford
{firstname.lastname }@eng.ox.ac.uk
Abstract. We address the problem of semantic segmentation using deep learning.
Most segmentation systems include a Conditional Random Field (CRF) to produce
a structured output that is consistent with the image’s visual features. Recent deep
learning approaches have incorporated CRFs into Convolutional Neural Networks
(CNNs), with some even training the CRF end-to-end with the rest of the network.
However, these approaches have not employed higher order potentials, which
have previously been shown to signiﬁcantly improve segmentation performance.
In this paper, we demonstrate that two types of higher order potential, based on
object detections and superpixels, can be included in a CRF embedded within a
deep network. We design these higher order potentials to allow inference with
the differentiable mean ﬁeld algorithm. As a result, all the parameters of our
richer CRF model can be learned end-to-end with our pixelwise CNN classiﬁer.
We achieve state-of-the-art segmentation performance on the PASCAL VOC


key: chroma:document
string_value:
We achieve state-of-the-art segmentation performance on the PASCAL VOC
benchmark with these trainable higher order potentials.
Keywords: Semantic Segmentation, Conditional Random Fields, Deep Learning,
Convolutional Neural Networks
1 Introduction
Semantic segmentation involves assigning a visual object class label to every pixel in an
image, resulting in a segmentation with a semantic meaning for each segment. While a
strong pixel-level classiﬁer is critical for obtaining high accuracy in this task, it is also
important to enforce the consistency of the semantic segmentation output with visual
features of the image. For example, segmentation boundaries should usually coincide
with strong edges in the image, and regions in the image with similar appearance should
have the same label.
Recent advances in deep learning have enabled researchers to create stronger classiﬁers, with automatically learned features, within a Convolutional Neural Network
(CNN) [1 –3]. This has resulted in large improvements in semantic segmentation accuracy on widely used benchmarks such as PASCAL VOC [4]. CNN classiﬁers are now
considered the standard choice for pixel-level classiﬁers used in semantic segmentation.
On the other hand, probabilistic graphical models have long been popular for structured prediction of labels, with constraints enforcing label consistency. Conditional


key: chroma:document
string_value:
On the other hand, probabilistic graphical models have long been popular for structured prediction of labels, with constraints enforcing label consistency. Conditional
Random Fields (CRFs) have been the most common framework, and various rich and
arXiv:1511.08119v4  [cs.CV]  29 Jul 2016


key: chroma:document
string_value:
Higher Order Conditional Random Fields in Deep Neural Networks 3
We evaluate our higher order potentials on the PASCAL VOC 2012 semantic segmentation benchmark as well as the PASCAL Context dataset, to show signiﬁcant
improvements over our baseline and achieve state-of-the art results.
2 Related Work
Before deep learning became prominent, semantic segmentation was performed with
dense hand-crafted features which were fed into a per-pixel or region classiﬁer [14]. The
individual predictions made by these classiﬁers were often noisy as they lacked global
context, and were thus post-processed with a CRF, making use of prior knowledge such
as the fact that nearby pixels, as well as pixels of similar appearance, are likely to share
the same class label [14, 15].
The CRF model of [14] initially contained only unary and pairwise terms in an
8-neighbourhood, which [16] showed can result in shrinkage bias. Numerous improvements to this model were subsequently proposed including: densely connected pairwise
potentials facilitating interactions between all pairs of image pixels [17], formulating
higher order potentials deﬁned over cliques larger than two nodes [5, 16] in order to
capture more context, modelling co-occurrence of object classes [18 –20], and utilising


key: chroma:document
string_value:
capture more context, modelling co-occurrence of object classes [18 –20], and utilising
the results of object detectors [6, 21, 22].
Recent advances in deep learning have allowed us to replace hand-crafted features
with features learned speciﬁcally for semantic segmentation. The strength of these
representations was illustrated by [3] who achieved signiﬁcant improvements over
previous hand-crafted methods without using any CRF post-processing. Chen et al. [12]
showed further improvements by post-processing the results of a CNN with a CRF.
Subsequent works [9 –11, 23] have taken this idea further by incorporating a CRF as
layers within a deep network and then learning parameters of both the CRF and CNN
together via backpropagation.
In terms of enhancements to conventional CRF models, Ladicky et al. [6] proposed using an off-the-shelf object detector to provide additional cues for semantic
segmentation. Unlike other approaches that reﬁne a bounding-box detection to produce
a segmentation [8, 24], this method used detector outputs as a soft constraint and can
thus recover from object detection errors. Their formulation, however, used graph-cut
inference, which was only tractable due to the absence of dense pairwise potentials.


key: chroma:document
string_value:
thus recover from object detection errors. Their formulation, however, used graph-cut
inference, which was only tractable due to the absence of dense pairwise potentials.
Object detectors have also been used by [21, 25], who also modelled variables that
describe the degree to which an object hypothesis is accepted.
We formulate the detection potential in a different manner to [6, 21, 25] so that it is
amenable to mean ﬁeld inference. Mean ﬁeld permits inference with dense pairwise connections, which results in substantial accuracy improvements [10,12, 17]. Furthermore,
mean ﬁeld updates related to our potentials are differentiable and its parameters can thus
be learned in our end-to-end trainable architecture.
We also note that while the semantic segmentation problem has mostly been formulated in terms of pixels [3,10,14], some have expressed it in terms of superpixels [26 –28].
Superpixels can capture more context than a single pixel and computational costs can
also be reduced if one considers pairwise interactions between superpixels rather than individual pixels [21]. However, such superpixel representations assume that the segments
4 Arnab et al.
share boundaries with objects in an image, which is not always true. As a result, several


key: chroma:document
string_value:
4 Arnab et al.
share boundaries with objects in an image, which is not always true. As a result, several
authors [5, 7] have employed higher order potentials deﬁned over superpixels that encourage label consistency over regions, but do not strictly enforce it. This approach also
allows multiple, non-hierarchical layers of superpixels to be integrated. Our formulation
uses this kind of higher order potential, but in an end-to-end trainable CNN.
Graphical models have been used with CNNs in other areas besides semantic segmentation, such as in pose-estimation [29] and group activity recognition [30]. Alternatively,
Ionescu et al. [31] incorporated structure into a deep network with structured matrix
layers and matrix backpropagation. However, the nature of models used in these works
is substantially different to ours. Some early works that advocated gradient backpropagation through graphical model inference for parameter optimisation include [32, 33]
and [34].
Our work differentiates from the above works since, to our knowledge, we are the ﬁrst
to propose and conduct a thorough experimental investigation of higher order potentials
that are based on detection outputs and superpixel segmentation in a CRF which is
learned end-to-end in a deep network. Note that although [7] formulated mean ﬁeld


key: chroma:document
string_value:
learned end-to-end in a deep network. Note that although [7] formulated mean ﬁeld
inference with higher order potentials, they did not consider object detection potentials
at all, nor were the parameters learned.
3 Conditional Random Fields
We now review conditional random ﬁelds used in semantic segmentation and introduce
the notation used in the paper. Take an image IwithNpixels, indexed 1,2,...,N . In
semantic segmentation, we attempt to assign every pixel a label from a predeﬁned set of
labels L={l1,l2,...,l L}. Deﬁne a set of random variables X1,X2,...,X N, one for
each pixel, where each Xi∈ L. LetX= [X1X2... X N]T. Any particular assignment
xtoXis thus a solution to the semantic segmentation problem.
We use notations {V}, andV(i)to represent the set of elements of a vector V, and
theithelement of V, respectively. Given a graph Gwhere the vertices are from {X}
and the edges deﬁne connections among these variables, the pair (I,X)is modelled as a


key: chroma:document
string_value:
and the edges deﬁne connections among these variables, the pair (I,X)is modelled as a
CRF characterised by Pr(X=x|I) = (1/Z(I)) exp( −E(x|I)), whereE(x|I)is the
energy of the assignment xandZ(I)is the normalisation factor known as the partition
function. We drop the conditioning on Ihereafter to keep the notation uncluttered. The
energyE(x)of an assignment is deﬁned using the set of cliques Cin the graph G. More
speciﬁcally, E(x) =/summationtext
c∈Cψc(xc),where xcis a vector formed by selecting elements
ofxthat correspond to random variables belonging to the clique c, andψc(.)is the
cost function for the clique c. The function, ψc(.), usually uses prior knowledge about a
good segmentation, as well as information from the image, the observation the CRF is
conditioned on.
Minimising the energy yields the maximum a posteriori (MAP) labelling of the
image i.e.the most probable label assignment given the observation (image). When
dense pairwise potentials are used in the CRF to obtain higher accuracy, exact inference


key: chroma:document
string_value:
image i.e.the most probable label assignment given the observation (image). When
dense pairwise potentials are used in the CRF to obtain higher accuracy, exact inference
is impracticable, and one has to resort to an approximate inference method such as mean
ﬁeld inference [17]. Mean ﬁeld inference is particularly appealing in a deep learning
setting since it is possible to formulate it as a Recurrent Neural Network [10].
Higher Order Conditional Random Fields in Deep Neural Networks 5
4 CRF with Higher Order Potentials
Many CRF models that have been incorporated into deep learning frameworks [10, 12]
have so far used only unary and pairwise potentials. However, potentials deﬁned on
higher order cliques have been shown to be useful in previous works such as [7, 16]. The
key contribution of this paper is to show that a number of explicit higher order potentials
can be added to CRFs to improve image segmentation, while staying compatible with
deep learning. We formulate these higher order potentials in a manner that mean ﬁeld
inference can still be used to solve the CRF. Advantages of mean ﬁeld inference are
twofold: First, it enables efﬁcient inference when using densely-connected pairwise


key: chroma:document
string_value:
twofold: First, it enables efﬁcient inference when using densely-connected pairwise
potentials. Multiple works, [10,33] have shown that dense pairwise connections result in
substantial accuracy improvements, particularly at image boundaries [12, 17]. Secondly,
we keep all our mean ﬁeld updates differentiable with respect to their inputs as well
as the CRF parameters introduced. This design enables us to use backpropagation to
automatically learn all the parameters in the introduced potentials.
We use two types of higher order potential, one based on object detections and the
other based on superpixels. These are detailed in Sections 4.1 and 4.2 respectively. Our
complete CRF model is represented by
E(x) =/summationdisplay
iψU
i(xi) +/summationdisplay
i < jψP
ij(xi,xj) +/summationdisplay
dψDet
d(xd) +/summationdisplay
sψSP
s(xs),(1)
where the ﬁrst two terms ψU
i(.)andψP
ij(.,.)are the usual unary and densely-connected
pairwise energies [17] and the last two terms are the newly introduced higher order


key: chroma:document
string_value:
i(.)andψP
ij(.,.)are the usual unary and densely-connected
pairwise energies [17] and the last two terms are the newly introduced higher order
energies. Energies from the object detection take the form ψDet
d(xd), where vector xd
is formed by elements of xthat correspond to the foreground pixels of the dthobject
detection. Superpixel label consistency based energies take the form ψSP
s(xs), where xs
is formed by elements of xthat correspond to the pixels belonging to the sthsuperpixel.
4.1 Object Detection Based Potentials
Semantic segmentation errors can be classiﬁed into two broad categories [35]: recognition
and boundary errors. Boundary errors occur when semantic labels are incorrect at the
edges of objects, and it has been shown that densely connected CRFs with appearanceconsistency terms are effective at combating this problem [17]. On the other hand,
recognition errors occur when object categories are recognised incorrectly or not at
all. A CRF with only unary and pairwise potentials cannot effectively correct these
errors since they are caused by poor unary classiﬁcation. However, we propose that a
state-of-the-art object detector [36,37] capable of recognising and localising objects, can


key: chroma:document
string_value:
state-of-the-art object detector [36,37] capable of recognising and localising objects, can
provide important information in this situation and help reduce the recognition error, as
shown in Fig. 2.
A key challenge in feeding-in object-detection potentials to semantic segmentation
are false detections. A na ¨ıve approach of adding an object detector’s output to a CRF
formulated to solve the problem of semantic segmentation would confuse the CRF
due to the presence of the false positives in the detector’s output. Therefore, a robust
formulation, which can automatically reject object detection false positives when they


key: chroma:document
string_value:
Higher Order Conditional Random Fields in Deep Neural Networks 7
positive: some of the detections obtained from the object detector may be identiﬁed to
be false following CRF inference.
AllYdvariables are added to the CRF which previously contained only Xivariables.
Let each (Xd,Yd), where {Xd}={Xi∈ {X}|i∈Fd}, form a clique cdin the
CRF. We deﬁne the detection-based higher order energy associated with a particular
assignment (xd,yd)to the clique (Xd,Yd)as follows:
ψDet
d(Xd=xd,Yd=yd) =

wDetsd
nd/summationtextnd
i=1[x(i)
d=ld]ifyd= 0,
wDetsd
nd/summationtextnd
i=1[x(i)
d/ne}ationslash=ld]ifyd= 1,(2)
wherend=|Fd|is the number of foreground pixels in the dthdetection,x(i)
dis theith
element of the vector xd,wDetis a learnable weight parameter, and [.]is the Iverson


key: chroma:document
string_value:
dis theith
element of the vector xd,wDetis a learnable weight parameter, and [.]is the Iverson
bracket. Note that this potential encourages X(i)
ds to take the value ldwhenYdis1, and
at the same time encourages Ydto be 0when manyX(i)
ds do not take ld. In other words,
it enforces the consistency among X(i)
ds andYd.
An important property of the above deﬁnition of ψDet
d(.)is that it can be simpliﬁed
as a sum of pairwise potentials between Ydand eachX(i)
dfori= 1,2,...,n d. That is,
ψDet
d(Xd=xd,Yd=yd) =nd/summationdisplay
i=1fd(x(i)
d,yd), where,
fd(x(i)
d,yd) =

wDetsd
nd[x(i)
d=ld] ifyd= 0,
wDetsd
nd[x(i)
d/ne}ationslash=ld] ifyd= 1.(3)


key: chroma:document
string_value:
d=ld] ifyd= 0,
wDetsd
nd[x(i)
d/ne}ationslash=ld] ifyd= 1.(3)
We make use of this simpliﬁcation in Section 5 when deriving the mean ﬁeld updates
associated with this potential.
For the latent Yvariables, in addition to the joint potentials with Xvariables,
described in Eq. (2)and(3), we also include unary potentials, which are initialised
from the score sdof the object detection. The underlying idea is that if the object
detector detects an object with high conﬁdence, the CRF in turn starts with a high initial
conﬁdence about the validity of that detection. This conﬁdence can, of course, change
during the CRF inference depending on other information ( e.g.segmentation unary
potentials) available to the CRF.
Examples of input images with multiple detections and GrabCut foreground masks
are shown in Figure 3. Note how false detections are ignored and erroneous parts of the
foreground mask are also largely ignored.
4.2 Superpixel Based Potentials
The next type of higher order potential we use is based on the idea that superpixels


key: chroma:document
string_value:
foreground mask are also largely ignored.
4.2 Superpixel Based Potentials
The next type of higher order potential we use is based on the idea that superpixels
obtained from oversegmentation [39, 40] quite often contain pixels from the same visual
object. It is therefore natural to encourage pixels inside a superpixel to have the same
semantic label. Once again, this should not be a hard constraint in order to keep the
8 Arnab et al.
(a) (b) (c) (d)
Fig. 3: Effects of imperfect foreground segmentation (a,b) Detected objects, as well
as the foreground masks obtained from GrabCut. (c,d) Output using detection potentials.
Incorrect parts of the foreground segmentation of the main aeroplane, and entire TV
detection have been ignored by CRF inference as they did not agree with the other energy
terms. The person is a failure case though as the detection has caused part of the sofa to
be erroneously labelled.
(a) (b) (c)
Fig. 4: Segmentation enhancement from superpixel based potentials (a) The output
of our system without any superpixel potentials. (b) Superpixels obtained from the image
using the method of [39]. Only one “layer” of superpixels is shown. In practice, we


key: chroma:document
string_value:
using the method of [39]. Only one “layer” of superpixels is shown. In practice, we
used four. (c) The output using superpixel potentials. The result has improved as we
encourage consistency over superpixel regions. This removes some of the spurious noise
that was present previously.
algorithm robust to initial superpixel segmentation errors and to violations of this key
assumption.
We use two types of energies in the CRF to encourage superpixel consistency in
semantic segmentation. Firstly, we use the Pn-Potts model type energy [41], which is
described by,
ψSP
s(Xs=xs) =

wLow(l) if allx(i)
s=l,
wHigh otherwise,(4)
wherewLow(l)<w High for alll, and{Xs} ⊂ {X}is a clique deﬁned by a superpixel.
The primary idea is that assigning different labels to pixels in the same superpixel incurs
a higher cost, whereas one obtains a lower cost if the labelling is consistent throughout
the superpixel. Costs wLow(l)andwHigh are learnable during the end-to-end training of
the network.


key: chroma:document
string_value:
the superpixel. Costs wLow(l)andwHigh are learnable during the end-to-end training of
the network.
Secondly, to make this potential stronger, we average initial unary potentials from the
classiﬁer (the CNN in our case), across all pixels in the superpixel and use the average
Higher Order Conditional Random Fields in Deep Neural Networks 9
as an additional unary potential for those pixels. During experiments, we observed that
superpixel based higher order energy helps in getting rid of small spurious regions of
wrong labels in the segmentation output, as shown in Fig. 4.
5 Mean Field Updates and Their Differentials
This section discusses the mean ﬁeld updates for the higher order potentials previously
introduced. These update operations are differentiable with respect to the Qi(Xi)distribution inputs at each iteration, as well as the parameters of our higher order potentials.
This allows us to train our CRF end-to-end as another layer of a neural network.
Take a CRF with random variables V1,V2,...,V Nand a set of cliques C, which
includes unary, pairwise and higher order cliques. Mean ﬁeld inference approximates
the joint distribution Pr(V=v)with the product of marginals/producttext
iQ(Vi=vi). We


key: chroma:document
string_value:
the joint distribution Pr(V=v)with the product of marginals/producttext
iQ(Vi=vi). We
useQ(Vc=vc)to denote the marginal probability mass for a subset {Vc}of these
variables. Where there is no ambiguity, we use the short-hand notation Q(vc)to represent
Q(Vc=vc). General mean ﬁeld updates of such a CRF take the form [13]
Qt+1(Vi=v) =1
Ziexp
−/summationdisplay
c∈C/summationdisplay
{vc|vi=v}Qt(vc−i)ψc(vc)
, (5)
whereQtis the marginal after the tthiteration, vcan assignment to all variables in
cliquec,vc−ian assignment to all variables in cexcept forVi,ψc(vc)is the cost of
assigning vcto the clique c, andZiis the normalisation constant that makes Q(Vi=v)
a probability mass function after the update.
Updates from Detection Based Potentials Following Eq. (3)above, we now use
Eq.(5)to derive the mean ﬁeld updates related to ψDet


key: chroma:document
string_value:
Updates from Detection Based Potentials Following Eq. (3)above, we now use
Eq.(5)to derive the mean ﬁeld updates related to ψDet
d. The contribution from ψDet
dto
the update of Q(X(i)
d=l)takes the form
/summationdisplay
{(xd,yd)|x(i)
d=l}Q(xd−i,yd)ψDet
d(xd,yd) =

wDetsd
ndQ(Yd= 0) ifl=ld,
wDetsd
ndQ(Yd= 1) otherwise,(6)
where xd−iis an assignment to Xdwith theithelement deleted. Using the same equations, we derive the contribution from the energy ψDet
dto the update of Q(Yd=b)to
take the form
/summationdisplay
{(xd,yd)|yd=b}Q(xd)ψDet
d(xd,yd) =

wDetsd
nd/summationtextnd
i=1Q(X(i)
d=ld) ifb= 0,
wDetsd
nd/summationtextnd


key: chroma:document
string_value:
nd/summationtextnd
i=1Q(X(i)
d=ld) ifb= 0,
wDetsd
nd/summationtextnd
i=1(1−Q(X(i)
d=ld))otherwise.
(7)
It is possible to increase the number of parameters in ψDet
d(.). Since we use backpropagation to learn these parameters automatically during end-to-end training, it is desirable
to have a high number of parameters to increase the ﬂexibility of the model. Following
10 Arnab et al.
this idea, we made the weight wDetclass speciﬁc, that is, a function wDet(ld)is used
instead ofwDetin Eqs. (2),(6)and (7). The underlying assumption is that detector
outputs can be very helpful for certain classes, while being not so useful for classes that
the detector performs poorly on, or classes for which foreground segmentation is often
inaccurate.
Note that due to the presence of detection potentials in the CRF, error differentials
calculated with respect to the Xvariable unary potentials and pairwise parameters will
no longer be valid in the forms described in [10]. The error differentials with respect to


key: chroma:document
string_value:
calculated with respect to the Xvariable unary potentials and pairwise parameters will
no longer be valid in the forms described in [10]. The error differentials with respect to
theXandYvariables, as well as class-speciﬁc detection potential weights wDet(l)are
included in the supplementary material.
Updates for Superpixel Based Potentials The contribution from the Pn-Potts type
potential to the mean ﬁeld update of Q(xi=l), where pixel iis in the superpixel clique
s, was derived in [7] as
/summationdisplay
{xs|x(i)
s=l}Q(xs−i)ψSP
s(xs) =wLow(l)/productdisplay
j∈c,j/ne}ationslash=iQ(Xj=l)+wHigh
1−/productdisplay
j∈c−iQ(Xj=l)
.
(8)
This update operation is differentiable with respect to the parameters wLow(l)andwHigh,
allowing us to optimise them via backpropagation, and also with respect to the Q(X)
values enabling us to optimise previous layers in the network.


key: chroma:document
string_value:
allowing us to optimise them via backpropagation, and also with respect to the Q(X)
values enabling us to optimise previous layers in the network.
Convergence of parallel mean ﬁeld updates Mean ﬁeld with parallel updates, as
proposed in [17] for speed, does not have any convergence guarantees in the general case.
However, we usually empirically observed convergence with higher order potentials,
without damping the mean ﬁeld update as described in [7, 42]. This may be explained by
the fact that the unaries from the initial pixelwise-prediction part of our network provide
a good initialisation. In cases where the mean ﬁeld energy did not converge, we still
empirically observed good ﬁnal segmentations.
6 Experiments
We evaluate our new CRF formulation on two different datasets using the CRF-RNN
network [10] as the main baseline, since we are essentially enriching the CRF model
of [10]. We then present ablation studies on our models.
6.1 Experimental set-up and results
Our deep network consists of two conceptually different, but jointly trained stages. The
ﬁrst, “unary” part of our network is formed by the FCN-8s architecture [3]. It is initialised


key: chroma:document
string_value:
ﬁrst, “unary” part of our network is formed by the FCN-8s architecture [3]. It is initialised
from the Imagenet-trained VGG-16 network [2], and then ﬁne-tuned with data from
the VOC 2012 training set [4], extra VOC annotations of [43] and the MS COCO [44]
dataset.
Higher Order Conditional Random Fields in Deep Neural Networks 11
Table 1: Comparison of each higher order
potential with baseline on VOC 2012 reduced validation set
MethodReduced
val set(%)
Baseline (unary + pairwise) [10] 72.9
Superpixels only 74.0
Detections only 74.9
Detections and Superpixels 75.8Table 2: Mean IoU accuracy on VOC 2012
test set. All methods are trained with MS
COCO [44] data
Method Test set(%)
Ours 77.9
DPN [9] 77.5
Centrale Super Boundaries [45] 75.7
Dilated Convolutions [46] 75.3
BoxSup [35] 75.2
DeepLab Attention [47] 75.1
CRF-RNN (baseline) [10] 74.7
DeepLab WSSL [48] 73.9


key: chroma:document
string_value:
DeepLab Attention [47] 75.1
CRF-RNN (baseline) [10] 74.7
DeepLab WSSL [48] 73.9
DeepLab [12] 72.7
Table 3: Mean Intersection over Union (IoU) results on PASCAL Context validation set
compared to other current methods.
Method Ours BoxSup [35] ParseNet [49] CRF-RNN [10] FCN-8s [3] CFM [28]
Mean IoU (%) 41.3 40.5 40.4 39.3 37.8 34.4
The output of the ﬁrst stage is fed into our CRF inference network. This is implemented using the mean ﬁeld update operations and their differentials described in
Section 5. Five iterations of mean ﬁeld inference were performed during training. Our
CRF network has two additional inputs in addition to segmentation unaries obtained from
the FCN-8s network: data from the object detector and superpixel oversegmentations of
the image.
We used the publicly available code and model of the Faster R-CNN [37] object detector. The fully automated version of GrabCut [38] was then used to obtain foregrounds
from the detection bounding boxes. These choices were made after conducting preliminary experiments with alternate detection and foreground segmentation algorithms.


key: chroma:document
string_value:
from the detection bounding boxes. These choices were made after conducting preliminary experiments with alternate detection and foreground segmentation algorithms.
Four levels of superpixel oversegmentations were used, with increasing superpixel
size to deﬁne the cliques used in this potential. Four levels were used since performance
on the VOC validation set stopped increasing after this number. We used the superpixel
method of [39] as it was shown to adhere to object boundaries the best [40], but our
method generalises to any oversegmentation algorithm.
We trained the full network end-to-end, optimising the weights of the CNN classiﬁer
(FCN-8s) and CRF parameters jointly. We initialised our network using the publicly
available weights of [10], and trained with a learning rate of 10−10and momentum of
0.99. The learning rate is low because the loss was not normalised by the number of
pixels in the training image. This is to have a larger loss for images with more pixels.
When training our CRF, we only used VOC 2012 data [4] as it has the most accurate
labelling, particularly around boundaries.
12 Arnab et al.
PASCAL VOC 2012 Dataset The improvement obtained by each higher order potential
was evaluated on the same reduced validation set [3] used by our baseline [10]. As Table


key: chroma:document
string_value:
PASCAL VOC 2012 Dataset The improvement obtained by each higher order potential
was evaluated on the same reduced validation set [3] used by our baseline [10]. As Table
1 shows, each new higher order potential improves the mean IoU over the baseline. We
only report test set results for our best method since the VOC guidelines discourage
the use of the test set for ablation studies. On the test set (Table 2), we outperform our
baseline by 3.2% which equates to a 12.6% reduction in the error rate. This sets a new
state-of-the-art on the VOC dataset. Qualitative results highlighting success and failure
cases of our algorithm, as well as more detailed results, are shown in our supplementary
material.
PASCAL Context Table 3 shows our state-of-the-art results on the recently released
PASCAL Context dataset [50]. We trained on the provided training set of 4998 images,
and evaluated on the validation set of 5105 images. This dataset augments VOC with
annotations for all objects in the scene. As a result, there are 59 classes as opposed to
the 20 in the VOC dataset. Many of these new labels are “stuff” classes such as “grass”
and “sky”. Our object detectors are therefore only trained for 20 of the 59 labels in this


key: chroma:document
string_value:
and “sky”. Our object detectors are therefore only trained for 20 of the 59 labels in this
dataset. Nevertheless, we improve by 0.8% over the previous state-of-the-art [35] and
2% over our baseline [10].
6.2 Ablation Studies
We perform additional experiments to determine the errors made by our system, show
the beneﬁts of end-to-end training and compare our detection potentials to a simpler
baseline. Unless otherwise stated, these experiments are performed on the VOC 2012
reduced validation set.
Error Analysis To analyse the improvements made by our higher order potentials, we
separately evaluate the performance on the “boundary” and “interior” regions in a similar
manner to [35]. As shown in Fig. 5 c) and d), we consider a narrow band (trimap [16])
around the “void” labels annotated in the VOC 2012 reduced validation set. The mean
IoU of pixels lying within this band is termed the “Boundary IoU” whilst the “Interior
IoU” is evaluated outside this region.
Fig. 5 shows our results as the trimap width is varied. Adding the detection potentials


key: chroma:document
string_value:
IoU” is evaluated outside this region.
Fig. 5 shows our results as the trimap width is varied. Adding the detection potentials
improves the Interior IoU over our baseline (only pairwise potentials [10]) as the object
detector may recognise objects in the image which the pixelwise classiﬁcation stage of
our network may have missed out. However, the detection potentials also improve the
Boundary IoU for all tested trimap widths as well. Improving the recognition of pixels
in the interior of an object also helps with delineating the boundaries since the strength
of the pairwise potentials exerted by the Qdistributions at each of the correctly-detected
pixels increase.
Our superpixel priors also increase the Interior IoU with respect to the baseline.
Encouraging consistency over regions helps to get rid of spurious regions of wrong
labels (as shown in Fig. 4). Fig. 5 suggests that most of this improvement occurs in the
interior of an object. The Boundary IoU is slightly lower than the baseline, and this may
Higher Order Conditional Random Fields in Deep Neural Networks 13
a) Image (b) Ground truth (c) Boundary (d) Interior
0 5 10 15 20 25 30 35 406668707274767880
Trimap WidthInterior IoU
  
Pairwise only


key: chroma:document
string_value:
0 5 10 15 20 25 30 35 406668707274767880
Trimap WidthInterior IoU
  
Pairwise only
Superpixels
Detections
Detections and Superpixels
0 5 10 15 20 25 30 35 40505560657075
Trimap WidthBoundary IoU
  
Pairwise only
Superpixels
Detections
Detections and Superpixels
(e) Interior IoU (f) Boundary IoU
Fig. 5: Error analysis on VOC 2012 reduced validation set The IoU is computed for
boundary and interior regions for various trimap widths. An example of the Boundary
and Interior regions for a sample image using a width of 9 pixels is shown in white in
the top row. Black regions are ignored in the IoU calculation.
be due to the fact that superpixels do not always align correctly with the edges of an
object (the “boundary recall” of various superpixel methods are evaluated in [40]).
We can see that the combination of detection and superpixel potentials results in a
substantial improvement in our Interior IoU. This is the primary reason our overall IoU
on the VOC benchmark increases with higher order potentials.
Beneﬁts of end-to-end training Table 4 shows how end-to-end training outperforms


key: chroma:document
string_value:
on the VOC benchmark increases with higher order potentials.
Beneﬁts of end-to-end training Table 4 shows how end-to-end training outperforms
piecewise training. We trained the CRF piecewise by freezing the weights of the unary
part of the network, and only learning the CRF parameters.
Our results in Table 2 used the FCN-8s [3] architecture to generate unaries. To show
that our higher order potentials improve performance regardless of the underlying CNN
used for producing unaries, we also perform an experiment using our reimplementation
Table 4: Comparison of mean IoU (%) obtained on VOC 2012 reduced validation set
from end-to-end and piecewise training
Method FCN-8s DCN
Unary only, ﬁne-tuned on COCO 68.3 68.6
Pairwise CRF trained piecewise 69.5 70.7
Pairwise CRF trained end-to-end 72.9 72.5
Higher Order CRF trained piecewise 73.6 73.5
Higher Order CRF trained end-to-end 75.8 75.0
Test set performance of best model 77.9 76.9
14 Arnab et al.
of the “front-end” module proposed in the Dilated Convolution Network (DCN) of [46]


key: chroma:document
string_value:
14 Arnab et al.
of the “front-end” module proposed in the Dilated Convolution Network (DCN) of [46]
instead of FCN-8s.
Table 4 shows that end-to-end training of the CRF yields considerable improvements
over piecewise training. This was the case when using either FCN-8s or DCN for
obtaining the initial unaries before performing CRF inference with higher order potentials.
This suggests that our CRF network module can be plugged into different architectures
and achieve performance improvements.
Baseline for detections To evaluate the efﬁcacy of our detection potentials, we formulate a simpler baseline since no other methods use detection information at inference
time (BoxSup [35] derives ground truth for training using ground-truth bounding boxes).
Our baseline is similar to CRF-RNN [10], but prior to CRF inference, we take the
segmentation mask from the object detection and add a unary potential proportional to
the detector’s conﬁdence to the unary potentials for those pixels. We then perform meanﬁeld inference (with only pairwise terms) on these “augmented” unaries. Using this


key: chroma:document
string_value:
method, the mean IoU increases from 72.9% to 73.6%, which is signiﬁcantly less than
the 74.9% which we obtained using only our detection potentials without superpixels
(Table 1).
Our detection potentials perform better since our latent Ydetection variables model
whether the detection hypothesis is accepted or not. Our CRF inference is able to evaluate
object detection inputs in light of other potentials. Inference increases the relative score
of detections which agree with the segmentation, and decreases the score of detections
which do not agree with other energies in the CRF. Figures 2 b) and d) show examples
of false-positive detections that have been ignored and correct detections that have been
used to reﬁne our segmentation. Our baseline, on the other hand, is far more sensitive to
erroneous detections as it cannot adjust the weight given to them during inference.
7 Conclusion
We presented a CRF model with two different higher order potentials to tackle the
semantic segmentation problem. The ﬁrst potential is based on the intuitive idea that
object detection can provide useful cues for semantic segmentation. Our formulation is
capable of automatically rejecting false object detections that do not agree at all with
the semantic segmentation. Secondly, we used a potential that encourages superpixels to


key: chroma:document
string_value:
capable of automatically rejecting false object detections that do not agree at all with
the semantic segmentation. Secondly, we used a potential that encourages superpixels to
have consistent labelling. These two new potentials can co-exist with the usual unary
and pairwise potentials in a CRF.
Importantly, we showed that efﬁcient mean ﬁeld inference is still possible in the
presence of the new higher order potentials and derived the explicit forms of the mean
ﬁeld updates and their differentials. This enabled us to implement the new CRF model
as a stack of CNN layers and to train it end-to-end in a uniﬁed deep network with a
pixelwise CNN classiﬁer. We experimentally showed that the addition of higher order
potentials results in a signiﬁcant increase in semantic segmentation accuracy allowing us
to reach state-of-the-art performance.
This work was supported by ERC grant ERC-2012-AdG 321162-HELIOS, EPSRC grant
Seebibyte EP/M013774/1, EPSRC/MURI grant EP/N019474/1 and the Clarendon Fund.
Higher Order Conditional Random Fields in Deep Neural Networks 15
References


key: chroma:document
string_value:
Higher Order Conditional Random Fields in Deep Neural Networks 15
References
1.Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classiﬁcation with deep convolutional
neural networks. In: NIPS. (2012) 1097–1105
2.Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image
recognition. In: ICLR. (2015)
3.Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation.
In: CVPR. (2015)
4.Everingham, M., Van Gool, L., Williams, C.K., Winn, J., Zisserman, A.: The pascal visual
object classes (voc) challenge. IJCV (2010)
5.Ladicky, L., Russell, C., Kohli, P., Torr, P.H.: Associative hierarchical crfs for object class
image segmentation. In: ICCV . (2009) 739–746
6.Ladick ´y, L., Sturgess, P., Alahari, K., Russell, C., Torr, P.H.S.: What, where and how many?


key: chroma:document
string_value:
combining object detectors and crfs. In: ECCV . (2010) 424–437
7.Vineet, V., Warrell, J., Torr, P.H.: Filter-based mean-ﬁeld inference for random ﬁelds with
higher-order terms and product label-spaces. IJCV (2014)
8.Hariharan, B., Arbel ´aez, P., Girshick, R., Malik, J.: Simultaneous detection and segmentation.
In: ECCV . Springer (2014) 297–312
9.Liu, Z., Li, X., Luo, P., Loy, C.C., Tang, X.: Semantic image segmentation via deep parsing
network. In: ICCV . (2015)
10.Zheng, S., Jayasumana, S., Romera-Paredes, B., Vineet, V., Su, Z., Du, D., Huang, C., Torr,
P.: Conditional random ﬁelds as recurrent neural networks. In: ICCV . (2015)
11.Lin, G., Shen, C., Reid, I.: Efﬁcient piecewise training of deep structured models for semantic
segmentation. In: CVPR. (2016)


key: chroma:document
string_value:
segmentation. In: CVPR. (2016)
12.Chen, L.C., Papandreou, G., Kokkinos, I., Murphy, K., Yuille, A.L.: Semantic image segmentation with deep convolutional nets and fully connected crfs. ICLR (2015)
13.Koller, D., Friedman, N.: Probabilistic graphical models: principles and techniques. MIT
press (2009)
14.Shotton, J., Winn, J., Rother, C., Criminisi, A.: Textonboost for image understanding: Multiclass object recognition and segmentation by jointly modeling texture, layout, and context.
IJCV (2009)
15.He, X., Zemel, R.S., Carreira-Perpi ˜n´an, M. ´A.: Multiscale conditional random ﬁelds for image
labeling. In: CVPR, IEEE (2004)
16.Kohli, P., Ladicky, L., Torr, P.: Robust higher order potentials for enforcing label consistency.
IJCV 82(3) (2009) 302–324
17.Kr¨ahenb ¨uhl, P., Koltun, V.: Efﬁcient inference in fully connected CRFs with Gaussian edge


key: chroma:document
string_value:
17.Kr¨ahenb ¨uhl, P., Koltun, V.: Efﬁcient inference in fully connected CRFs with Gaussian edge
potentials. In: NIPS. (2011)
18.Ladicky, L., Russell, C., Kohli, P., Torr, P.H.: Graph cut based inference with co-occurrence
statistics. In: ECCV . (2010) 239–253
19.Rabinovich, A., Vedaldi, A., Galleguillos, C., Wiewiora, E., Belongie, S.: Objects in context.
In: ICCV . (2007) 1–8
20.Gonfaus, J.M., Boix, X., Van de Weijer, J., Bagdanov, A.D., Serrat, J., Gonzalez, J.: Harmony
potentials for joint classiﬁcation and segmentation. In: CVPR, IEEE (2010) 3280–3287
21.Yao, J., Fidler, S., Urtasun, R.: Describing the Scene as a Whole: Joint Object Detection,
Scene Classiﬁcation and Semantic Segmentation. In: CVPR. (2012) 702–709


key: chroma:document
string_value:
Scene Classiﬁcation and Semantic Segmentation. In: CVPR. (2012) 702–709
22.Wojek, C., Schiele, B.: A dynamic conditional random ﬁeld model for joint labeling of object
and scene classes. In: ECCV , Springer (2008) 733–747
23.Lin, G., Shen, C., Reid, I., van den Hengel, A.: Deeply learning the messages in message
passing inference. In: NIPS. (2015) 361–369
16 Arnab et al.
24.Yang, Y., Hallman, S., Ramanan, D., Fowlkes, C.C.: Layered object models for image
segmentation. PAMI (2012)
25.Sun, M., Kim, B.s., Kohli, P., Savarese, S.: Relating things and stuff via objectproperty
interactions. PAMI 36(7) (2014) 1370–1383
26.Carreira, J., Caseiro, R., Batista, J., Sminchisescu, C.: Semantic segmentation with secondorder pooling. In: ECCV . (2012) 430–443
27.Farabet, C., Couprie, C., Najman, L., LeCun, Y.: Learning hierarchical features for scene


key: chroma:document
string_value:
27.Farabet, C., Couprie, C., Najman, L., LeCun, Y.: Learning hierarchical features for scene
labeling. PAMI (2013)
28.Dai, J., He, K., Sun, J.: Convolutional feature masking for joint object and stuff segmentation.
CVPR (2015)
29.Tompson, J.J., Jain, A., LeCun, Y ., Bregler, C.: Joint training of a convolutional network and
a graphical model for human pose estimation. In: NIPS. (2014) 1799–1807
30.Deng, Z., Zhai, M., Chen, L., Liu, Y., Muralidharan, S., Roshtkhari, M.J., Mori, G.: Deep
structured models for group activity recognition. In: BMVC. (2015)
31.Ionescu, C., Vantzos, O., Sminchisescu, C.: Matrix backpropagation for deep networks with
structured layers. In: ICCV . (2015) 2965–2973
32.Domke, J.: Learning graphical model parameters with approximate marginal inference. PAMI
(2013)
33.Kr¨ahenb ¨uhl, P., Koltun, V.: Parameter learning and convergent inference for dense random


key: chroma:document
string_value:
(2013)
33.Kr¨ahenb ¨uhl, P., Koltun, V.: Parameter learning and convergent inference for dense random
ﬁelds. In: ICML. (2013)
34.Ross, S., Munoz, D., Hebert, M., Bagnell, J.A.: Learning message-passing inference machines
for structured prediction. In: CVPR. (2011)
35.Dai, J., He, K., Sun, J.: Boxsup: Exploiting bounding boxes to supervise convolutional
networks for semantic segmentation. In: ICCV . (2015)
36. Girshick, R.: Fast r-cnn. In: ICCV . (2015)
37.Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: Towards real-time object detection with
region proposal networks. In: NIPS. (2015)
38.Rother, C., Kolmogorov, V., Blake, A.: Grabcut: Interactive foreground extraction using
iterated graph cuts. ACM TOG (2004)
39.Felzenszwalb, P.F., Huttenlocher, D.P.: Efﬁcient graph-based image segmentation. IJCV
(2004)


key: chroma:document
string_value:
(2004)
40.Achanta, R., Shaji, A., Smith, K., Lucchi, A., Fua, P., Susstrunk, S.: Slic superpixels compared
to state-of-the-art superpixel methods. PAMI 34(11) (2012) 2274–2282
41. Kohli, P., Kumar, M.P., Torr, P.H.: P3 & beyond: Solving energies with higher order cliques.
In: CVPR. (2007)
42.Baqu, P., Bagautdinov, T., Fleuret, F., Fua, P.: Principled Parallel Mean-Field Inference for
Discrete Random Fields. In: CVPR. (2016)
43.Hariharan, B., Arbel ´aez, P., Bourdev, L., Maji, S., Malik, J.: Semantic contours from inverse
detectors. In: ICCV , IEEE (2011) 991–998
44.Lin, T.Y ., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Doll ´ar, P., Zitnick, C.L.:
Microsoft coco: Common objects in context. In: ECCV . Springer (2014) 740–755


key: chroma:document
string_value:
Microsoft coco: Common objects in context. In: ECCV . Springer (2014) 740–755
45.Kokkinos, I.: Pushing the boundaries of boundary detection using deep learning. In: ICLR.
(2016)
46.Yu, F., Koltun, V .: Multi-scale context aggregation by dilated convolutions. In: ICLR. (2016)
47.Chen, L.C., Yang, Y ., Wang, J., Xu, W., Yuille, A.L.: Attention to scale: Scale-aware semantic
image segmentation. In: CVPR. (2016)
48.Papandreou, G., Chen, L., Murphy, K., Yuille, A.L.: Weakly- and semi-supervised learning of
a DCNN for semantic image segmentation. In: ICCV . (2015)
49.Liu, W., Rabinovich, A., Berg, A.C.: Parsenet: Looking wider to see better. arXiv preprint
arXiv:1506.04579 (2015)
Higher Order Conditional Random Fields in Deep Neural Networks 17
50.Mottaghi, R., Chen, X., Liu, X., Cho, N.G., Lee, S.W., Fidler, S., Urtasun, R., et al.: The


key: chroma:document
string_value:
role of context for object detection and semantic segmentation in the wild. In: CVPR, IEEE
(2014) 891–898
51.Noh, H., Hong, S., Han, B.: Learning deconvolution network for semantic segmentation. In:
ICCV . (2015)
52.Mostajabi, M., Yadollahpour, P., Shakhnarovich, G.: Feedforward semantic segmentation
with zoom-out features. In: CVPR. (2015)
53.Dong, J., Chen, Q., Yan, S., Yuille, A.: Towards uniﬁed object detection and semantic
segmentation. In: ECCV . (2014) 299–314
54.Carreira, J., Caseiro, R., Batista, J., Sminchisescu, C.: Free-form region description with
second-order pooling. PAMI (2014)
Appendix
Appendix A of this supplementary material presents the derivatives of the mean ﬁeld
updates which we use for inference in our Conditional Random Field (CRF). Appendix
B shows detailed qualitative results for the experiments described in our main paper.
A Derivatives of Mean Field Updates
The pseudocode for the mean ﬁeld inference algorithm with latent Ydetection variables


key: chroma:document
string_value:
A Derivatives of Mean Field Updates
The pseudocode for the mean ﬁeld inference algorithm with latent Ydetection variables
is shown below in Algorithm 1. We use the same notation used in the main paper.
Algorithm 1 Mean Field Inference
Q0(Xi=l)←1
Ziexp/parenleftbig
−ψU
i(l)/parenrightbig
,∀i,l
Q0(Yd=b)←sb
d(1−sd)(1−b),∀d,b
⊿Initialisation
fort = 0 :T−1do
Et(Xi=l)←UnaryUpdate + PairwiseUpdate +
DetectionUpdate + SuperpixelUpdate, ∀i,l
Et(Yd=b)←YUnaryUpdate + Y DetectionUpdate
⊿Mean ﬁeld updates
Qt+1(Xi=l)←1
Ziexp/parenleftbig
−Et(Xi=l)/parenrightbig
,∀i,l
Qt+1(Yd=b)←1
Zdexp/parenleftbig
−Et(Yd=b)/parenrightbig
,∀d,b
⊿Normalising
end for


key: chroma:document
string_value:
Zdexp/parenleftbig
−Et(Yd=b)/parenrightbig
,∀d,b
⊿Normalising
end for
For the explicit forms of the UnaryUpdate andPairwiseUpdate above, and their
differentials, we refer the reader to [10] and discuss the terms DetectionUpdate and
SuperpixelUpdate in detail below.
Let us assume that only one object detection of the form (ld,sd,Fd)is available for
the image under consideration. When multiple detections are present, simply a summation of the updates and differentials discussed below apply. Therefore, no generality
18 Arnab et al.
is lost with this assumption. Similarly, we can assume that only one superpixel clique
{Xs}is present, without a loss of generality.
Assuming that pixel iin Algorithm 1 belongs to Fd, Eq. (6) in the main paper
described the exact form of DetectionUpdate . Similarly, assuming that pixel ibelongs
to{Xs}Eq. (8) described the form of SuperpixelUpdate .
LetLdenote the value of the loss function calculated at the output of the deep
network. This could be the softmax loss or any other appropriate loss function. During
backpropagation, we get the error signal∂L


key: chroma:document
string_value:
network. This could be the softmax loss or any other appropriate loss function. During
backpropagation, we get the error signal∂L
∂QTat the output of the mean ﬁeld inference.
Using this error information, we need to compute the derivative of the loss Lwith respect
to theXunaries and various CRF parameters. Note that, if we compute the relevent
differentials for only one iteration of the mean ﬁeld algorithm, it is possible to calculate
them for multiple iterations using the recurrent behaviour of the iterations.
Note that, by looking at Normalising step of Algorithm 1, it is trivial to calculate
∂Qt+1
∂Et. Therefore, we can then calculate∂L
∂Etusing the chain rule. This is same as
backpropagation of the usual softmax operation in a deep network (up to a negative sign).
Using this observation we can calculate the necessary differentials to take the forms
shown below:
∂L
∂wDet=sd
ndnd/summationdisplay
i=1/parenleftBig∂L
Et(X(i)
d=ld)Qt(Yd= 1) + (9)
/summationdisplay
l/prime/negationslash=ld∂L


key: chroma:document
string_value:
d=ld)Qt(Yd= 1) + (9)
/summationdisplay
l/prime/negationslash=ld∂L
∂Et(X(i)
d=l/prime)Qt(Yd= 1)/parenrightBig
+
∂L
∂Et(Yd= 0)sd
ndnd/summationdisplay
i=1Qt(X(i)
d=ld) +
∂L
∂Et(Yd= 1)sd
ndnd/summationdisplay
i=1/parenleftBig
1−Qt(X(i)
d=ld)/parenrightBig
∂L
∂Qt(X(i)
d=ld)=wDet∂L
∂Et(Yd= 0)−wDet∂L
∂Et(Yd= 1)(10)
∂L
∂Qt(Yd= 0)=wDetsd
ndnd/summationdisplay
i=1/parenleftBigg
∂L
Et(X(i)
d=ld)/parenrightBigg
(11)
∂L
∂Qt(Yd= 1)=wDetsd
ndnd/summationdisplay


key: chroma:document
string_value:
d=ld)/parenrightBigg
(11)
∂L
∂Qt(Yd= 1)=wDetsd
ndnd/summationdisplay
i=1/summationdisplay
l/negationslash=ld/parenleftBigg
∂L
∂Et(X(i)
d=l/prime)/parenrightBigg
(12)
∂L
∂wLow(l)=/summationdisplay
i∈s
∂L
∂Et(X(i)
s=l)/productdisplay
j∈c,j/negationslash=iQt(Xj=l)
 (13)
Higher Order Conditional Random Fields in Deep Neural Networks 19
∂L
∂wHigh=/summationdisplay
i∈s/summationdisplay
l∈L
∂L
∂Et(X(i)
s=l)
1−/productdisplay
j∈c,j/negationslash=iQt(Xj=l)

 (14)
Effect of the superpixel potentials on the derivatives∂L


key: chroma:document
string_value:

 (14)
Effect of the superpixel potentials on the derivatives∂L
∂Qt(Xi=l)were negligible.
Therefore, we ignored them in our calculations.
B Additional Experimental Results
Table 5 presents more detailed results of our method, and that of other state-of-the-art
techniques, on the PASCAL VOC 2012 test set. In particular, we present the accuracy
for every class in the VOC test set. Note that our per-class accuracy improves over our
baseline, CRF-RNN [10], for all of the 20 classes in PASCAL VOC.
Figure 6 shows more sample results of our system, compared to our baseline, CRFas-RNN [10]. Figure 7 shows examples of failure cases of our method. Figure 8 examines
the effect of each of our potentials. Finally, Figure 9 shows a qualitative comparison
between the output of our system and other current methods on the PASCAL VOC 2012
test set.
20 Arnab et al.Table 5: Comparison of the mean Intersection over Union (IoU) accuracy of our approach and other state-of-the-art methods on the Pascal
VOC 2012 test set. Scores for other methods were taken from the original authors’ publications.
Methods trained with
COCOMean


key: chroma:document
string_value:
VOC 2012 test set. Scores for other methods were taken from the original authors’ publications.
Methods trained with
COCOMean
IoU(%)aeroplanebike bird boat bottle bus car cat chair cow table dog horse mbikepersonplant sheep sofa train tv
Our method 77.9 92.5 59.1 90.3 70.6 74.4 92.4 84.1 88.3 36.8 85.6 67.1 85.1 86.9 88.2 82.6 62.6 85.0 56.2 81.9 72.5
DPN [9] 77.5 89.0 61.6 87.7 66.8 74.7 91.2 84.3 87.6 36.5 86.3 66.1 84.4 87.8 85.6 85.4 63.6 87.3 61.3 79.4 66.4
Super Bound. [45] 75.7 90.3 37.9 89.6 67.8 74.6 89.3 84.1 89.1 35.8 83.6 66.2 82.9 81.7 85.6 84.6 60.3 84.8 60.7 78.3 68.3


key: chroma:document
string_value:
Dilated Conv. [46] 75.3 91.7 39.6 87.8 63.1 71.8 89.7 82.9 89.8 37.2 84.0 63.0 83.3 89.0 83.8 85.1 56.8 87.6 56.0 80.2 64.7
BoxSup [35] 75.2 89.8 38.0 89.2 68.9 68.0 89.6 83.0 87.7 34.4 83.6 67.1 81.5 83.7 85.2 83.5 58.6 84.9 55.8 81.2 70.7
Attention [47] 75.1 92.0 41.2 87.8 57.2 72.7 92.8 85.9 90.5 30.5 78.0 62.8 85.8 85.3 87.2 85.6 57.7 85.1 56.5 83.0 65.0
CRF-as-RNN [10] 74.7 90.4 55.3 88.7 68.4 69.8 88.3 82.4 85.1 32.6 78.5 64.4 79.6 81.9 86.4 81.8 58.6 82.4 53.5 77.4 70.1


key: chroma:document
string_value:
WSSL [48] 73.9 89.2 46.7 88.5 63.5 68.4 87.0 81.2 86.3 32.6 80.7 62.4 81.0 81.3 84.3 82.1 56.2 84.6 58.2 76.2 67.2
DeepLab [12] 72.7 89.1 38.3 88.1 63.3 69.7 87.1 83.1 85.0 29.3 76.5 56.5 79.8 77.9 85.8 82.4 57.4 84.3 54.9 80.5 64.1
Methods trained
without COCO
Our method 73.9 89.3 40.0 81.6 65.1 71.7 90.1 81.3 85.7 32.4 82.1 62.2 82.6 83.7 84.5 81.1 60.8 85.2 49.6 80.0 69.9
DPN [9] 74.1 87.7 59.4 78.4 64.9 70.3 89.3 83.5 86.1 31.7 79.9 62.6 81.9 80.0 83.5 82.3 60.5 83.2 53.4 77.9 65.0


key: chroma:document
string_value:
DeconvNet [51] 72.5 89.9 39.3 79.7 63.9 68.2 87.4 81.2 86.1 28.5 77.0 62.0 79.0 80.3 83.6 80.2 58.8 83.4 54.3 80.7 65.0
CRF-as-RNN [10] 72.0 87.5 39.0 79.7 64.2 68.3 87.6 80.8 84.4 30.4 78.2 60.4 80.5 77.8 83.1 80.6 59.5 82.8 47.8 78.3 67.1
DeepLab [12] 71.6 84.4 54.5 81.5 63.6 65.9 85.1 79.1 83.4 30.7 74.1 59.8 79.0 76.1 83.2 80.8 59.7 82.2 50.4 73.1 63.7
Piecewise [11] 70.7 87.5 37.7 75.8 57.4 72.3 88.4 82.6 80.0 33.4 71.5 55.0 79.3 78.4 81.3 82.7 56.1 79.8 48.6 77.1 66.3


key: chroma:document
string_value:
Zoomout [52] 69.6 85.6 37.3 83.2 62.5 66.0 85.1 80.7 84.9 27.2 73.2 57.5 78.1 79.2 81.1 77.1 53.6 74.0 49.2 71.7 63.3
FCN-8s [3] 62.2 76.8 34.2 68.9 49.4 60.3 75.3 74.7 77.6 21.4 62.5 46.8 71.8 63.9 76.5 73.9 45.2 72.4 37.4 70.9 55.1
CFM [28] 61.8 75.7 26.7 69.5 48.8 65.6 81.0 69.2 73.3 30.0 68.7 51.5 69.1 68.1 71.7 67.5 50.4 66.5 44.4 58.9 53.5
NUS UDS [53] 50.0 67.0 24.5 47.2 45.0 47.9 65.3 60.6 58.5 15.5 50.8 37.4 45.8 59.9 62.0 52.7 40.8 48.2 36.8 53.1 45.6


key: chroma:document
string_value:
O2P [54] 47.8 64.0 27.3 54.1 39.2 48.7 56.6 57.7 52.5 14.2 54.8 29.6 42.2 58.0 54.8 50.2 36.6 58.6 31.6 48.4 38.6
Higher Order Conditional Random Fields in Deep Neural Networks 21
Input image CRF-as-RNN [10] Our method Ground truth
Fig. 6: Examples of images where our method has improved over our baseline,
CRF-as-RNN [10]. The input images have the detection bounding boxes overlaid on
them. Note that the method of [10] does not make use of this information. The improvements from our method are due to our detection potentials, as well as our superpixel
based potentials. Note that all images are from the reduced validation set of VOC 2012
and have not been trained on at all. Best viewed in color.
22 Arnab et al.
Input image CRF-as-RNN [10] Our method Ground truth
Fig. 7: Examples of failure cases where our method has performed poorly. The ﬁrst
row shows an example of how the detection of the person has now resulted in the sofa
being misclassiﬁed (although our system is able to reject the other false detection). Our


key: chroma:document
string_value:
row shows an example of how the detection of the person has now resulted in the sofa
being misclassiﬁed (although our system is able to reject the other false detection). Our
superpixel potentials have a tendency to remove spurious noise by enforcing consistency
within regions. However, as shown in the second row, sometimes the “noise” being
removed is actually the correct label. In the other cases, we are limited by our pixelwise
classiﬁcation unaries which are poor. Our superpixel and detection potentials are not
always able to compensate for this. Note that all images are from the reduced validation
set of VOC 2012 and have not been trained on at all. The input images have the detection
bounding boxes overlaid on them. Note that the method of [10] does not make use of
this information. Best viewed in color.
Higher Order Conditional Random Fields in Deep Neural Networks 23
Input image Pairwise onlySuperpixels
onlyDetections
onlyDetections
and
SuperpixelsGround truth
Fig. 8: Comparison of pairwise potentials, superpixel and pairwise potentials, detection and pairwise potentials, and a combination of all three (Row 1 and 2) These
are examples where superpixel potentials help to remove spurious noise in the output


key: chroma:document
string_value:
are examples where superpixel potentials help to remove spurious noise in the output
but detection potentials do not affect the result. The ﬁnal result still improves when
all potentials are combined. (Row 3) Detection potentials greatly improve the result
by recognising the train correctly (the pixelwise unaries are largest for “bus”). And
superpixels, when combined with detections, slightly improve the output. (Row 4) An
example where both superpixel and detection potentials improve the ﬁnal output. (Row 5)
A case where the superpixel worsens the result as, although the output is more consistent
among superpixel regions, some pixels have had their correct labels removed. However,
the correct detection improves the result, and the output of combining superpixel and
detection potentials is actually better than either potential in isolation. (Row 6) Here, the
detection (although correct) worsens the output due to its imprecise foreground mask.
Superpixel potentials also exacerbate the result, since the legs of the chair and the chair’s
shadow are confused to be part of the same superpixel region. However, when the two
potentials are combined, the result is slightly better than with only detection potentials.
24 Arnab et al.


key: chroma:document
string_value:
potentials are combined, the result is slightly better than with only detection potentials.
24 Arnab et al.
Input image FCN-8s [3] Deeplab [12] CRF-as-RNN [10] Our method Ground truth
Fig. 9: Qualitative comparison with other current methods. Sample results of our
method compared to other current techniques on VOC 2012. We reproduced the segmentation results of Deeplab from their original publication, whilst we reproduced the results
of FCN-8s and CRF-as-RNN from their publicly-available source code. Best viewed in
colour.


key: chroma:document
string_value:
Deep Speech 2: End-to-End Speech Recognition in
English and Mandarin
Baidu Research – Silicon Valley AI Lab
Dario Amodei, Rishita Anubhai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro,
Jingdong Chen, Mike Chrzanowski, Adam Coates, Greg Diamos, Erich Elsen, Jesse Engel,
Linxi Fan, Christopher Fougner, Tony Han, Awni Hannun, Billy Jun, Patrick LeGresley,
Libby Lin, Sharan Narang, Andrew Ng, Sherjil Ozair, Ryan Prenger, Jonathan Raiman,
Sanjeev Satheesh, David Seetapun, Shubho Sengupta, Yi Wang, Zhiqian Wang, Chong Wang,
Bo Xiao, Dani Yogatama, Jun Zhan, Zhenyao Zhu
Abstract
We show that an end-to-end deep learning approach can be used to recognize
either English or Mandarin Chinese speech—two vastly different languages. Because it replaces entire pipelines of hand-engineered components with neural networks, end-to-end learning allows us to handle a diverse variety of speech including noisy environments, accents and different languages. Key to our approach is
our application of HPC techniques, resulting in a 7x speedup over our previous


key: chroma:document
string_value:
our application of HPC techniques, resulting in a 7x speedup over our previous
system [26]. Because of this efﬁciency, experiments that previously took weeks
now run in days. This enables us to iterate more quickly to identify superior architectures and algorithms. As a result, in several cases, our system is competitive
with the transcription of human workers when benchmarked on standard datasets.
Finally, using a technique called Batch Dispatch with GPUs in the data center, we
show that our system can be inexpensively deployed in an online setting, delivering low latency when serving users at scale.
1 Introduction
Decades worth of hand-engineered domain knowledge has gone into current state-of-the-art automatic speech recognition (ASR) pipelines. A simple but powerful alternative solution is to train such
ASR models end-to-end, using deep learning to replace most modules with a single model [26]. We
present the second generation of our speech system that exempliﬁes the major advantages of endto-end learning. The Deep Speech 2 ASR pipeline approaches or exceeds the accuracy of Amazon
Mechanical Turk human workers on several benchmarks, works in multiple languages with little
modiﬁcation, and is deployable in a production setting. It thus represents a signiﬁcant step towards


key: chroma:document
string_value:
modiﬁcation, and is deployable in a production setting. It thus represents a signiﬁcant step towards
a single ASR system that addresses the entire range of speech recognition contexts handled by humans. Since our system is built on end-to-end deep learning, we can employ a spectrum of deep
learning techniques: capturing large training sets, training larger models with high performance
computing, and methodically exploring the space of neural network architectures. We show that
through these techniques we are able to reduce error rates of our previous end-to-end system [26] in
English by up to 43%, and can also recognize Mandarin speech with high accuracy.
One of the challenges of speech recognition is the wide range of variability in speech and acoustics.
As a result, modern ASR pipelines are made up of numerous components including complex feature
extraction, acoustic models, language and pronunciation models, speaker adaptation, etc. Building and tuning these individual components makes developing a new speech recognizer very hard,
especially for a new language. Indeed, many parts do not generalize well across environments or
languages and it is often necessary to support multiple application-speciﬁc systems in order to provide acceptable accuracy. This state of affairs is different from human speech recognition: people
Authorship order is alphabetical.


key: chroma:document
string_value:
Authorship order is alphabetical.
1arXiv:1512.02595v1  [cs.CL]  8 Dec 2015
have the innate ability to learn any language during childhood, using general skills to learn language.
After learning to read and write, most humans can transcribe speech with robustness to variation in
environment, speaker accent and noise, without additional training for the transcription task. To
meet the expectations of speech recognition users, we believe that a single engine must learn to be
similarly competent; able to handle most applications with only minor modiﬁcations and able to
learn new languages from scratch without dramatic changes. Our end-to-end system puts this goal
within reach, allowing us to approach or exceed the performance of human workers on several tests
in two very different languages: Mandarin and English.
Since Deep Speech 2 (DS2) is an end-to-end deep learning system, we can achieve performance
gains by focusing on three crucial components: the model architecture, large labeled training
datasets, and computational scale. This approach has also yielded great advances in other application areas such as computer vision and natural language. This paper details our contribution to
these three areas for speech recognition, including an extensive investigation of model architectures


key: chroma:document
string_value:
these three areas for speech recognition, including an extensive investigation of model architectures
and the effect of data and model size on recognition performance. In particular, we describe numerous experiments with neural networks trained with the Connectionist Temporal Classiﬁcation (CTC)
loss function [22] to predict speech transcriptions from audio. We consider networks composed of
many layers of recurrent connections, convolutional ﬁlters, and nonlinearities, as well as the impact
of a speciﬁc instance of Batch Normalization [63] (BatchNorm) applied to RNNs. We not only
ﬁnd networks that produce much better predictions than those in previous work [26], but also ﬁnd
instances of recurrent models that can be deployed in a production setting with no signiﬁcant loss in
accuracy.
Beyond the search for better model architecture, deep learning systems beneﬁt greatly from large
quantities of training data. We detail our data capturing pipeline that has enabled us to create larger
datasets than what is typically used to train speech recognition systems. Our English speech system
is trained on 11,940 hours of speech, while the Mandarin system is trained on 9,400 hours. We use
data synthesis to further augment the data during training.
Training on large quantities of data usually requires the use of larger models. Indeed, our models


key: chroma:document
string_value:
data synthesis to further augment the data during training.
Training on large quantities of data usually requires the use of larger models. Indeed, our models
have many more parameters than those used in our previous system. Training a single model at
these scales requires tens of exaFLOPs1that would require 3-6 weeks to execute on a single GPU.
This makes model exploration a very time consuming exercise, so we have built a highly optimized
training system that uses 8 or 16 GPUs to train one model. In contrast to previous large-scale training
approaches that use parameter servers and asynchronous updates [18, 10], we use synchronous SGD,
which is easier to debug while testing new ideas, and also converges faster for the same degree of
data parallelism. To make the entire system efﬁcient, we describe optimizations for a single GPU
as well as improvements to scalability for multiple GPUs. We employ optimization techniques
typically found in High Performance Computing to improve scalability. These optimizations include
a fast implementation of the CTC loss function on the GPU, and a custom memory allocator. We
also use carefully integrated compute nodes and a custom implementation of all-reduce to accelerate
inter-GPU communication. Overall the system sustains approximately 50 teraFLOP/second when
training on 16 GPUs. This amounts to 3 teraFLOP/second per GPU which is about 50% of peak


key: chroma:document
string_value:
training on 16 GPUs. This amounts to 3 teraFLOP/second per GPU which is about 50% of peak
theoretical performance. This scalability and efﬁciency cuts training times down to 3 to 5 days,
allowing us to iterate more quickly on our models and datasets.
We benchmark our system on several publicly available test sets and compare the results to our
previous end-to-end system [26]. Our goal is to eventually reach human-level performance not only
on speciﬁc benchmarks, where it is possible to improve through dataset-speciﬁc tuning, but on a
range of benchmarks that reﬂects a diverse set of scenarios. To that end, we have also measured
the performance of human workers on each benchmark for comparison. We ﬁnd that our system
outperforms humans in some commonly-studied benchmarks and has signiﬁcantly closed the gap in
much harder cases. In addition to public benchmarks, we show the performance of our Mandarin
system on internal datasets that reﬂect real-world product scenarios.
Deep learning systems can be challenging to deploy at scale. Large neural networks are computationally expensive to evaluate for each user utterance, and some network architectures are more


key: chroma:document
string_value:
Deep learning systems can be challenging to deploy at scale. Large neural networks are computationally expensive to evaluate for each user utterance, and some network architectures are more
easily deployed than others. Through model exploration, we ﬁnd high-accuracy, deployable network architectures, which we detail here. We also employ a batching scheme suitable for GPU
11 exaFLOP = 1018FLoating-point OPerations.
2
hardware called Batch Dispatch that leads to an efﬁcient, real-time implementation of our Mandarin
engine on production servers. Our implementation achieves a 98th percentile compute latency of 67
milliseconds, while the server is loaded with 10 simultaneous audio streams.
The remainder of the paper is as follows. We begin with a review of related work in deep learning,
end-to-end speech recognition, and scalability in Section 2. Section 3 describes the architectural and
algorithmic improvements to the model and Section 4 explains how to efﬁciently compute them. We
discuss the training data and steps taken to further augment the training set in Section 5. An analysis
of results for the DS2 system in English and Mandarin is presented in Section 6. We end with a
description of the steps needed to deploy DS2 to real users in Section 7.
2 Related Work


key: chroma:document
string_value:
description of the steps needed to deploy DS2 to real users in Section 7.
2 Related Work
This work is inspired by previous work in both deep learning and speech recognition. Feed-forward
neural network acoustic models were explored more than 20 years ago [7, 50, 19]. Recurrent neural networks and networks with convolution were also used in speech recognition around the same
time [51, 67]. More recently DNNs have become a ﬁxture in the ASR pipeline with almost all
state of the art speech work containing some form of deep neural network [42, 29, 17, 16, 43, 58].
Convolutional networks have also been found beneﬁcial for acoustic models [1, 53]. Recurrent
neural networks, typically LSTMs, are just beginning to be deployed in state-of-the art recognizers [24, 25, 55] and work well together with convolutional layers for the feature extraction [52].
Models with both bidirectional [24] and unidirectional recurrence have been explored as well.
End-to-end speech recognition is an active area of research, showing compelling results when used
to re-score the outputs of a DNN-HMM [23] and standalone [26]. Two methods are currently used to


key: chroma:document
string_value:
to re-score the outputs of a DNN-HMM [23] and standalone [26]. Two methods are currently used to
map variable length audio sequences directly to variable length transcriptions. The RNN encoderdecoder paradigm uses an encoder RNN to map the input to a ﬁxed length vector and a decoder
network to expand the ﬁxed length vector into a sequence of output predictions [11, 62]. Adding an
attentional mechanism to the decoder greatly improves performance of the system, particularly with
long inputs or outputs [2]. In speech, the RNN encoder-decoder with attention performs well both
in predicting phonemes [12] or graphemes [3, 8].
The other commonly used technique for mapping variable length audio input to variable length
output is the CTC loss function [22] coupled with an RNN to model temporal information. The CTCRNN model performs well in end-to-end speech recognition with grapheme outputs [23, 27, 26, 40].
The CTC-RNN model has also been shown to work well in predicting phonemes [41, 54], though
a lexicon is still needed in this case. Furthermore it has been necessary to pre-train the CTC-RNN
network with a DNN cross-entropy network that is fed frame-wise alignments from a GMM-HMM


key: chroma:document
string_value:
network with a DNN cross-entropy network that is fed frame-wise alignments from a GMM-HMM
system [54]. In contrast, we train the CTC-RNN networks from scratch without the need of framewise alignments for pre-training.
Exploiting scale in deep learning has been central to the success of the ﬁeld thus far [36, 38]. Training on a single GPU resulted in substantial performance gains [49], which were subsequently scaled
linearly to two [36] or more GPUs [15]. We take advantage of work in increasing individual GPU
efﬁciency for low-level deep learning primitives [9]. We build on the past work in using modelparallelism [15], data-parallelism [18] or a combination of the two [64, 26] to create a fast and
highly scalable system for training deep RNNs in speech recognition.
Data has also been central to the success of end-to-end speech recognition, with over 7000 hours
of labeled speech used in Deep Speech 1 (DS1) [26]. Data augmentation has been highly effective
in improving the performance of deep learning in computer vision [39, 56, 14]. This has also been
shown to improve speech systems [21, 26]. Techniques used for data augmentation in speech range


key: chroma:document
string_value:
shown to improve speech systems [21, 26]. Techniques used for data augmentation in speech range
from simple noise addition [26] to complex perturbations such as simulating changes to the vocal
tract length and rate of speech of the speaker [31, 35].
Existing speech systems can also be used to bootstrap new data collection. In one approach, the
authors use one speech engine to align and ﬁlter a thousand hours of read speech [46]. In another
approach, a heavy-weight ofﬂine speech recognizer is used to generate transcriptions for tens of
thousands of hours of speech [33]. This is then passed through a ﬁlter and used to re-train the recognizer, resulting in signiﬁcant performance gains. We draw inspiration from these past approaches in
3
bootstrapping larger datasets and data augmentation to increase the effective amount of labeled data
for our system.
3 Model Architecture
A simple multi-layer model with a single recurrent layer cannot exploit thousands of hours of labelled speech. In order to learn from datasets this large, we increase the model capacity via depth.
We explore architectures with up to 11 layers including many bidirectional recurrent layers and convolutional layers. These models have nearly 8 times the amount of computation per data example as
the models in Deep Speech 1 making fast optimization and computation critical. In order to optimize


key: chroma:document
string_value:
the models in Deep Speech 1 making fast optimization and computation critical. In order to optimize
these models successfully, we use Batch Normalization for RNNs and a novel optimization curriculum we call SortaGrad. We also exploit long strides between RNN inputs to reduce computation
per example by a factor of 3. This is helpful for both training and evaluation, though requires some
modiﬁcations in order to work well with CTC. Finally, though many of our research results make
use of bidirectional recurrent layers, we ﬁnd that excellent models exist using only unidirectional
recurrent layers—a feature that makes such models much easier to deploy. Taken together these
features allow us to tractably optimize deep RNNs and improve performance by more than 40% in
both English and Mandarin error rates over the smaller baseline models.
3.1 Preliminaries
Figure 1 shows the architecture of the DS2 system which at its core is similar to the previous DS1
system [26]: a recurrent neural network (RNN) trained to ingest speech spectrograms and generate
text transcriptions.
Let a single utterance x(i)and label y(i)be sampled from a training set X =


key: chroma:document
string_value:
text transcriptions.
Let a single utterance x(i)and label y(i)be sampled from a training set X =
f(x(1),y(1)), (x(2),y(2)),:::g. Each utterance, x(i), is a time-series of length T(i)where every
time-slice is a vector of audio features, x(i)
t,t= 0,:::,T(i)1. We use a spectrogram of power
normalized audio clips as the features to the system, so x(i)
t,pdenotes the power of the p’th frequency
bin in the audio frame at time t. The goal of the RNN is to convert an input sequence x(i)into a
ﬁnal transcription y(i). For notational convenience, we drop the superscripts and use xto denote a
chosen utterance and ythe corresponding label.
The outputs of the network are the graphemes of each language. At each output time-step t, the RNN
makes a prediction over characters, p(`tjx), where`tis either a character in the alphabet or the blank
symbol. In English we have `t2fa, b, c,:::,z,space ,apostrophe ,blankg, where we have added


key: chroma:document
string_value:
symbol. In English we have `t2fa, b, c,:::,z,space ,apostrophe ,blankg, where we have added
theapostrophe as well as a space symbol to denote word boundaries. For the Mandarin system the
network outputs simpliﬁed Chinese characters. We describe this in more detail in Section 3.9.
The RNN model is composed of several layers of hidden units. The architectures we experiment
with consist of one or more convolutional layers, followed by one or more recurrent layers, followed
by one or more fully connected layers.
The hidden representation at layer lis given byhlwith the convention that h0represents the input
x. The bottom of the network is one or more convolutions over the time dimension of the input. For
a context window of size c, thei-th activation at time-step tof the convolutional layer is given by
hl
t,i=f(wl
ihl1
tc:t+c) (1)
wheredenotes the element-wise product between the i-th ﬁlter and the context window of the


key: chroma:document
string_value:
tc:t+c) (1)
wheredenotes the element-wise product between the i-th ﬁlter and the context window of the
previous layers activations, and fdenotes a unary nonlinear function. We use the clipped rectiﬁedlinear (ReLU) function (x) = minfmaxfx, 0g, 20gas our nonlinearity. In some layers, usually
the ﬁrst, we sub-sample by striding the convolution by sframes. The goal is to shorten the number
of time-steps for the recurrent layers above.
Following the convolutional layers are one or more bidirectional recurrent layers [57]. The forward
in time !hland backward in time hlrecurrent layer activations are computed as
 !hl
t=g(hl1
t, !hl
t1)
 hl
t=g(hl1
t, hl
t+1)(2)
4
CTC
SpectrogramRecurrentorGRU(Bidirectional)1D or 2DInvariantConvolutionFully Connected
BatchNormalizationFigure 1: Architecture of the DS2 system used to train on both English and Mandarin speech. We explore


key: chroma:document
string_value:
BatchNormalizationFigure 1: Architecture of the DS2 system used to train on both English and Mandarin speech. We explore
variants of this architecture by varying the number of convolutional layers from 1 to 3 and the number of
recurrent or GRU layers from 1 to 7.
The two sets of activations are summed to form the output activations for the layer hl= !hl+ hl.
The function g()can be the standard recurrent operation
 !hl
t=f(Wlhl1
t+ !Ul !hl
t1+bl) (3)
whereWlis the input-hidden weight matrix, !Ulis the recurrent weight matrix and blis a bias term.
In this case the input-hidden weights are shared for both directions of the recurrence. The function
g()can also represent more complex recurrence operations such as the Long Short-Term Memory
(LSTM) units [30] and the gated recurrent units (GRU) [11].
After the bidirectional recurrent layers we apply one or more fully connected layers with
hl
t=f(Wlhl1
t+bl) (4)
The output layer Lis a softmax computing a probability distribution over characters given by
p(`t=kjx) =exp(wL
khL1


key: chroma:document
string_value:
The output layer Lis a softmax computing a probability distribution over characters given by
p(`t=kjx) =exp(wL
khL1
t)P
jexp(wL
jhL1
t)(5)
The model is trained using the CTC loss function [22]. Given an input-output pair (x,y)and the
current parameters of the network , we compute the loss function L(x,y;)and its derivative with
respect to the parameters of the network rL(x,y;). This derivative is then used to update the
network parameters through the backpropagation through time algorithm.
In the following subsections we describe the architectural and algorithmic improvements made relative to DS1 [26]. Unless otherwise stated these improvements are language agnostic. We report
results on an English speaker held out development set which is an internal dataset containing 2048
utterances of primarily read speech. All models are trained on datasets described in Section 5.
We report Word Error Rate (WER) for the English system and Character Error Rate (CER) for the
Mandarin system. In both cases we integrate a language model in a beam search decoding step as
described in Section 3.8.
5
Architecture Hidden Units Train Dev
Baseline BatchNorm Baseline BatchNorm


key: chroma:document
string_value:
described in Section 3.8.
5
Architecture Hidden Units Train Dev
Baseline BatchNorm Baseline BatchNorm
1 RNN, 5 total 2400 10.55 11.99 13.55 14.40
3 RNN, 5 total 1880 9.55 8.29 11.61 10.56
5 RNN, 7 total 1510 8.59 7.61 10.77 9.78
7 RNN, 9 total 1280 8.76 7.68 10.83 9.52
Table 1: Comparison of WER on a training and development set for various depths of RNN, with and without
BatchNorm. The number of parameters is kept constant as the depth increases, thus the number of hidden units
per layer decreases. All networks have 38 million parameters. The architecture “M RNN, N total” implies 1
layer of 1D convolution at the input, M consecutive bidirectional RNN layers, and the rest as fully-connected
layers with N total layers in the network.
3.2 Batch Normalization for Deep RNNs
To efﬁciently scale our model as we scale the training set, we increase the depth of the networks by
adding more hidden layers, rather than making each layer larger. Previous work has examined doing
so by increasing the number of consecutive bidirectional recurrent layers [24]. We explore Batch


key: chroma:document
string_value:
adding more hidden layers, rather than making each layer larger. Previous work has examined doing
so by increasing the number of consecutive bidirectional recurrent layers [24]. We explore Batch
Normalization (BatchNorm) as a technique to accelerate training for such networks [63] since they
often suffer from optimization issues.
Recent research has shown that BatchNorm improves the speed of convergence of recurrent nets,
without showing any improvement in generalization performance [37]. In contrast, we demonstrate
that when applied to very deep networks of simple RNNs on large data sets, batch normalization
substantially improves ﬁnal generalization error while greatly accelerating training.
In a typical feed-forward layer containing an afﬁne transformation followed by a non-linearity f(),
we insert a BatchNorm transformation by applying f(B(Wh))instead off(Wh+b), where
B(x) =
xE[x]
(Var[x] +)1=2+. (6)
The terms EandVar are the empirical mean and variance over a minibatch. The bias bof the
layer is dropped since its effect is cancelled by mean removal. The learnable parameters 
and
allow the layer to scale and shift each hidden unit as desired. The constant is small and positive,


key: chroma:document
string_value:
allow the layer to scale and shift each hidden unit as desired. The constant is small and positive,
and is included only for numerical stability. In our convolutional layers the mean and variance
are estimated over all the temporal output units for a given convolutional ﬁlter on a minibatch.
The BatchNorm transformation reduces internal covariate shift by insulating a given layer from
potentially uninteresting changes in the mean and variance of the layer’s input.
We consider two methods of extending BatchNorm to bidirectional RNNs [37]. A natural extension
is to insert a BatchNorm transformation immediately before every non-linearity. Equation 3 then
becomes !hl
t=f(B(Wlhl1
t+ !Ul !hl
t1)). (7)
In this case the mean and variance statistics are accumulated over a single time-step of the minibatch.
The sequential dependence between time-steps prevents averaging over all time-steps. We ﬁnd that
this technique does not lead to improvements in optimization. We also tried accumulating an average
over successive time-steps, so later time-steps are normalized over all present and previous timesteps. This also proved ineffective and greatly complicated backpropagation.


key: chroma:document
string_value:
over successive time-steps, so later time-steps are normalized over all present and previous timesteps. This also proved ineffective and greatly complicated backpropagation.
We ﬁnd that sequence-wise normalization [37] overcomes these issues. The recurrent computation
is given by !hl
t=f(B(Wlhl1
t) + !Ul !hl
t1). (8)
For each hidden unit, we compute the mean and variance statistics over all items in the minibatch
over the length of the sequence. Figure 2 shows that deep networks converge faster with sequencewise normalization. Table 1 shows that the performance improvement from sequence-wise normalization increases with the depth of the network, with a 12% performance difference for the deepest
network. When comparing depth, in order to control for model size we hold constant the total
6
50 100 150 200 250 300
Iteration ( ⇥103)2030405060Cost5-1 BN
5-1 No BN
9-7 BN
9-7 No BNFigure 2: Training curves of two models trained with and without BatchNorm. We start the plot after the ﬁrst
epoch of training as the curve is more difﬁcult to interpret due to the SortaGrad curriculum method mentioned
in Section 3.3
Train Dev


key: chroma:document
string_value:
epoch of training as the curve is more difﬁcult to interpret due to the SortaGrad curriculum method mentioned
in Section 3.3
Train Dev
Baseline BatchNorm Baseline BatchNorm
Not Sorted 10.71 8.04 11.96 9.78
Sorted 8.76 7.68 10.83 9.52
Table 2: Comparison of WER on a training and development set with and without SortaGrad, and with and
without batch normalization.
number of parameters and still see strong performance gains. We would expect to see even larger
improvements from depth if we held constant the number of activations per layer and added layers. We also ﬁnd that BatchNorm harms generalization error for the shallowest network just as it
converges slower for shallower networks.
The BatchNorm approach works well in training, but is difﬁcult to implement for a deployed ASR
system, since it is often necessary to evaluate a single utterance in deployment rather than a batch.
We ﬁnd that normalizing each neuron to its mean and variance over just the sequence degrades
performance. Instead, we store a running average of the mean and variance for the neuron collected
during training, and use these for evaluation in deployment [63]. Using this technique, we can


key: chroma:document
string_value:
performance. Instead, we store a running average of the mean and variance for the neuron collected
during training, and use these for evaluation in deployment [63]. Using this technique, we can
evaluate a single utterance at a time with better results than evaluating with a large batch.
3.3 SortaGrad
Training on examples of varying length pose some algorithmic challenges. One possible solution is
truncating backpropagation through time [68], so that all examples have the same sequence length
during training [52]. However, this can inhibit the ability to learn longer term dependencies. Other
works have found that presenting examples in order of difﬁculty can accelerate online learning [6,
70]. A common theme in many sequence learning problems including machine translation and
speech recognition is that longer examples tend to be more challenging [11].
The CTC cost function that we use implicitly depends on the length of the utterance,
L(x,y;) =logX
`2Align (x,y)TY
tpctc(`tjx;). (9)
where Align (x,y)is the set of all possible alignments of the characters of the transcription yto
frames of input xunder the CTC operator. In equation 9, the inner term is a product over time-steps


key: chroma:document
string_value:
frames of input xunder the CTC operator. In equation 9, the inner term is a product over time-steps
of the sequence, which shrinks with the length of the sequence since pctc(`tjx;)<1. This motivates a curriculum learning strategy we title SortaGrad. SortaGrad uses the length of the utterance
as a heuristic for difﬁculty, since long utterances have higher cost than short utterances.
7
Architecture Simple RNN GRU
5 layers, 1 Recurrent 14.40 10.53
5 layers, 3 Recurrent 10.56 8.00
7 layers, 5 Recurrent 9.78 7.79
9 layers, 7 Recurrent 9.52 8.19
Table 3: Comparison of development set WER for networks with either simple RNN or GRU, for various
depths. All models have batch normalization, one layer of 1D-invariant convolution, and approximately 38
million parameters.
In the ﬁrst training epoch, we iterate through the training set in increasing order of the length of
the longest utterance in the minibatch. After the ﬁrst epoch, training reverts back to a random order
over minibatches. Table 2 shows a comparison of training cost with and without SortaGrad on the


key: chroma:document
string_value:
over minibatches. Table 2 shows a comparison of training cost with and without SortaGrad on the
9 layer model with 7 recurrent layers. This effect is particularly pronounced for networks without
BatchNorm, since they are numerically less stable. In some sense the two techniques substitute for
one another, though we still ﬁnd gains when applying SortaGrad and BatchNorm together. Even
with BatchNorm we ﬁnd that this curriculum improves numerical stability and sensitivity to small
changes in training. Numerical instability can arise from different transcendental function implementations in the CPU and the GPU, especially when computing the CTC cost. This curriculum
gives comparable results for both implementations.
We suspect that these beneﬁts occur primarily because long utterances tend to have larger gradients,
yet we use a ﬁxed learning rate independent of utterance length. Furthermore, longer utterances are
more likely to cause the internal state of the RNNs to explode at an early stage in training.
3.4 Comparison of simple RNNs and GRUs
The models we have shown so far are simple RNNs that have bidirectional recurrent layers with the
recurrence for both the forward in time and backward in time directions modeled by Equation 3.


key: chroma:document
string_value:
recurrence for both the forward in time and backward in time directions modeled by Equation 3.
Current research in speech and language processing has shown that having a more complex recurrence can allow the network to remember state over more time-steps while making them more
computationally expensive to train [52, 8, 62, 2]. Two commonly used recurrent architectures are the
Long Short-Term Memory (LSTM) units [30] and the Gated Recurrent Units (GRU) [11], though
many other variations exist. A recent comprehensive study of thousands of variations of LSTM and
GRU architectures showed that a GRU is comparable to an LSTM with a properly initialized forget
gate bias, and their best variants are competitive with each other [32]. We decided to examine GRUs
because experiments on smaller data sets showed the GRU and LSTM reach similar accuracy for
the same number of parameters, but the GRUs were faster to train and less likely to diverge.
The GRUs we use are computed by
zt=(Wzxt+Uzht1+bz)
rt=(Wrxt+Urht1+br)
~ht=f(Whxt+rtUhht1+bh)
ht= (1zt)ht1+zt~ht(10)


key: chroma:document
string_value:
~ht=f(Whxt+rtUhht1+bh)
ht= (1zt)ht1+zt~ht(10)
where()is the sigmoid function, zandrrepresent the update andreset gates respectively, and
we drop the layer superscripts for simplicity. We differ slightly from the standard GRU in that we
multiply the hidden state ht1byUhprior to scaling by the reset gate. This allows for all operations
onht1to be computed in a single matrix multiplication. The output nonlinearity f()is typically
the hyperbolic tangent function tanh . However, we ﬁnd similar performance for tanh and clippedReLU nonlinearities and choose to use the clipped-ReLU for simplicity and uniformity with the rest
of the network.
Both GRU and simple RNN architectures beneﬁt from batch normalization and show strong results with deep networks. However, Table 3 shows that for a ﬁxed number of parameters, the GRU
architectures achieve better WER for all network depths. This is clear evidence of the long term
dependencies inherent in the speech recognition task present both within individual words and be8
Architecture Channels Filter dimension Stride Regular Dev Noisy Dev


key: chroma:document
string_value:
dependencies inherent in the speech recognition task present both within individual words and be8
Architecture Channels Filter dimension Stride Regular Dev Noisy Dev
1-layer 1D 1280 11 2 9.52 19.36
2-layer 1D 640, 640 5, 5 1, 2 9.67 19.21
3-layer 1D 512, 512, 512 5, 5, 5 1, 1, 2 9.20 20.22
1-layer 2D 32 41x11 2x2 8.94 16.22
2-layer 2D 32, 32 41x11, 21x11 2x2, 2x1 9.06 15.71
3-layer 2D 32, 32, 96 41x11, 21x11, 21x11 2x2, 2x1, 2x1 8.61 14.74
Table 4: Comparison of WER for various arrangements of convolutional layers. In all cases, the convolutions
are followed by 7 recurrent layers and 1 fully connected layer. For 2D-invariant convolutions the ﬁrst dimension is frequency and the second dimension is time. All models have BatchNorm, SortaGrad, and 35 million
parameters.
tween words. As we discuss in Section 3.8, even simple RNNs are able to implicitly learn a language


key: chroma:document
string_value:
parameters.
tween words. As we discuss in Section 3.8, even simple RNNs are able to implicitly learn a language
model due to the large amount of training data. Interestingly, the GRU networks with 5 or more recurrent layers do not signiﬁcantly improve performance. We attribute this to the thinning from 1728
hidden units per layer for 1 recurrent layer to 768 hidden units per layer for 7 recurrent layers, to
keep the total number of parameters constant.
The GRU networks outperform the simple RNNs in Table 3. However, in later results (Section 6) we
ﬁnd that as we scale up the model size, for a ﬁxed computational budget the simple RNN networks
perform slightly better. Given this, most of the remaining experiments use the simple RNN layers
rather than the GRUs.
3.5 Frequency Convolutions
Temporal convolution is commonly used in speech recognition to efﬁciently model temporal translation invariance for variable length utterances. This type of convolution was ﬁrst proposed for
neural networks in speech more than 25 years ago [67]. Many neural network speech models have a
ﬁrst layer that processes input frames with some context window [16, 66]. This can be viewed as a
temporal convolution with a stride of one.


key: chroma:document
string_value:
ﬁrst layer that processes input frames with some context window [16, 66]. This can be viewed as a
temporal convolution with a stride of one.
Additionally, sub-sampling is essential to make recurrent neural networks computationally tractable
with high sample-rate audio. The DS1 system accomplished this through the use of a spectrogram
as input and temporal convolution in the ﬁrst layer with a stride parameter to reduce the number of
time-steps [26].
Convolutions in frequency and time domains, when applied to the spectral input features prior to
any other processing, can slightly improve ASR performance [1, 53, 60]. Convolution in frequency
attempts to model spectral variance due to speaker variability more concisely than what is possible with large fully connected networks. Since spectral ordering of features is removed by fullyconnected and recurrent layers, frequency convolutions work better as the ﬁrst layers of the network.
We experiment with adding between one and three layers of convolution. These are both in the timeand-frequency domain (2D invariance) and in the time-only domain (1D invariance). In all cases we
use a same convolution, preserving the number of input features in both frequency and time. In
some cases, we specify a stride across either dimension which reduces the size of the output. We do


key: chroma:document
string_value:
use a same convolution, preserving the number of input features in both frequency and time. In
some cases, we specify a stride across either dimension which reduces the size of the output. We do
not explicitly control for the number of parameters, since convolutional layers add a small fraction
of parameters to our networks. All networks shown in Table 4 have about 35 million parameters.
We report results on two datasets—a development set of 2048 utterances (“Regular Dev”) and a
much noisier dataset of 2048 utterances (“Noisy Dev”) randomly sampled from the CHiME 2015
development datasets [4]. We ﬁnd that multiple layers of 1D-invariant convolutions provides a very
small beneﬁt. The 2D-invariant convolutions improve results substantially on noisy data, while
providing a small beneﬁt on clean data. The change from one layer of 1D-invariant convolution to
three layers of 2D-invariant convolution improves WER by 23.9% on the noisy development set.
9
Dev no LM Dev LM
Stride Unigrams Bigrams Unigrams Bigrams
2 14.93 14.56 9.52 9.66
3 15.01 15.60 9.65 10.06
4 18.86 14.84 11.92 9.93


key: chroma:document
string_value:
3 15.01 15.60 9.65 10.06
4 18.86 14.84 11.92 9.93
Table 5: Comparison of WER with different amounts of striding for unigram and bigram outputs on a model
with 1 layer of 1D-invariant convolution, 7 recurrent layers, and 1 fully connected layer. All models have
BatchNorm, SortaGrad, and 35 million parameters. The models are compared on a development set with and
without the use of a 5-gram language model.
3.6 Striding
In the convolutional layers, we apply a longer stride and wider context to speed up training as fewer
time-steps are required to model a given utterance. Downsampling the input sound (through FFT and
convolutional striding) reduces the number of time-steps and computation required in the following
layers, but at the expense of reduced performance.
In our Mandarin models, we employ striding in the straightforward way. However, in English,
striding can reduce accuracy simply because the output of our network requires at least one timestep per output character, and the number of characters in English speech per time-step is high
enough to cause problems when striding2. To overcome this, we can enrich the English alphabet


key: chroma:document
string_value:
enough to cause problems when striding2. To overcome this, we can enrich the English alphabet
with symbols representing alternate labellings like whole words, syllables or non-overlapping ngrams. In practice, we use non-overlapping bi-graphemes or bigrams, since these are simple to
construct, unlike syllables, and there are few of them compared to alternatives such as whole words.
We transform unigram labels into bigram labels through a simple isomorphism.
Non-overlapping bigrams shorten the length of the output transcription and thus allow for a decrease
in the length of the unrolled RNN. The sentence the cat sat with non-overlapping bigrams is segmented as [th,e,space ,ca,t,space ,sa,t]. Notice that for words with odd number of characters, the
last character becomes an unigram and space is treated as an unigram as well. This isomorphism
ensures that the same words are always composed of the same bigram and unigram tokens. The
output set of bigrams consists of all bigrams that occur in the training set.
In Table 5 we show results for both the bigram and unigram systems for various levels of striding,
with or without a language model. We observe that bigrams allow for larger strides without any


key: chroma:document
string_value:
with or without a language model. We observe that bigrams allow for larger strides without any
sacriﬁce in in the word error rate. This allows us to reduce the number of time-steps of the unrolled
RNN beneﬁting both computation and memory usage.
3.7 Row Convolution and Unidirectional Models
Bidirectional RNN models are challenging to deploy in an online, low-latency setting, because they
are built to operate on an entire sample, and so it is not possible to perform the transcription process
as the utterance streams from the user. We have found an unidirectional architecture that performs
as well as our bidirectional models. This allows us to use unidirectional, forward-only RNN layers
in our deployment system.
To accomplish this, we employ a special layer that we call row convolution, shown in Figure 3. The
intuition behind this layer is that we only need a small portion of future information to make an
accurate prediction at the current time-step. Suppose at time-step t, we use a future context of 
steps. We now have a feature matrix ht:t+= [ht,ht+1, ...,ht+]of sized(+ 1) . We deﬁne a


key: chroma:document
string_value:
parameter matrix Wof the same size as ht:t+. The activations rtfor the new layer at time-step t
are
2Chinese characters are more similar to English syllables than English characters. This is reﬂected in our
training data, where there are on average 14.1 characters/s in English, while only 3.3 characters/s in Mandarin.
Conversely, the Shannon entropy per character as calculated from occurrence in the training set, is less in
English due to the smaller character set—4.9 bits/char compared to 12.6 bits/char in Mandarin. This implies
that spoken Mandarin has a lower temporal entropy density, 41 bits/s compared to 58 bits/s, and can thus
more easily be temporally compressed without losing character information.
10
Recurrent layerRow conv layerhtht+1ht+2ht+3rt+3rt+2rt+1rtFigure 3: Row convolution architecture with future context size of 2
rt,i=+1X
j=1Wi,jht+j1,i, for 1id. (11)
Since the convolution-like operation in Eq. 11 is row oriented for both Wandht:t+, we call this
layer row convolution.


key: chroma:document
string_value:
Since the convolution-like operation in Eq. 11 is row oriented for both Wandht:t+, we call this
layer row convolution.
We place the row convolution layer above all recurrent layers. This has two advantages. First, this
allows us to stream all computation below the row convolution layer on a ﬁner granularity given little
future context is needed. Second, this results in better CER compared to the best bidirectional model
for Mandarin. We conjecture that the recurrent layers have learned good feature representations,
so the row convolution layer simply gathers the appropriate information to feed to the classiﬁer.
Results for a unidirectional Mandarin speech system with row convolution and a comparison to a
bidirectional model are given in Section 7 on deployment.
3.8 Language Model
We train our RNN Models over millions of unique utterances, which enables the network to learn a
powerful implicit language model. Our best models are quite adept at spelling, without any external
language constraints. Further, in our development datasets we ﬁnd many cases where our models can
implicitly disambiguate homophones—for example, “he expects the Japanese agent tosell it for two
hundred seventy ﬁve thousand dollars”. Nevertheless, the labeled training data is small compared


key: chroma:document
string_value:
hundred seventy ﬁve thousand dollars”. Nevertheless, the labeled training data is small compared
to the size of unlabeled text corpora that are available. Thus we ﬁnd that WER improves when we
supplement our system with a language model trained from external text.
We use an n-gram language model since they scale well to large amounts of unlabeled text [26].
For English, our language model is a Kneser-Ney smoothed 5-gram model with pruning that is
trained using the KenLM toolkit [28] on cleaned text from the Common Crawl Repository3. The
vocabulary is the most frequently used 400,000 words from 250 million lines of text, which produces
a language model with about 850 million n-grams. For Mandarin, the language model is a KneserNey smoothed character level 5-gram model with pruning that is trained on an internal text corpus
of 8 billion lines of text. This produces a language model with about 2 billion n-grams.
During inference we search for the transcription ythat maximizes Q(y)shown in Equation 12. This
is a linear combination of logprobabilities from the CTC trained network and language model, along
with a word insertion term [26].


key: chroma:document
string_value:
is a linear combination of logprobabilities from the CTC trained network and language model, along
with a word insertion term [26].
Q(y) = log(pctc(yjx)) +log(plm(y)) +word_count (y) (12)
The weightcontrols the relative contributions of the language model and the CTC network. The
weightencourages more words in the transcription. These parameters are tuned on a development
set. We use a beam search to ﬁnd the optimal transcription [27].
3http://commoncrawl.org
11
Language Architecture Dev no LM Dev LM
English 5-layer, 1 RNN 27.79 14.39
English 9-layer, 7 RNN 14.93 9.52
Mandarin 5-layer, 1 RNN 9.80 7.13
Mandarin 9-layer, 7 RNN 7.55 5.81
Table 6: Comparison of WER for English and CER for Mandarin with and without a language model. These
are simple RNN models with only one layer of 1D invariant convolution.
Table 6 shows that an external language model helps both English and Mandarin speech systems.
The relative improvement given by the language model drops from 48% to 36% in English and 27%


key: chroma:document
string_value:
Table 6 shows that an external language model helps both English and Mandarin speech systems.
The relative improvement given by the language model drops from 48% to 36% in English and 27%
to 23% in Mandarin, as we go from a model with 5 layers and 1 recurrent layer to a model with 9
layers and 7 recurrent layers. We hypothesize that the network builds a stronger implicit language
model with more recurrent layers.
The relative performance improvement from a language model is higher in English than in Mandarin.
We attribute this to the fact that a Chinese character represents a larger block of information than
an English character. For example, if we output directly to syllables or words in English, the model
would make fewer spelling mistakes and the language model would likely help less.
3.9 Adaptation to Mandarin
The techniques that we have described so far can be used to build an end-to-end Mandarin speech
recognition system that outputs Chinese characters directly. This precludes the need to construct a
pronunciation model, which is often a fairly involved component for porting speech systems to other
languages [59]. Direct output to characters also precludes the need to explicitly model language
speciﬁc pronunciation features. For example we do not need to model Mandarin tones explicitly, as
some speech systems must do [59, 45].
The only architectural changes we make to our networks are due to the characteristics of the Chinese


key: chroma:document
string_value:
some speech systems must do [59, 45].
The only architectural changes we make to our networks are due to the characteristics of the Chinese
character set. Firstly, the output layer of the network outputs about 6000 characters, which includes
the Roman alphabet, since hybrid Chinese-English transcripts are common. We incur an out of
vocabulary error at evaluation time if a character is not contained in this set. This is not a major
concern, as our test set has only 0.74% out of vocab characters.
We use a character level language model in Mandarin as words are not usually segmented in text.
The word insertion term of Equation 12 becomes a character insertion term. In addition, we ﬁnd that
the performance of the beam search during decoding levels off at a smaller beam size. This allows
us to use a beam size of 200 with a negligible degradation in CER. In Section 6.2, we show that
our Mandarin speech models show roughly the same improvements to architectural changes as our
English speech models.
4 System Optimizations
Our networks have tens of millions of parameters, and the training algorithm takes tens of singleprecision exaFLOPs to converge. Since our ability to evaluate hypotheses about our data and models depends on the ability to train models quickly, we built a highly optimized training system.


key: chroma:document
string_value:
This system has two main components—a deep learning library written in C ++, along with a highperformance linear algebra library written in both CUDA and C ++. Our optimized software, running
on dense compute nodes with 8 Titan X GPUs per node, allows us to sustain 24 single-precision
teraFLOP/second when training a single model on one node. This is 45% of the theoretical peak
computational throughput of each node. We also can scale to multiple nodes, as outlined in the next
subsection.
4.1 Scalability and Data-Parallelism
We use the standard technique of data-parallelism to train on multiple GPUs using synchronous
SGD. Our most common conﬁguration uses a minibatch of 512on8GPUs. Our training pipeline
12
binds one process to each GPU. These processes then exchange gradient matrices during the backpropagation by using all-reduce, which exchanges a matrix between multiple processes and sums
the result so that at the end, each process has a copy of the sum of all matrices from all processes.
We ﬁnd synchronous SGD useful because it is reproducible and deterministic. We have found
that the appearance of non-determinism in our system often signals a serious bug, and so having
reproducibility as a goal has greatly facilitates debugging. In contrast, asynchronous methods such


key: chroma:document
string_value:
that the appearance of non-determinism in our system often signals a serious bug, and so having
reproducibility as a goal has greatly facilitates debugging. In contrast, asynchronous methods such
as asynchronous SGD with parameter servers as found in Dean et al. [18] typically do not provide
reproducibility and are therefore more difﬁcult to debug. Synchronous SGD is simple to understand
and implement. It scales well as we add multiple nodes to the training process.
2021222324252627
GPUs211212213214215216217218219Time (seconds)5-3 (2560)
9-7 (1760)
Figure 4: Scaling comparison of two networks—a 5 layer model with 3 recurrent layers containing 2560
hidden units in each layer and a 9 layer model with 7 recurrent layers containing 1760 hidden units in each
layer. The times shown are to train 1 epoch. The 5 layer model trains faster because it uses larger matrices and
is more computationally efﬁcient.
Figure 4 shows that time taken to train one epoch halves as we double the number of GPUs that
we train on, thus achieving near-linear weak scaling. We keep the minibatch per GPU constant at
64 during this experiment, effectively doubling the minibatch as we double the number of GPUs.


key: chroma:document
string_value:
64 during this experiment, effectively doubling the minibatch as we double the number of GPUs.
Although we have the ability to scale to large minibatches, we typically use either 8 or 16 GPUs
during training with a minibatch of 512 or 1024, in order to converge to the best result.
Since all-reduce is critical to the scalability of our training, we wrote our own implementation of
the ring algorithm [48, 65] for higher performance and better stability. Our implementation avoids
extraneous copies between CPU and GPU, and is fundamental to our scalability. We conﬁgure
OpenMPI with the smcuda transport that can send and receive buffers residing in the memory of
two different GPUs by using GPUDirect. When two GPUs are in the same PCI root complex,
this avoids any unnecessary copies to CPU memory. This also takes advantage of tree-structured
interconnects by running multiple segments of the ring concurrently between neighboring devices.
We built our implementation using MPI send and receive, along with CUDA kernels for the elementwise operations.
Table 7 compares the performance of our all-reduce implementation with that provided by OpenMPI
version 1.8.5. We report the time spent in all-reduce for a full training run that ran for one epoch
on our English dataset using a 5 layer, 3 recurrent layer architecture with 2560 hidden units for all


key: chroma:document
string_value:
on our English dataset using a 5 layer, 3 recurrent layer architecture with 2560 hidden units for all
layers. In this table, we use a minibatch of 64 per GPU, expanding the algorithmic minibatch as we
scale to more GPUs. We see that our implementation is considerably faster than OpenMPI’s when
the communication is within a node (8 GPUs or less). As we increase the number of GPUs and
increase the amount of inter-node communication, the gap shrinks, although our implementation is
still 2-4X faster.
All of our training runs use either 8 or 16 GPUs, and in this regime, our all-reduce implementation
results in 2.5faster training for the full training run, compared to using OpenMPI directly. Optimizing all-reduce has thus resulted in important productivity beneﬁts for our experiments, and has
made our simple synchronous SGD approach scalable.
13
GPU OpenMPI Our Performance
all-reduce all-reduce Gain
4 55359.1 2587.4 21.4
8 48881.6 2470.9 19.8
16 21562.6 1393.7 15.5
32 8191.8 1339.6 6.1
64 1395.2 611.0 2.3
128 1602.1 422.6 3.8


key: chroma:document
string_value:
32 8191.8 1339.6 6.1
64 1395.2 611.0 2.3
128 1602.1 422.6 3.8
Table 7: Comparison of two different all-reduce implementations. All times are in seconds. Performance gain
is the ratio of OpenMPI all-reduce time to our all-reduce time.
Language Architecture CPU CTC Time GPU CTC Time Speedup
English 5-layer, 3 RNN 5888.12 203.56 28.9
Mandarin 5-layer, 3 RNN 1688.01 135.05 12.5
Table 8: Comparison of time spent in seconds in computing the CTC loss function and gradient in one epoch
for two different implementations. Speedup is the ratio of CPU CTC time to GPU CTC time.
4.2 GPU implementation of CTC loss function
Calculating the CTC loss function is more complicated than performing forward and back propagation on our RNN architectures. Originally, we transferred activations from the GPUs to the
CPU, where we calculated the loss function using an OpenMP parallelized implementation of CTC.
However, this implementation limited our scalability rather signiﬁcantly, for two reasons. Firstly,
it became computationally more signiﬁcant as we improved efﬁciency and scalability of the RNN


key: chroma:document
string_value:
it became computationally more signiﬁcant as we improved efﬁciency and scalability of the RNN
itself. Secondly, transferring large activation matrices between CPU and GPU required us to spend
interconnect bandwidth for CTC, rather than on transferring gradient matrices to allow us to scale
using data parallelism to more processors.
To overcome this, we wrote a GPU implementation of the CTC loss function. Our parallel implementation relies on a slight refactoring to simplify the dependences in the CTC calculation, as well
as the use of optimized parallel sort implementations from ModernGPU [5]. We give more details
of this parallelization in the Appendix.
Table 8 compares the performance of two CTC implementations. The GPU implementation saves
us 95 minutes per epoch in English, and 25 minutes in Mandarin. This reduces overall training time
by 10-20%, which is also an important productivity beneﬁt for our experiments.
4.3 Memory allocation
Our system makes frequent use of dynamic memory allocations to GPU and CPU memory, mainly
to store activation data for variable length utterances, and for intermediate results. Individual allocations can be very large; over 1 GB for the longest utterances. For these very large allocations
we found that CUDA’s memory allocator and even std::malloc introduced signiﬁcant overhead


key: chroma:document
string_value:
we found that CUDA’s memory allocator and even std::malloc introduced signiﬁcant overhead
into our application—over a 2x slowdown from using std::malloc in some cases. This is because
both cudaMalloc andstd::malloc forward very large allocations to the operating system or GPU
driver to update the system page tables. This is a good optimization for systems running multiple
applications, all sharing memory resources, but editing page tables is pure overhead for our system
where nodes are dedicated entirely to running a single model. To get around this limitation, we
wrote our own memory allocator for both CPU and GPU allocations. Our implementation follows
the approach of the last level shared allocator in jemalloc: all allocations are carved out of contiguous memory blocks using the buddy algorithm [34]. To avoid fragmentation, we preallocate all of
GPU memory at the start of training and subdivide individual allocations from this block. Similarly, we set the CPU memory block size that we forward to mmap to be substantially larger than
std::malloc , at 12GB.
14
Dataset Speech Type Hours
WSJ read 80
Switchboard conversational 300
Fisher conversational 2000
LibriSpeech read 960
Baidu read 5000
Baidu mixed 3600
Total 11940


key: chroma:document
string_value:
WSJ read 80
Switchboard conversational 300
Fisher conversational 2000
LibriSpeech read 960
Baidu read 5000
Baidu mixed 3600
Total 11940
Table 9: Summary of the datasets used to train DS2 in English. The Wall Street Journal (WSJ), Switchboard
and Fisher [13] corpora are all published by the Linguistic Data Consortium. The LibriSpeech dataset [46] is
available free on-line. The other datasets are internal Baidu corpora.
Most of the memory required for training deep recurrent networks is used to store activations through
each layer for use by back propagation, not to store the parameters of the network. For example,
storing the weights for a 70M parameter network with 9 layers requires approximately 280 MB of
memory, but storing the activations for a batch of 64, seven-second utterances requires 1.5 GB of
memory. TitanX GPUs include 12GB of GDDR5 RAM, and sometimes very deep networks can
exceed the GPU memory capacity when processing long utterances. This can happen unpredictably,
especially when the distribution of utterance lengths includes outliers, and it is desirable to avoid a
catastrophic failure when this occurs. When a requested memory allocation exceeds available GPU


key: chroma:document
string_value:
especially when the distribution of utterance lengths includes outliers, and it is desirable to avoid a
catastrophic failure when this occurs. When a requested memory allocation exceeds available GPU
memory, we allocate page-locked GPU-memory-mapped CPU memory using cudaMallocHost instead. This memory can be accessed directly by the GPU by forwarding individual memory transactions over PCIe at reduced bandwidth, and it allows a model to continue to make progress even
after encountering an outlier.
The combination of fast memory allocation with a fallback mechanism that allows us to slightly
overﬂow available GPU memory in exceptional cases makes the system signiﬁcantly simpler, more
robust, and more efﬁcient.
5 Training Data
Large-scale deep learning systems require an abundance of labeled training data. We have collected
an extensive training dataset for both English and Mandarin speech models, in addition to augmenting our training with publicly available datasets. In English we use 11,940 hours of labeled speech
data containing 8 million utterances summarized in Table 9. For the Mandarin system we use 9,400
hours of labeled audio containing 11 million utterances. The Mandarin speech data consists of internal Baidu corpora, representing a mix of read speech and spontaneous speech, in both standard
Mandarin and accented Mandarin.
5.1 Dataset Construction


key: chroma:document
string_value:
Mandarin and accented Mandarin.
5.1 Dataset Construction
Some of the internal English (3,600 hours) and Mandarin (1,400 hours) datasets were created from
raw data captured as long audio clips with noisy transcriptions. The length of these clips ranged from
several minutes to more than hour, making it impractical to unroll them in time in the RNN during
training. To solve this problem, we developed an alignment, segmentation and ﬁltering pipeline that
can generate a training set with shorter utterances and few erroneous transcriptions.
The ﬁrst step in the pipeline is to use an existing bidirectional RNN model trained with CTC to
align the transcription to the frames of audio. For a given audio-transcript pair, (x,y), we ﬁnd the
alignment that maximizes
`= arg max
`2Align (x,y)TY
tpctc(`tjx;). (13)
This is essentially a Viterbi alignment found using a RNN model trained with CTC. Since Equation 9
integrates over the alignment, the CTC loss function is never explicitly asked to produce an accurate
alignment. In principle, CTC could choose to emit all the characters of the transcription after some
15


key: chroma:document
string_value:
alignment. In principle, CTC could choose to emit all the characters of the transcription after some
15
ﬁxed delay and this can happen with unidirectional RNNs [54]. However, we found that CTC
produces an accurate alignment when trained with a bidirectional RNN.
Following the alignment is a segmentation step that splices the audio and the corresponding aligned
transcription whenever it encounters a long series of consecutive blank labels occurs, since this
usually denotes a stretch of silence. By tuning the number of consecutive blank s, we can tune the
length of the utterances generated. For the English speech data, we also require a space token to be
within the stretch of blank s in order to segment only on word boundaries. We tune the segmentation
to generate utterances that are on average 7 seconds long.
The ﬁnal step in the pipeline removes erroneous examples that arise from a failed alignment. We
crowd source the ground truth transcriptions for several thousand examples. The word level edit
distance between the ground truth and the aligned transcription is used to produce a good orbad
label. The threshold for the word level edit distance is chosen such that the resulting WER of the
good portion of the development set is less than 5%. We then train a linear classiﬁer to accurately


key: chroma:document
string_value:
good portion of the development set is less than 5%. We then train a linear classiﬁer to accurately
predict bad examples given the input features generated from the speech recognizer. We ﬁnd the
following features useful: the raw CTC cost, the CTC cost normalized by the sequence length,
the CTC cost normalized by the transcript length, the ratio of the sequence length to the transcript
length, the number of words in the transcription and the number of characters in the transcription.
For the English dataset, we ﬁnd that the ﬁltering pipeline reduces the WER from 17% to 5% while
retaining more than 50% of the examples.
5.2 Data Augmentation
We augment our training data by adding noise to increase the effective size of our training data
and to improve our robustness to noisy speech [26]. Although the training data contains some
intrinsic noise, we can increase the quantity and variety of noise through augmentation. Too much
noise augmentation tends to make optimization difﬁcult and can lead to worse results, and too little
noise augmentation makes the system less robust to low signal-to-noise speech. We ﬁnd that a good
balance is to add noise to 40% of the utterances that are chosen at random. The noise source consists


key: chroma:document
string_value:
balance is to add noise to 40% of the utterances that are chosen at random. The noise source consists
of several thousand hours of randomly selected audio clips combined to produce hundreds of hours
of noise.
5.3 Scaling Data
Our English and Mandarin corpora are substantially larger than those commonly reported in speech
recognition literature. In Table 10, we show the effect of increasing the amount of labeled training
data on WER. This is done by randomly sampling the full dataset before training. For each dataset,
the model was trained for up to 20 epochs though usually early-stopped based on the error on a held
out development set. We note that the WER decreases with a power law for both the regular and
noisy development sets. The WER decreases by 40% relative for each factor of 10 increase in
training set size. We also observe a consistent gap in WER ( 60% relative) between the regular and
noisy datasets, implying that more data beneﬁts both cases equally.
This implies that a speech system will continue to improve with more labeled training data. We
hypothesize that equally as important as increasing raw number of hours is increasing the number
of speech contexts that are captured in the dataset. A context can be any property that makes speech
unique including different speakers, background noise, environment, and microphone hardware.


key: chroma:document
string_value:
of speech contexts that are captured in the dataset. A context can be any property that makes speech
unique including different speakers, background noise, environment, and microphone hardware.
While we do not have the labels needed to validate this claim, we suspect that measuring WER as
a function of speakers in the dataset would lead to much larger relative gains than simple random
sampling.
6 Results
To better assess the real-world applicability of our speech system, we evaluate on a wide range of
test sets. We use several publicly available benchmarks and several test sets collected internally.
Together these test sets represent a wide range of challenging speech environments including low
signal-to-noise ratios (noisy and far-ﬁeld), accented, read, spontaneous and conversational speech.
16
Fraction of Data Hours Regular Dev Noisy Dev
1% 120 29.23 50.97
10% 1200 13.80 22.99
20% 2400 11.65 20.41
50% 6000 9.51 15.90
100% 12000 8.46 13.59
Table 10: Comparison of English WER for Regular and Noisy development sets on increasing training dataset
size. The architecture is a 9-layer model with 2 layers of 2D-invariant convolution and 7 recurrent layers with
68M parameters.


key: chroma:document
string_value:
size. The architecture is a 9-layer model with 2 layers of 2D-invariant convolution and 7 recurrent layers with
68M parameters.
All models are trained for 20 epochs on either the full English dataset, described in Table 9, or
the full Mandarin dataset described in Section 5. We use stochastic gradient descent with Nesterov
momentum [61] along with a minibatch of 512 utterances. If the norm of the gradient exceeds
a threshold of 400, it is rescaled to 400 [47]. The model which performs the best on a held-out
development set during training is chosen for evaluation. The learning rate is chosen from [1
104, 6104]to yield fastest convergence and annealed by a constant factor of 1.2 after each
epoch. We use a momentum of 0.99 for all models.
The language models used are those described in Section 3.8. The decoding parameters from Equation 12 are tuned on a held-out development set. We use a beam size of 500 for the English decoder
and a beam size of 200 for the Mandarin decoder.
6.1 English
The best DS2 model has 11 layers with 3 layers of 2D convolution, 7 bidirectional recurrent layers,


key: chroma:document
string_value:
6.1 English
The best DS2 model has 11 layers with 3 layers of 2D convolution, 7 bidirectional recurrent layers,
a fully-connected output layer along with Batch Normalization. The ﬁrst layer outputs to bigrams
with a temporal stride of 3. By comparison the DS1 model has 5 layers with a single bidirectional
recurrent layer and it outputs to unigrams with a temporal stride of 2 in the ﬁrst layer. We report
results on several test sets for both the DS2 and DS1 model. We do not tune or adapt either model
to any of the speech conditions in the test sets. Language model decoding parameters are set once
on a held-out development set.
To put the performance of our system in context, we benchmark most of our results against human
workers, since speech recognition is an audio perception and language understanding problem that
humans excel at. We obtain a measure of human level performance by paying workers from Amazon
Mechanical Turk to hand-transcribe all of our test sets. Two workers transcribe the same audio clip,
that is typically about 5 seconds long, and we use the better of the two transcriptions for the ﬁnal
WER calculation. They are free to listen to the audio clip as many times as they like. These workers


key: chroma:document
string_value:
WER calculation. They are free to listen to the audio clip as many times as they like. These workers
are mostly based in the United States, and on average spend about 27 seconds per transcription.
The hand-transcribed results are compared to the existing ground truth to produce a WER. While
the existing ground truth transcriptions do have some label error, this is rarely more than 1%. This
implies that disagreement between the ground truth transcripts and the human level transcripts is a
good heuristic for human level performance.
6.1.1 Model Size
Our English speech training set is substantially larger than the size of commonly used speech
datasets. Furthermore, the data is augmented with noise synthesis. To get the best generalization
error, we expect that the model size must increase to fully exploit the patterns in the data. In Section 3.2 we explored the effect of model depth while ﬁxing the number of parameters. In contrast,
here we show the effect of varying model size on the performance of the speech system. We only
vary the size of each layer, while keeping the depth and other architectural parameters constant. We
evaluate the models on the same Regular and Noisy development sets that we use in Section 3.5.
The models in Table 11 differ from those in Table 3 in that we increase the the stride to 3 and output


key: chroma:document
string_value:
The models in Table 11 differ from those in Table 3 in that we increase the the stride to 3 and output
to bigrams. Because we increase the model size to as many as 100 million parameters, we ﬁnd that
an increase in stride is necessary for fast computation and memory constraints. However, in this
regime we note that the performance advantage of the GRU networks appears to diminish over the
17
Model size Model type Regular Dev Noisy Dev
18106GRU 10.59 21.38
38106GRU 9.06 17.07
70106GRU 8.54 15.98
70106RNN 8.44 15.09
100106GRU 7.78 14.17
100106RNN 7.73 13.06
Table 11: Comparing the effect of model size on the WER of the English speech system on both the regular and
noisy development sets. We vary the number of hidden units in all but the convolutional layers. The GRU model
has 3 layers of bidirectional GRUs with 1 layer of 2D-invariant convolution. The RNN model has 7 layers of
bidirectional simple recurrence with 3 layers of 2D-invariant convolution. Both models output bigrams with a
temporal stride of 3. All models contain approximately 35 million parameters and are trained with BatchNorm


key: chroma:document
string_value:
temporal stride of 3. All models contain approximately 35 million parameters and are trained with BatchNorm
and SortaGrad.
Test set DS1 DS2
Baidu Test 24.01 13.59
Table 12: Comparison of DS1 and DS2 WER on an internal test set of 3,300 examples. The test set contains a
wide variety of speech including accents, low signal-to-noise speech, spontaneous and conversational speech.
simple RNN. In fact, for the 100 million parameter networks the simple RNN performs better than
the GRU network and is faster to train despite the 2 extra layers of convolution.
Table 11 shows that the performance of the system improves consistently up to 100 million parameters. All further English DS2 results are reported with this same 100 million parameter RNN model
since it achieves the lowest generalization errors.
Table 12 shows that the 100 million parameter RNN model (DS2) gives a 43.4% relative improvement over the 5-layer model with 1 recurrent layer (DS1) on an internal Baidu dataset of 3,300
utterances that contains a wide variety of speech including challenging accents, low signal-to-noise
ratios from far-ﬁeld or background noise, spontaneous and conversational speech.
6.1.2 Read Speech


key: chroma:document
string_value:
ratios from far-ﬁeld or background noise, spontaneous and conversational speech.
6.1.2 Read Speech
Read speech with high signal-to-noise ratio is arguably the easiest large vocabulary for a continuous
speech recognition task. We benchmark our system on two test sets from the Wall Street Journal
(WSJ) corpus of read news articles. These are available in the LDC catalog as LDC94S13B and
LDC93S6B. We also take advantage of the recently developed LibriSpeech corpus constructed using
audio books from the LibriV ox project [46].
Table 13 shows that the DS2 system outperforms humans in 3 out of the 4 test sets and is competitive
on the fourth. Given this result, we suspect that there is little room for a generic speech system to
further improve on clean read speech without further domain adaptation.
Read Speech
Test set DS1 DS2 Human
WSJ eval’92 4.94 3.60 5.03
WSJ eval’93 6.94 4.98 8.08
LibriSpeech test-clean 7.89 5.33 5.83
LibriSpeech test-other 21.74 13.25 12.69
Table 13: Comparison of WER for two speech systems and human level performance on read speech.
18
Accented Speech
Test set DS1 DS2 Human


key: chroma:document
string_value:
Table 13: Comparison of WER for two speech systems and human level performance on read speech.
18
Accented Speech
Test set DS1 DS2 Human
V oxForge American-Canadian 15.01 7.55 4.85
V oxForge Commonwealth 28.46 13.56 8.15
V oxForge European 31.20 17.55 12.76
V oxForge Indian 45.35 22.44 22.15
Table 14: Comparing WER of the DS1 system to the DS2 system on accented speech.
Noisy Speech
Test set DS1 DS2 Human
CHiME eval clean 6.30 3.34 3.46
CHiME eval real 67.94 21.79 11.84
CHiME eval sim 80.27 45.05 31.33
Table 15: Comparison of DS1 and DS2 system on noisy speech. “CHiME eval clean” is a noise-free baseline.
The “CHiME eval real” dataset is collected in real noisy environments and the “CHiME eval sim” dataset has
similar noise synthetically added to clean speech. Note that we use only one of the six channels to test each
utterance.
6.1.3 Accented Speech
Our source for accented speech is the publicly available V oxForge ( http://www.voxforge.org )


key: chroma:document
string_value:
utterance.
6.1.3 Accented Speech
Our source for accented speech is the publicly available V oxForge ( http://www.voxforge.org )
dataset, which has clean speech read from speakers with many different accents. We group these
accents into four categories. The American-Canadian and Indian groups are self-explanatory. The
Commonwealth accent denotes speakers with British, Irish, South African, Australian and New
Zealand accents. The European group contains speakers with accents from countries in Europe that
do not have English as a ﬁrst language. We construct a test set from the V oxForge data with 1024
examples from each accent group for a total of 4096 examples.
Performance on these test sets is to some extent a measure of the breadth and quality of our training
data. Table 14 shows that our performance improved on all the accents when we include more
accented training data and use an architecture that can effectively train on that data. However human
level performance is still notably better than that of DS2 for all but the Indian accent.
6.1.4 Noisy Speech
We test our performance on noisy speech using the publicly available test sets from the recently
completed third CHiME challenge [4]. This dataset has 1320 utterances from the WSJ test set


key: chroma:document
string_value:
We test our performance on noisy speech using the publicly available test sets from the recently
completed third CHiME challenge [4]. This dataset has 1320 utterances from the WSJ test set
read in various noisy environments, including a bus, a cafe, a street and a pedestrian area. The
CHiME set also includes 1320 utterances with simulated noise from the same environments as well
as the control set containing the same utterances delivered by the same speakers in a noise-free
environment. Differences between results on the control set and the noisy sets provide a measure of
the network’s ability to handle a variety of real and synthetic noise conditions. The CHiME audio
has 6 channels and using all of them can provide substantial performance improvements [69]. We
use a single channel for all our results, since multi-channel audio is not pervasive on most devices.
Table 15 shows that DS2 substantially improves upon DS1, however DS2 is worse than human level
performance on noisy data. The relative gap between DS2 and human level performance is larger
when the data comes from a real noisy environment instead of synthetically adding noise to clean
speech.
6.2 Mandarin
In Table 16 we compare several architectures trained on the Mandarin Chinese speech, on a development set of 2000 utterances as well as a test set of 1882 examples of noisy speech. This development


key: chroma:document
string_value:
In Table 16 we compare several architectures trained on the Mandarin Chinese speech, on a development set of 2000 utterances as well as a test set of 1882 examples of noisy speech. This development
set was also used to tune the decoding parameters We see that the deepest model with 2D-invariant
19
convolution and BatchNorm outperforms the shallow RNN by 48% relative, thus continuing the
trend that we saw with the English system—multiple layers of bidirectional recurrence improves
performance substantially.
Architecture Dev Test
5-layer, 1 RNN 7.13 15.41
5-layer, 3 RNN 6.49 11.85
5-layer, 3 RNN + BatchNorm 6.22 9.39
9-layer, 7 RNN + BatchNorm + 2D conv 5.81 7.93
Table 16: Comparison of the improvements in DeepSpeech with architectural improvements. The development
and test sets are Baidu internal corpora. All the models in the table have about 80 million parameters each
We ﬁnd that our best Mandarin Chinese speech system transcribes short voice-query like utterances
better than a typical Mandarin Chinese speaker. To benchmark against humans we ran a test with
100 randomly selected utterances and had a group of 5 humans label all of them together. The group


key: chroma:document
string_value:
better than a typical Mandarin Chinese speaker. To benchmark against humans we ran a test with
100 randomly selected utterances and had a group of 5 humans label all of them together. The group
of humans had an error rate of 4.0% as compared to the speech systems performance of 3.7%. We
also compared a single human transcriber to the speech system on 250 randomly selected utterances.
In this case the speech system performs much better: 9.7% for the human compared to 5.7% for the
speech model.
7 Deployment
Real-world applications usually require a speech system to transcribe in real time or with relatively
low latency. The system used in Section 6.1 is not well-designed for this task, for several reasons.
First, since the RNN has several bidirectional layers, transcribing the ﬁrst part of an utterance requires the entire utterance to be presented to the RNN. Second, since we use a wide beam when
decoding with a language model, beam search can be expensive, particularly in Mandarin where the
number of possible next characters is very large (around 6000). Third, as described in Section 3, we
normalize power across an entire utterance, which again requires the entire utterance to be available
in advance.
We solve the power normalization problem by using some statistics from our training set to perform


key: chroma:document
string_value:
in advance.
We solve the power normalization problem by using some statistics from our training set to perform
an adaptive normalization of speech inputs during online transcription. We can solve the other
problems by modifying our network and decoding procedure to produce a model that performs
almost as well while having much lower latency. We focus on our Mandarin system since some
aspects of that system are more challenging to deploy (e.g. the large character set), but the same
techniques could also be applied in English.
In this section, latency refers to the computational latency of our speech system as measured from
the end of an utterance until the transcription is produced. This latency does not include data transmission over the internet, and does not measure latency from the beginning of an utterance until the
ﬁrst transcription is produced. We focus on latency from end of utterance to transcription because it
is important to applications using speech recognition.
7.1 Batch Dispatch
In order to deploy our relatively large deep neural networks at low latency, we have paid special attention to efﬁciency during deployment. Most internet applications process requests individually as
they arrive in the data center. This makes for a straightforward implementation where each request
can be managed by one thread. However, processing requests individually is inefﬁcient computationally, for two main reasons. Firstly, when processing requests individually, the processor must


key: chroma:document
string_value:
can be managed by one thread. However, processing requests individually is inefﬁcient computationally, for two main reasons. Firstly, when processing requests individually, the processor must
load all the weights of the network for each request. This lowers the arithmetic intensity of the workload, and tends to make the computation memory bandwidth bound, as it is difﬁcult to effectively
use on-chip caches when requests are presented individually. Secondly, the amount of parallelism
that can be exploited to classify one request is limited, making it difﬁcult to exploit SIMD or multicore parallelism. RNNs are especially challenging to deploy because evaluating RNNs sample by
20
0 1 2 3 4 5 6 7 8 9 10 11
Batch size0.00.10.20.30.4Probability10 streams
20 streams
30 streamsFigure 5: Probability that a request is processed in a batch of given size
sample relies on sequential matrix vector multiplications, which are bandwidth bound and difﬁcult
to parallelize.
To overcome these issues, we built a batching scheduler called Batch Dispatch that assembles
streams of data from user requests into batches before performing forward propagation on these
batches. In this case, there is a tradeoff between increased batch size, and consequently improved


key: chroma:document
string_value:
streams of data from user requests into batches before performing forward propagation on these
batches. In this case, there is a tradeoff between increased batch size, and consequently improved
efﬁciency, and increased latency. The more we buffer user requests to assemble a large batch, the
longer users must wait for their results. This places constraints on the amount of batching we can
perform.
We use an eager batching scheme that processes each batch as soon as the previous batch is completed, regardless of how much work is ready by that point. This scheduling algorithm has proved
to be the best at reducing end-user latency, despite the fact that it is less efﬁcient computationally,
since it does not attempt to maximize batch size.
Figure 5 shows the probability that a request is processed in a batch of given size for our production
system running on a single NVIDIA Quadro K1200 GPU, with 10-30 concurrent user requests. As
expected, batching works best when the server is heavily loaded: as load increases, the distribution
shifts to favor processing requests in larger batches. However, even with a light load of only 10
concurrent user requests, our system performs more than half the work in batches with at least 2
samples.
0 10 20 30 40
Number of concurrent streams050100latency (ms)50%ile
98%ile


key: chroma:document
string_value:
samples.
0 10 20 30 40
Number of concurrent streams050100latency (ms)50%ile
98%ile
Figure 6: Median and 98 percentile latencies as a function of server load
We see in Figure 6, that our system achieves a median latency of 44 ms, and a 98 percentile latency
of 70 ms when loaded with 10 concurrent streams. As the load increases on the server, the batching
scheduler shifts work to more efﬁcient batches, which keeps latency low. This shows that Batch
Dispatch makes it possible to deploy these large models at high throughput and low latency.
21
1 2 3 4 5 6 7 8 9 10
Batch size0.00.10.20.30.40.5TeraFLOP/snervana
baiduFigure 7: Comparison of kernels that compute Ax=bwhere Ais a matrix with dimension 25602560 , and
xis a matrix with dimension 2560Batch size, where Batch size 2[1, 10] . All matrices are in half-precision
format.
7.2 Deployment Optimized Matrix Multiply Kernels
We have found that deploying our models using half-precision (16-bit) ﬂoating-point arithmetic does
not measurably change recognition accuracy. Because deployment does not require any updates to


key: chroma:document
string_value:
not measurably change recognition accuracy. Because deployment does not require any updates to
the network weights, it is far less sensitive to numerical precision than training. Using half-precision
arithmetic saves memory space and bandwidth, which is especially useful for deployment, since
RNN evaluation is dominated by the cost of caching and streaming the weight matrices.
As seen in Section 7.1, the batch size during deployment is much smaller than in training. We
found that standard BLAS libraries are inefﬁcient at this batch size. To overcome this, we wrote our
own half-precision matrix-matrix multiply kernel. For 10 simultaneous streams over 90 percent of
batches are for N4, a regime where the matrix multiply will be bandwidth bound. We store the A
matrix transposed to maximize bandwidth by using the widest possible vector loads while avoiding
transposition after loading. Each warp computes four rows of output for all Noutput columns. Note
that forN4theBmatrix ﬁts entirely in the L1 cache. This scheme achieves 90 percent of peak
bandwidth for N4but starts to lose efﬁciency for larger Nas theBmatrix stops ﬁtting into the
L1 cache. Nonetheless, it continues to provide improved performance over existing libraries up to
N= 10 .


key: chroma:document
string_value:
L1 cache. Nonetheless, it continues to provide improved performance over existing libraries up to
N= 10 .
Figure 7 shows that our deployment kernel sustains a higher computational throughput than those
from Nervana Systems [44] on the K1200 GPU, across the entire range of batch sizes that we use
in deployment. Both our kernels and the Nervana kernels are signiﬁcantly faster than NVIDIA
CUBLAS version 7.0, more details are found here [20].
7.3 Beam Search
Performing the beam search involves repeated lookups in the n-gram language model, most of which
translate to uncached reads from memory. The direct implementation of beam search means that
each time-step dispatches one lookup per character for each beam. In Mandarin, this results in over
1M lookups per 40ms stride of speech data, which is too slow for deployment. To deal with this
problem, we use a heuristic to further prune the beam search. Rather than considering all characters
as viable additions to the beam, we only consider the fewest number of characters whose cumulative
probability is at least p. In practice, we have found that p= 0.99 works well. Additionally, we limit
ourselves to no more than 40 characters. This speeds up the Mandarin language model lookup time


key: chroma:document
string_value:
ourselves to no more than 40 characters. This speeds up the Mandarin language model lookup time
by a factor of 150x, and has a negligible effect on the CER (0.1-0.3% relative).
7.4 Results
We can deploy our system at low latency and high throughput without sacriﬁcing much accuracy.
On a held-out set of 2000 utterances, our research system achieves 5.81 character error rate whereas
the deployed system achieves 6.10 character error rate. This is only a 5% relative degradation for
22
the deployed system. In order to accomplish this, we employ a neural network architecture with low
deployment latency, reduce the precision of our network to 16-bit, built a batching scheduler to more
efﬁciently evaluate RNNs, and ﬁnd a simple heuristic to reduce beam search cost. The model has
ﬁve forward-only recurrent layers with 2560 hidden units, one row convolution layer (Section 3.7)
with= 19 , and one fully-connected layer with 2560 hidden units. These techniques allow us to
deploy Deep Speech at low cost to interactive applications.
8 Conclusion
End-to-end deep learning presents the exciting opportunity to improve speech recognition systems
continually with increases in data and computation. Indeed, our results show that, compared to the


key: chroma:document
string_value:
8 Conclusion
End-to-end deep learning presents the exciting opportunity to improve speech recognition systems
continually with increases in data and computation. Indeed, our results show that, compared to the
previous incarnation, Deep Speech has signiﬁcantly closed the gap in transcription performance with
human workers by leveraging more data and larger models. Further, since the approach is highly
generic, we’ve shown that it can quickly be applied to new languages. Creating high-performing
recognizers for two very different languages, English and Mandarin, required essentially no expert
knowledge of the languages. Finally, we have also shown that this approach can be efﬁciently
deployed by batching user requests together on a GPU server, paving the way to deliver end-to-end
Deep Learning technologies to users.
To achieve these results, we have explored various network architectures, ﬁnding several effective
techniques: enhancements to numerical optimization through SortaGrad and Batch Normalization,
evaluation of RNNs with larger strides with bigram outputs for English, searching through both
bidirectional and unidirectional models. This exploration was powered by a well optimized, High
Performance Computing inspired training system that allows us to train new, full-scale models on
our large datasets in just a few days.


key: chroma:document
string_value:
Performance Computing inspired training system that allows us to train new, full-scale models on
our large datasets in just a few days.
Overall, we believe our results conﬁrm and exemplify the value of end-to-end Deep Learning methods for speech recognition in several settings. In those cases where our system is not already comparable to humans, the difference has fallen rapidly, largely because of application-agnostic Deep
Learning techniques. We believe these techniques will continue to scale, and thus conclude that the
vision of a single speech system that outperforms humans in most scenarios is imminently achievable.
Acknowledgments
We are grateful to Baidu’s speech technology group for help with data preparation and useful conversations. We would like to thank Scott Gray, Amir Khosrowshahi and all of Nervana Systems for
their excellent matrix multiply routines and useful discussions. We would also like to thank Natalia
Gimelshein of NVIDIA for useful discussions and thoughts on implementing our fast deployment
matrix multiply.
References
[1] O. Abdel-Hamid, A.-r. Mohamed, H. Jang, and G. Penn. Applying convolutional neural networks concepts to hybrid nn-hmm model for speech recognition. In ICASSP , 2012.


key: chroma:document
string_value:
[2] D. Bahdanau, K. Cho, and Y . Bengio. Neural machine translation by jointly learning to align and translate.
InICLR , 2015.
[3] D. Bahdanau, J. Chorowski, D. Serdyuk, P. Brakel, and Y . Bengio. End-to-end attention-based large
vocabulary speech recognition. abs/1508.04395, 2015. http://arxiv.org/abs/1508.04395.
[4] J. Barker, E. Marxer, Ricard Vincent, and S. Watanabe. The third ’CHiME’ speech separation and recognition challenge: Dataset, task and baselines. 2015. Submitted to IEEE 2015 Automatic Speech Recognition
and Understanding Workshop (ASRU).
[5] S. Baxter. Modern GPU. https://nvlabs.github.io/moderngpu/ .
[6] Y . Bengio, J. Louradour, R. Collobert, and J. Weston. Curriculum learning. In International Conference
on Machine Learning , 2009.
[7] H. Bourlard and N. Morgan. Connectionist Speech Recognition: A Hybrid Approach . Kluwer Academic
Publishers, Norwell, MA, 1993.
23


key: chroma:document
string_value:
[7] H. Bourlard and N. Morgan. Connectionist Speech Recognition: A Hybrid Approach . Kluwer Academic
Publishers, Norwell, MA, 1993.
23
[8] W. Chan, N. Jaitly, Q. Le, and O. Vinyals. Listen, attend, and spell. abs/1508.01211, 2015.
http://arxiv.org/abs/1508.01211.
[9] S. Chetlur, C. Woolley, P. Vandermersch, J. Cohen, J. Tran, B. Catanzaro, and E. Shelhamer. cuDNN:
Efﬁcient primitives for deep learning.
[10] T. Chilimbi, Y . Suzue, J. Apacible, and K. Kalyanaraman. Project adam: Building an efﬁcient and scalable
deep learning training system. In USENIX Symposium on Operating Systems Design and Implementation ,
2014.
[11] K. Cho, B. Van Merrienboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y . Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. In EMNLP , 2014.


key: chroma:document
string_value:
[12] J. Chorowski, D. Bahdanau, K. Cho, and Y . Bengio. End-to-end continuous speech recognition using
attention-based recurrent nn: First results. abs/1412.1602, 2015. http://arxiv.org/abs/1412.1602.
[13] C. Cieri, D. Miller, and K. Walker. The Fisher corpus: a resource for the next generations of speech-totext. In LREC , volume 4, pages 69–71, 2004.
[14] A. Coates, B. Carpenter, C. Case, S. Satheesh, B. Suresh, T. Wang, D. J. Wu, and A. Y . Ng. Text detection
and character recognition in scene images with unsupervised feature learning. In International Conference
on Document Analysis and Recognition , 2011.
[15] A. Coates, B. Huval, T. Wang, D. J. Wu, A. Y . Ng, and B. Catanzaro. Deep learning with COTS HPC. In
International Conference on Machine Learning , 2013.
[16] G. Dahl, D. Yu, and L. Deng. Large vocabulary continuous speech recognition with context-dependent
DBN-HMMs. In Proc. ICASSP , 2011.


key: chroma:document
string_value:
DBN-HMMs. In Proc. ICASSP , 2011.
[17] G. Dahl, D. Yu, L. Deng, and A. Acero. Context-dependent pre-trained deep neural networks for large
vocabulary speech recognition. IEEE Transactions on Audio, Speech, and Language Processing , 2011.
[18] J. Dean, G. S. Corrado, R. Monga, K. Chen, M. Devin, Q. Le, M. Mao, M. Ranzato, A. Senior, P. Tucker,
K. Yang, and A. Ng. Large scale distributed deep networks. In Advances in Neural Information Processing
Systems 25 , 2012.
[19] D. Ellis and N. Morgan. Size matters: An empirical study of neural network training for large vocabulary
continuous speech recognition. In ICASSP , volume 2, pages 1013–1016. IEEE, 1999.
[20] E. Elsen. Optimizing RNN performance. http://svail.github.io/rnn_perf . Accessed: 2015-11-24.
[21] M. J. F. Gales, A. Ragni, H. Aldamarki, and C. Gautier. Support vector machines for noise robust ASR.
InASRU , pages 205–2010, 2009.


key: chroma:document
string_value:
InASRU , pages 205–2010, 2009.
[22] A. Graves, S. Fernández, F. Gomez, and J. Schmidhuber. Connectionist temporal classiﬁcation: Labelling
unsegmented sequence data with recurrent neural networks. In ICML , pages 369–376. ACM, 2006.
[23] A. Graves and N. Jaitly. Towards end-to-end speech recognition with recurrent neural networks. In ICML ,
2014.
[24] A. Graves, A.-r. Mohamed, and G. Hinton. Speech recognition with deep recurrent neural networks. In
ICASSP , 2013.
[25] H. H. Sak, A. Senior, and F. Beaufays. Long short-term memory recurrent neural network architectures
for large scale acoustic modeling. In Interspeech , 2014.
[26] A. Hannun, C. Case, J. Casper, B. Catanzaro, G. Diamos, E. Elsen, R. Prenger, S. Satheesh, S. Sengupta,
A. Coates, and A. Y . Ng. Deep speech: Scaling up end-to-end speech recognition. 1412.5567, 2014.
http://arxiv.org/abs/1412.5567.


key: chroma:document
string_value:
http://arxiv.org/abs/1412.5567.
[27] A. Y . Hannun, A. L. Maas, D. Jurafsky, and A. Y . Ng. First-pass large vocabulary continuous speech
recognition using bi-directional recurrent DNNs. abs/1408.2873, 2014. http://arxiv.org/abs/1408.2873.
[28] K. Heaﬁeld, I. Pouzyrevsky, J. H. Clark, and P. Koehn. Scalable modiﬁed Kneser-Ney language model
estimation. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics ,
Soﬁa, Bulgaria, 8 2013.
[29] G. Hinton, L. Deng, D. Yu, G. Dahl, A. Mohamed, N. Jaitly, A. Senior, V . Vanhoucke, P. Nguyen,
T. Sainath, and B. Kingsbury. Deep neural networks for acoustic modeling in speech recognition. IEEE
Signal Processing Magazine , 29(November):82–97, 2012.


key: chroma:document
string_value:
T. Sainath, and B. Kingsbury. Deep neural networks for acoustic modeling in speech recognition. IEEE
Signal Processing Magazine , 29(November):82–97, 2012.
[30] S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Computation , 9(8):1735—1780,
1997.
[31] N. Jaitly and G. Hinton. V ocal tract length perturbation (VTLP) improves speech recognition. In ICML
Workshop on Deep Learning for Audio, Speech, and Language Processing , 2013.
[32] R. Jozefowicz, W. Zaremba, and I. Sutskever. An empirical exploration of recurrent network architectures.
InICML , 2015.
24
[33] O. Kapralova, J. Alex, E. Weinstein, P. Moreno, and O. Siohan. A big data approach to acoustic model
training corpus selection. In Interspeech , 2014.
[34] K. C. Knowlton. A fast storage allocator. Commun. ACM , 8(10):623–624, Oct. 1965.


key: chroma:document
string_value:
[34] K. C. Knowlton. A fast storage allocator. Commun. ACM , 8(10):623–624, Oct. 1965.
[35] T. Ko, V . Peddinti, D. Povey, and S. Khudanpur. Audio augmentation for speech recognition. In Interspeech , 2015.
[36] A. Krizhevsky, I. Sutskever, and G. Hinton. Imagenet classiﬁcation with deep convolutional neural networks. In Advances in Neural Information Processing Systems 25 , pages 1106–1114, 2012.
[37] C. Laurent, G. Pereyra, P. Brakel, Y . Zhang, and Y . Bengio. Batch normalized recurrent neural networks.
abs/1510.01378, 2015. http://arxiv.org/abs/1510.01378.
[38] Q. Le, M. Ranzato, R. Monga, M. Devin, K. Chen, G. Corrado, J. Dean, and A. Ng. Building high-level
features using large scale unsupervised learning. In International Conference on Machine Learning , 2012.
[39] Y . LeCun, F. J. Huang, and L. Bottou. Learning methods for generic object recognition with invariance


key: chroma:document
string_value:
[39] Y . LeCun, F. J. Huang, and L. Bottou. Learning methods for generic object recognition with invariance
to pose and lighting. In Computer Vision and Pattern Recognition , volume 2, pages 97–104, 2004.
[40] A. Maas, Z. Xie, D. Jurafsky, and A. Ng. Lexicon-free conversational speech recognition with neural
networks. In NAACL , 2015.
[41] Y . Miao, M. Gowayyed, and F. Metz. EESEN: End-to-end speech recognition using deep rnn models and
wfst-based decoding. In ASRU , 2015.
[42] A. Mohamed, G. Dahl, and G. Hinton. Acoustic modeling using deep belief networks. IEEE Transactions
on Audio, Speech, and Language Processing , (99), 2011.
[43] A. S. N. Jaitly, P. Nguyen and V . Vanhoucke. Application of pretrained deep neural networks to large
vocabulary speech recognition. In Interspeech , 2012.
[44] Nervana Systems. Nervana GPU. https://github.com/NervanaSystems/nervanagpu . Accessed:
2015-11-06.


key: chroma:document
string_value:
[44] Nervana Systems. Nervana GPU. https://github.com/NervanaSystems/nervanagpu . Accessed:
2015-11-06.
[45] J. Niu, L. Xie, L. Jia, and N. Hu. Context-dependent deep neural networks for commercial mandarin
speech recognition applications. In APSIPA , 2013.
[46] V . Panayotov, G. Chen, D. Povey, and S. Khudanpur. Librispeech: an asr corpus based on public domain
audio books. In ICASSP , 2015.
[47] R. Pascanu, T. Mikolov, and Y . Bengio. On the difﬁculty of training recurrent neural networks.
abs/1211.5063, 2012. http://arxiv.org/abs/1211.5063.
[48] P. Patarasuk and X. Yuan. Bandwidth optimal all-reduce algorithms for clusters of workstations. J.
Parallel Distrib. Comput. , 69(2):117–124, Feb. 2009.
[49] R. Raina, A. Madhavan, and A. Ng. Large-scale deep unsupervised learning using graphics processors.
In26th International Conference on Machine Learning , 2009.


key: chroma:document
string_value:
In26th International Conference on Machine Learning , 2009.
[50] S. Renals, N. Morgan, H. Bourlard, M. Cohen, and H. Franco. Connectionist probability estimators in
HMM speech recognition. IEEE Transactions on Speech and Audio Processing , 2(1):161–174, 1994.
[51] T. Robinson, M. Hochberg, and S. Renals. The use of recurrent neural networks in continuous speech
recognition. pages 253–258, 1996.
[52] T. Sainath, O. Vinyals, A. Senior, and H. Sak. Convolutional, long short-term memory, fully connected
deep neural networks. In ICASSP , 2015.
[53] T. N. Sainath, A. rahman Mohamed, B. Kingsbury, and B. Ramabhadran. Deep convolutional neural
networks for LVCSR. In ICASSP , 2013.
[54] H. Sak, A. Senior, K. Rao, and F. Beaufays. Fast and accurate recurrent neural network acoustic models
for speech recognition. abs/1507.06947, 2015. http://arxiv.org/abs/1507.06947.


key: chroma:document
string_value:
for speech recognition. abs/1507.06947, 2015. http://arxiv.org/abs/1507.06947.
[55] H. Sak, O. Vinyals, G. Heigold, A. Senior, E. McDermott, R. Monga, and M. Mao. Sequence discriminative distributed training of long shortterm memory recurrent neural networks. In Interspeech , 2014.
[56] B. Sapp, A. Saxena, and A. Ng. A fast data collection and augmentation procedure for object recognition.
InAAAI Twenty-Third Conference on Artiﬁcial Intelligence , 2008.
[57] M. Schuster and K. K. Paliwal. Bidirectional recurrent neural networks. IEEE Transactions on Signal
Processing , 45(11):2673–2681, 1997.
[58] F. Seide, G. Li, and D. Yu. Conversational speech transcription using context-dependent deep neural
networks. In Interspeech , pages 437–440, 2011.
[59] J. Shan, G. Wu, Z. Hu, X. Tang, M. Jansche, and P. Moreno. Search by voice in mandarin chinese. In
Interspeech , 2010.


key: chroma:document
string_value:
Interspeech , 2010.
[60] H. Soltau, G. Saon, and T. Sainath. Joint training of convolutional and non-convolutional neural networks.
InICASSP , 2014.
25
[61] I. Sutskever, J. Martens, G. Dahl, and G. Hinton. On the importance of momentum and initialization in
deep learning. In 30th International Conference on Machine Learning , 2013.
[62] I. Sutskever, O. Vinyals, and Q. V . Le. Sequence to sequence learning with neural networks. 2014.
http://arxiv.org/abs/1409.3215.
[63] C. Szegedy and S. Ioffe. Batch normalization: Accelerating deep network training by reducing internal
covariate shift. abs/1502.03167, 2015. http://arxiv.org/abs/1502.03167.
[64] C. Szegedy, W. Liu, Y . Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V . Vanhoucke, and A. Rabinovich. Going deeper with convolutions. 2014.


key: chroma:document
string_value:
[65] R. Thakur and R. Rabenseifner. Optimization of collective communication operations in mpich. International Journal of High Performance Computing Applications , 19:49–66, 2005.
[66] K. Vesely, A. Ghoshal, L. Burget, and D. Povey. Sequence-discriminative training of deep neural networks. In Interspeech , 2013.
[67] A. Waibel, T. Hanazawa, G. Hinton, K. Shikano, and K. Lang. Phoneme recognition using time-delay
neural networks,â ˘A˙I acoustics speech and signal processing. IEEE Transactions on Acoustics, Speech
and Signal Processing , 37(3):328–339, 1989.
[68] R. Williams and J. Peng. An efﬁcient gradient-based algorithm for online training of recurrent network
trajectories. Neural computation , 2:490–501, 1990.
[69] T. Yoshioka, N. Ito, M. Delcroix, A. Ogawa, K. Kinoshita, M. F. C. Yu, W. J. Fabian, M. Espi, T. Higuchi,


