# embedding_metadata

key: chroma:document
string_value:
S. Araki, and T. Nakatani. The ntt chime-3 system: Advances in speech enhancement and recognition for
mobile multi-microphone devices. In IEEE ASRU , 2015.
[70] W. Zaremba and I. Sutskever. Learning to execute. abs/1410.4615, 2014. http://arxiv.org/abs/1410.4615.
A Scalability improvements
In this section, we discuss some of our scalability improvements in more detail.
A.1 Node and cluster architecture
The software stack runs on a compute dense node built from 2Intel CPUs and 8NVIDIA Titan
X GPUs, with peak single-precision computational throughput of 53teraFLOP/second. Each node
also has 384GB of CPU memory and an 8TB storage volume built from two 4 TB hard disks in
RAID-0 conﬁguration. We use the CPU memory to cache our input data so that we are not directly
exposed to the low bandwidth and high latency of spinning disks. We replicate our English and
Mandarin datasets on each node’s local hard disk. This allows us to use our network only for weight
updates and avoids having to rely on centralized ﬁle servers.
GPUGPUGPUGPUGPUGPUGPUGPUPLXPLXPLXCPUCPUPLX


key: chroma:document
string_value:
updates and avoids having to rely on centralized ﬁle servers.
GPUGPUGPUGPUGPUGPUGPUGPUPLXPLXPLXCPUCPUPLX
Figure 8: Schematic of our training node where PLX indicates a PCI switch and the dotted box includes all
devices that are connected by the same PCI root complex.
Figure 8 shows a schematic diagram of one our nodes, where all devices connected by the same PCI
root complex are encapsulated in a dotted box. We have tried to maximize the number of GPUs
within the root complex for faster communication between GPUs using GPUDirect. This allows us
to use an efﬁcient communication mechanism to transfer gradient matrices between GPUs.
26
All the nodes in our cluster are connected through Fourteen Data Rate (FDR) Inﬁniband which is
primarily used for gradient transfer during back-propagation.
A.2 GPU Implementation of CTC Loss Function
The CTC loss function that we use to train our models has two passes: forward and backward, and
the gradient computation involves element-wise addition of two matrices, and, generated during
the forward and backward passes respectively. Finally, we sum the gradients using the character in


key: chroma:document
string_value:
the gradient computation involves element-wise addition of two matrices, and, generated during
the forward and backward passes respectively. Finally, we sum the gradients using the character in
the utterance label as the key, to generate one gradient per character. These gradients are then backpropagated through the network. The input to the CTC loss function are probabilities calculated by
the softmax function which can be very small, so we compute in logprobability space for better
numerical stability.
The forward pass of the CTC algorithm calculates the matrix, which has Srows and Tcolumns,
where S= 2(L+ 1) . The variable Lis the number of characters in the label and Tis the number
of time-steps in the utterance. Our CPU-based implementation of the CTC algorithm assigns one
thread to each utterance label in a minibatch, performing the CTC calculation for the utterances in
parallel. Each thread calculates the relevant entries of the matrix sequentially. This is inefﬁcient for
two reasons.
Firstly, since the remainder of our network is computed on the GPU, the output of the softmax
function has to be copied to the CPU for CTC calculation. The gradient matrices from the CTC
function then has to be copied back to the GPU for backpropagation. For languages like Mandarin


key: chroma:document
string_value:
function then has to be copied back to the GPU for backpropagation. For languages like Mandarin
with large character sets, these matrices have hundreds of millions of entries, making this copy
expensive. Furthermore, we need as much interconnect bandwidth as possible for synchronizing the
gradient updates with data parallelism, so this copy incurs a substantial opportunity cost.
Secondly, although entries in each column of the matrix can be computed in parallel, the number
of entries to calculate in each column depends both on the column and the number of repeated
characters in the utterance label. Due to this complexity, the CPU implementation does not use
SIMD parallelism optimally, making the computation inefﬁcient.
We wrote a GPU-based implementation of CTC in order to overcome these two problems. The key
insight behind our implementation is that we can compute all elements in each column of the 
matrix, rather than just the valid entries. If we do so, Figure 9 shows that invalid elements either
contain a ﬁnite garbage value ( G), or1 (I), when we use a special summation function that adds
probabilities in logspace that discards inputs that are 1. This summation is shown in Figure 9
where arrows incident on a circle are inputs and the result is stored in the circle. However, when we


key: chroma:document
string_value:
where arrows incident on a circle are inputs and the result is stored in the circle. However, when we
compute the ﬁnal gradient by element-wise summing and, all ﬁnite garbage values will be added
with a corresponding 1 value from the other matrix, which results in 1, effectively ignoring the
garbage value and computing the correct result. One important observation is that this element-wise
sum ofandis a simple sum and does not use our summation function.
To compute the gradient, we take each column of the matrix generated from element-wise addition
ofandmatrices, and do a key-value reduction using the character as key, using the ModernGPU
library [5]. This means elements of the column corresponding to the same character will sum up
their values. In the example shown in Figure 9, the blank character, B, is the only repeated character
and at some columns, say for t= 1 oft= 2, both valid elements (gray) and 1 correspond to
it. Since our summation function in logspace effectively ignores the 1 elements, only the valid
elements are combined in the reduction.
In our GPU implementation, we map each utterance in the minibatch to a CUDA thread block .


key: chroma:document
string_value:
elements are combined in the reduction.
In our GPU implementation, we map each utterance in the minibatch to a CUDA thread block .
Since there are no dependencies between the elements of a column, all of them can be computed in
parallel by the threads in a threadblock. There are dependencies between columns, since the column
corresponding to time-step t+ 1cannot be computed before the column corresponding to time-step
t. The reverse happens when computing the matrix, when column corresponding to time-step
tcannot be computed before the column corresponding to time-step t+ 1. Thus, in both cases,
columns are processed sequentially by the thread block.
Mapping the forward and backward passes to corresponding CUDA kernels is straightforward since
there are no data dependencies between elements of a column. The kernel that does the backward
pass also computes the gradient. However, since the gradients must be summed up based on the label
27
IIIIATCIIIIIGGGGGGGG
G1234T-3T-2T-1T
GGGGATCGGGGGIIIIIIII
I↵BBBBBBBBFigure 9: Forward and backward pass for GPU implementation of CTC. Gray circles contain valid values,


key: chroma:document
string_value:
GGGGATCGGGGGIIIIIIII
I↵BBBBBBBBFigure 9: Forward and backward pass for GPU implementation of CTC. Gray circles contain valid values,
circle with Icontain1 and circle with Gcontain garbage values that are ﬁnite. Bstand for the blank character
that the CTC algorithm adds to the input utterance label. Column labels on top show different time-steps going
from1toT.
values, with each character as key, we must deal with data dependencies due to repeated characters
in an utterance label. For languages with small character sets like English, this happens with high
probability. Even if there are no repeated characters, the CTC algorithm adds L+ 1blank characters
to the utterance label. We solve this problem by performing a key-value sort, where the keys are the
characters in the utterance label, and the values are the indices of each character in the utterance.
After sorting, all occurrences of a given character are arranged in contiguous segments. We only
need to do the sort once for each utterance. The indices generated by the sort are then used to
sequentially sum up the gradients for each character. This sum is done once per column and in
parallel over all characters in the utterance. Amortizing the cost of key-value sort over Tcolumns is


key: chroma:document
string_value:
parallel over all characters in the utterance. Amortizing the cost of key-value sort over Tcolumns is
a key insight that makes the gradient calculation fast.
Our GPU implementation uses fast shared memory and registers to achieve high performance when
performing this task. Both forward and backward kernels store the matrix in shared memory .
Since shared memory is a limited resource, it is not possible to store the entire matrix. However,
as we go backward in time, we only need to keep one column of the matrix as we compute the
gradient, adding element-wise the column of the matrix with the corresponding column of the
matrix. Due to on-chip memory space constraints, we read the output of the softmax function
directly from off-chip global memory .
Due to inaccuracies in ﬂoating-point arithmetic, especially in transcendental functions, our GPU and
CPU implementation are not bit-wise identical. This is not an impediment in practice, since both
implementations train models equally well when coupled with the technique of sorting utterances
by length mentioned in Section 3.3.
28


key: chroma:document
string_value:
Published as a conference paper at ICLR 2017
RECURRENT BATCH NORMALIZATION
Tim Cooijmans, Nicolas Ballas, César Laurent, Ça ˘glar Gülçehre & Aaron Courville
MILA - Université de Montréal
firstname.lastname@umontreal.ca
ABSTRACT
We propose a reparameterization of LSTM that brings the beneﬁts of batch normalization to recurrent neural networks. Whereas previous works only apply batch
normalization to the input-to-hidden transformation of RNNs, we demonstrate that
it is both possible and beneﬁcial to batch-normalize the hidden-to-hidden transition, thereby reducing internal covariate shift between time steps.
We evaluate our proposal on various sequential problems such as sequence classiﬁcation, language modeling and question answering. Our empirical results show
that our batch-normalized LSTM consistently leads to faster convergence and improved generalization.
1 I NTRODUCTION
Recurrent neural network architectures such as LSTM (Hochreiter & Schmidhuber, 1997) and
GRU (Cho et al., 2014) have recently exhibited state-of-the-art performance on a wide range of


key: chroma:document
string_value:
GRU (Cho et al., 2014) have recently exhibited state-of-the-art performance on a wide range of
complex sequential problems including speech recognition Amodei et al. (2015), machine translation (Bahdanau et al., 2015) and image and video captioning (Xu et al., 2015; Yao et al., 2015).
Top-performing models, however, are based on very high-capacity networks that are computationally intensive and costly to train. Effective optimization of recurrent neural networks is thus an
active area of study (Pascanu et al., 2012; Martens & Sutskever, 2011; Ollivier, 2013).
It is well-known that for deep feed-forward neural networks, covariate shift (Shimodaira, 2000; Ioffe
& Szegedy, 2015) degrades the efﬁciency of training. Covariate shift is a change in the distribution
of the inputs to a model. This occurs continuously during training of feed-forward neural networks,
where changing the parameters of a layer affects the distribution of the inputs to all layers above it.
As a result, the upper layers are continually adapting to the shifting input distribution and unable
to learn effectively. This internal covariate shift (Ioffe & Szegedy, 2015) may play an especially
important role in recurrent neural networks, which resemble very deep feed-forward networks.


key: chroma:document
string_value:
important role in recurrent neural networks, which resemble very deep feed-forward networks.
Batch normalization (Ioffe & Szegedy, 2015) is a recently proposed technique for controlling the
distributions of feed-forward neural network activations, thereby reducing internal covariate shift. It
involves standardizing the activations going into each layer, enforcing their means and variances to
be invariant to changes in the parameters of the underlying layers. This effectively decouples each
layer’s parameters from those of other layers, leading to a better-conditioned optimization problem.
Indeed, deep neural networks trained with batch normalization converge signiﬁcantly faster and
generalize better.
Although batch normalization has demonstrated signiﬁcant training speed-ups and generalization
beneﬁts in feed-forward networks, it is proven to be difﬁcult to apply in recurrent architectures (Laurent et al., 2016; Amodei et al., 2015). It has found limited use in stacked RNNs, where the normalization is applied “vertically”, i.e. to the input of each RNN, but not “horizontally” between
timesteps. RNNs are deeper in the time direction, and as such batch normalization would be most


key: chroma:document
string_value:
timesteps. RNNs are deeper in the time direction, and as such batch normalization would be most
beneﬁcial when applied horizontally. However, Laurent et al. (2016) hypothesized that applying
batch normalization in this way hurts training because of exploding gradients due to repeated rescaling.
Our ﬁndings run counter to this hypothesis. We show that it is both possible and highly beneﬁcial to
apply batch normalization in the hidden-to-hidden transition of recurrent models. In particular, we
describe a reparameterization of LSTM (Section 3) that involves batch normalization and demonstrate that it is easier to optimize and generalizes better. In addition, we empirically analyze the
1arXiv:1603.09025v5  [cs.LG]  28 Feb 2017
Published as a conference paper at ICLR 2017
gradient backpropagation and show that proper initialization of the batch normalization parameters
is crucial to avoiding vanishing gradient (Section 4). We evaluate our proposal on several sequential problems and show (Section 5) that our LSTM reparameterization consistently outperforms the
LSTM baseline across tasks, in terms of both time to convergence and performance.
Liao & Poggio (2016) simultaneously investigated batch normalization in recurrent neural networks,


key: chroma:document
string_value:
LSTM baseline across tasks, in terms of both time to convergence and performance.
Liao & Poggio (2016) simultaneously investigated batch normalization in recurrent neural networks,
albeit only for very short sequences (10 steps). Ba et al. (2016) independently developed a variant
of batch normalization that is also applicable to recurrent neural networks and delivers similar improvements as our method.
2 P REREQUISITES
2.1 LSTM
Long Short-Term Memory (LSTM) networks are an instance of a more general class of recurrent
neural networks (RNNs), which we review brieﬂy in this paper. Given an input sequence X=
(x1;x2;:::;xT), an RNN deﬁnes a sequence of hidden states htaccording to
ht=(Whht1+Wxxt+b); (1)
where Wh2Rdhdh;Wx2Rdxdh;b2Rdhand the initial state h02Rdhare model parameters. A popular choice for the activation function ()istanh .
RNNs are popular in sequence modeling thanks to their natural ability to process variable-length


key: chroma:document
string_value:
RNNs are popular in sequence modeling thanks to their natural ability to process variable-length
sequences. However, training RNNs using ﬁrst-order stochastic gradient descent (SGD) is notoriously difﬁcult due to the well-known problem of exploding/vanishing gradients (Bengio et al., 1994;
Hochreiter, 1991; Pascanu et al., 2012). Gradient vanishing occurs when states htare not inﬂuenced
by small changes in much earlier states h,t, preventing learning of long-term dependencies in
the input data. Although learning long-term dependencies is fundamentally difﬁcult (Bengio et al.,
1994), its effects can be mitigated through architectural variations such as LSTM (Hochreiter &
Schmidhuber, 1997), GRU (Cho et al., 2014) and iRNN/uRNN (Le et al., 2015; Arjovsky et al.,
2015).
In what follows, we focus on the LSTM architecture (Hochreiter & Schmidhuber, 1997) with recurrent transition given by
0
BB@~ft
~it
~ot
~gt1
CCA=Whht1+Wxxt+b (2)


key: chroma:document
string_value:
0
BB@~ft
~it
~ot
~gt1
CCA=Whht1+Wxxt+b (2)
ct=(~ft)ct1+(~it)tanh( ~gt) (3)
ht=(~ot)tanh(ct); (4)
where Wh2Rdh4dh;WxRdx4dh;b2R4dhand the initial states h02Rdh;c02Rdhare
model parameters. is the logistic sigmoid function, and the operator denotes the Hadamard
product.
The LSTM differs from simple RNNs in that it has an additional memory cellctwhose update is
nearly linear which allows the gradient to ﬂow back through time more easily. In addition, unlike
the RNN which overwrites its content at each timestep, the update of the LSTM cell is regulated by
a set of gates. The forget gate ftdetermines the extent to which information is carried over from the
previous timestep, and the input gate itcontrols the ﬂow of information from the current input xt.
The output gate otallows the model to read from the cell. This carefully controlled interaction with


key: chroma:document
string_value:
The output gate otallows the model to read from the cell. This carefully controlled interaction with
the cell is what allows the LSTM to robustly retain information for long periods of time.
2.2 B ATCH NORMALIZATION
Covariate shift (Shimodaira, 2000) is a phenomenon in machine learning where the features presented to a model change in distribution. In order for learning to succeed in the presence of covariate shift, the model’s parameters must be adjusted not just to learn the concept at hand but also to
adapt to the changing distribution of the inputs. In deep neural networks, this problem manifests as
2
Published as a conference paper at ICLR 2017
internal covariate shift (Ioffe & Szegedy, 2015), where changing the parameters of a layer affects
the distribution of the inputs to all layers above it.
Batch Normalization (Ioffe & Szegedy, 2015) is a recently proposed network reparameterization
which aims to reduce internal covariate shift. It does so by standardizing the activations using
empirical estimates of their means and standard deviations. However, it does not decorrelate the
activations due to the computationally costly matrix inversion. The batch normalizing transform is
as follows:
BN(h;
;) =+
hbE[h]q


key: chroma:document
string_value:
as follows:
BN(h;
;) =+
hbE[h]q
dVar[h] +(5)
where h2Rdis the vector of (pre)activations to be normalized, 
2Rd;2Rdare model
parameters that determine the mean and standard deviation of the normalized activation, and 2R
is a regularization hyperparameter. The division should be understood to proceed elementwise.
At training time, the statistics E[h]andVar[h]are estimated by the sample mean and sample variance of the current minibatch. This allows for backpropagation through the statistics, preserving the
convergence properties of stochastic gradient descent. During inference, the statistics are typically
estimated based on the entire training set, so as to produce a deterministic prediction.
3 B ATCH -NORMALIZED LSTM
This section introduces a reparameterization of LSTM that takes advantage of batch normalization.
Contrary to Laurent et al. (2016); Amodei et al. (2015), we leverage batch normalization in both
the input-to-hidden andthe hidden-to-hidden transformations. We introduce the batch-normalizing
transform BN(;
;)into the LSTM as follows:
0
BB@~ft


key: chroma:document
string_value:
transform BN(;
;)into the LSTM as follows:
0
BB@~ft
~it
~ot
~gt1
CCA= BN( Whht1;
h;h) + BN( Wxxt;
x;x) +b (6)
ct=(~ft)ct1+(~it)tanh( ~gt) (7)
ht=(~ot)tanh(BN( ct;
c;c)) (8)
In our formulation, we normalize the recurrent term Whht1and the input term Wxxtseparately.
Normalizing these terms individually gives the model better control over the relative contribution
of the terms using the 
hand
xparameters. We set h=x=0to avoid unnecessary redundancy, instead relying on the pre-existing parameter vector bto account for both biases. In order to
leave the LSTM dynamics intact and preserve the gradient ﬂow through ct, we do not apply batch
normalization in the cell update.
The batch normalization transform relies on batch statistics to standardize the LSTM activations. It
would seem natural to share the statistics that are used for normalization across time, just as recurrent


key: chroma:document
string_value:
The batch normalization transform relies on batch statistics to standardize the LSTM activations. It
would seem natural to share the statistics that are used for normalization across time, just as recurrent
neural networks share their parameters over time. However, we ﬁnd that simply averaging statistics
over time severely degrades performance. Although LSTM activations do converge to a stationary
distribution, we observe that their statistics during the initial transient differ signiﬁcantly (see Figure 5 in Appendix A). Consequently, we recommend using separate statistics for each timestep to
preserve information of the initial transient phase in the activations.1
Generalizing the model to sequences longer than those seen during training is straightforward thanks
to the rapid convergence of the activations to their steady-state distributions (cf. Figure 5). For our
experiments we estimate the population statistics separately for each timestep 1;:::;Tmax where
1Note that we separate only the statistics over time and not the 
andparameters.
3
Published as a conference paper at ICLR 2017
Tmax is the length of the longest training sequence. When at test time we need to generalize beyond
Tmax, we use the population statistic of time Tmax for all time steps beyond it.
During training we estimate the statistics across the minibatch, independently for each timestep. At


key: chroma:document
string_value:
Tmax, we use the population statistic of time Tmax for all time steps beyond it.
During training we estimate the statistics across the minibatch, independently for each timestep. At
test time we use estimates obtained by averaging the minibatch estimates over the training set.
4 I NITIALIZING 
FOR GRADIENT FLOW
Although batch normalization allows for easy control of the pre-activation variance through the 
parameters, common practice is to normalize to unit variance. We suspect that the previous difﬁculties with recurrent batch normalization reported in Laurent et al. (2016); Amodei et al. (2015) are
largely due to improper initialization of the batch normalization parameters, and 
in particular. In
this section we demonstrate the impact of 
on gradient ﬂow.
0 100 200 300 400 500 600 700 800
t10-2610-2410-2210-2010-1810-1610-1410-1210-1010-810-610-410-2100||dloss/dh_t||_2RNN gradient propagation
gamma=0.10
gamma=0.20
gamma=0.30
gamma=0.40
gamma=0.50
gamma=0.60
gamma=0.70
gamma=0.80


key: chroma:document
string_value:
gamma=0.40
gamma=0.50
gamma=0.60
gamma=0.70
gamma=0.80
gamma=0.90
gamma=1.00
(a) We visualize the gradient ﬂow through a batchnormalized tanh RNN as a function of 
. High
variance causes vanishing gradient.
0.0 0.2 0.4 0.6 0.8 1.0
input standard deviation0.00.20.40.60.81.0expected derivative (and IQR range)derivative through tanh(b) We show the empirical expected derivative and
interquartile range of tanh nonlinearity as a function of input variance. High variance causes saturation, which decreases the expected derivative.
Figure 1: Inﬂuence of pre-activation variance on gradient propagation.
In Figure 1(a), we show how the pre-activation variance impacts gradient propagation in a simple
RNN on the sequential MNIST task described in Section 5.1. Since backpropagation operates in
reverse, the plot is best read from right to left. The quantity plotted is the norm of the gradient
of the loss with respect to the hidden state at different time steps. For large values of 
, the norm


key: chroma:document
string_value:
of the loss with respect to the hidden state at different time steps. For large values of 
, the norm
quickly goes to zero as gradient is propagated back in time. For small values of 
the norm is nearly
constant.
To demonstrate what we think is the cause of this vanishing, we drew samples xfrom a set of
centered Gaussian distributions with standard deviation ranging from 0 to 1, and computed the
derivative tanh0(x) = 1tanh2(x)2[0;1]for each. Figure 1(b) shows the empirical distribution
of the derivative as a function of standard deviation. When the input standard deviation is low, the
input tends to be close to the origin where the derivative is close to 1. As the standard deviation
increases, the expected derivative decreases as the input is more likely to be in the saturation regime.
At unit standard deviation, the expected derivative is much smaller than 1.
We conjecture that this is what causes the gradient to vanish, and recommend initializing 
to a
small value. In our trials we found that values of 0.01 or lower caused instabilities during training.
Our choice of 0.1 seems to work well across different tasks.
5 E XPERIMENTS


key: chroma:document
string_value:
Our choice of 0.1 seems to work well across different tasks.
5 E XPERIMENTS
This section presents an empirical evaluation of the proposed batch-normalized LSTM on four different tasks. Note that for all the experiments, we initialize the batch normalization scale and shift
parameters
andto0:1and0respectively.
4
Published as a conference paper at ICLR 2017
0 20000 40000 60000 80000 100000
Training Iteration0.00.20.40.60.81.0AccuracyPixel-by-Pixel MNIST (Validation Set)
lstm
bn_lstm
0 20000 40000 60000 80000 100000
Training Iteration0.10.20.30.40.50.60.70.80.91.0AccuracyPixel-by-Pixel Permuted-MNIST (Validation Set)
lstm
bn_lstm
Figure 2: Accuracy on the validation set for the pixel by pixel MNIST classiﬁcation tasks. The
batch-normalized LSTM is able to converge faster relatively to a baseline LSTM. Batch-normalized
LSTM also shows some improve generalization on the permuted sequential MNIST that require to
preserve long-term memory information.
5.1 S EQUENTIAL MNIST


key: chroma:document
string_value:
LSTM also shows some improve generalization on the permuted sequential MNIST that require to
preserve long-term memory information.
5.1 S EQUENTIAL MNIST
We evaluate our batch-normalized LSTM on a sequential version of the MNIST classiﬁcation
task (Le et al., 2015). The model processes each image one pixel at a time and ﬁnally predicts
the label. We consider both sequential MNIST tasks, MNIST and permuted MNIST ( pMNIST). In
MNIST, the pixels are processed in scanline order. In pMNIST the pixels are processed in a ﬁxed
random order.
Our baseline consists of an LSTM with 100 hidden units, with a softmax classiﬁer to produce a
prediction from the ﬁnal hidden state. We use orthogonal initialization for all weight matrices,
except for the hidden-to-hidden weight matrix which we initialize to be the identity matrix, as this
yields better generalization performance on this task for both models. The model is trained using
RMSProp (Tieleman & Hinton, 2012) with learning rate of 103and0:9momentum. We apply
gradient clipping at 1 to avoid exploding gradients.


key: chroma:document
string_value:
gradient clipping at 1 to avoid exploding gradients.
The in-order MNIST task poses a unique problem for our model: the input for the ﬁrst hundred or so
timesteps is constant across examples since the upper pixels are almost always black. This causes the
variance of the hidden states to be exactly zero for a long period of time. Normalizing these zerovariance activations involves dividing zero by a small number at many timesteps, which does not
affect the forward-propagated activations but causes the back-propagated gradient to explode. We
work around this by adding Gaussian noise to the initial hidden states. Although the normalization
ampliﬁes the noise to signal level, we ﬁnd that it does not hurt performance compared to datadependent ways of initializing the hidden states.
Model MNIST pMNIST
TANH-RNN (Le et al., 2015) 35.0 35.0
iRNN (Le et al., 2015) 97.0 82.0
uRNN (Arjovsky et al., 2015) 95.1 91.4
sTANH-RNN (Zhang et al., 2016) 98.1 94.0
LSTM (ours) 98.9 90.2
BN-LSTM (ours) 99.0 95.4


key: chroma:document
string_value:
LSTM (ours) 98.9 90.2
BN-LSTM (ours) 99.0 95.4
Table 1: Accuracy obtained on the test set for the pixel by pixel MNIST classiﬁcation tasks
In Figure 2 we show the validation accuracy while training for both LSTM and batch-normalized
LSTM (BN-LSTM). BN-LSTM converges faster than LSTM on both tasks. Additionally, we observe that BN-LSTM generalizes signiﬁcantly better on pMNIST. It has been highlighted in Arjovsky et al. (2015) that pMNIST contains many longer term dependencies across pixels than in
the original pixel ordering, where a lot of structure is local. A recurrent network therefore needs to
5
Published as a conference paper at ICLR 2017
Model Penn Treebank
LSTM (Graves, 2013) 1.262
HF-MRNN (Mikolov et al., 2012) 1.41
Norm-stabilized LSTM (Krueger & Memisevic, 2016) 1.39
ME n-gram (Mikolov et al., 2012) 1.37
LSTM (ours) 1.38
BN-LSTM (ours) 1.32
Zoneout (Krueger et al., 2016) 1.27


key: chroma:document
string_value:
LSTM (ours) 1.38
BN-LSTM (ours) 1.32
Zoneout (Krueger et al., 2016) 1.27
HM-LSTM (Chung et al., 2016) 1.24
HyperNetworks (Ha et al., 2016) 1.22
Table 2: Bits-per-character on the Penn Treebank test sequence.
characterize dependencies across varying time scales in order to solve this task. Our results suggest
that BN-LSTM is better able to capture these long-term dependencies.
Table 1 reports the test set accuracy of the early stop model for LSTM and BN-LSTM using the population statistics. Recurrent batch normalization leads to a better test score, especially for pMNIST
where models have to leverage long-term temporal depencies. In addition, Table 1 shows that our
batch-normalized LSTM achieves state of the art on both MNIST and pMNIST.
5.2 C HARACTER -LEVEL PENN TREEBANK
We evaluate our model on the task of character-level language modeling on the Penn Treebank
corpus (Marcus et al., 1993) according to the train/valid/test partition of Mikolov et al. (2012). For
training, we segment the training sequence into examples of length 100. The training sequence does


key: chroma:document
string_value:
training, we segment the training sequence into examples of length 100. The training sequence does
not cleanly divide by 100, so for each epoch we randomly crop a subsequence that does and segment
that instead.
Our baseline is an LSTM with 1000 units, trained to predict the next character using a softmax
classiﬁer on the hidden state ht. We use stochastic gradient descent on minibatches of size 64, with
gradient clipping at 1.0 and step rule determined by Adam (Kingma & Ba, 2014) with learning rate
0.002. We use orthogonal initialization for all weight matrices. The setup for the batch-normalized
LSTM is the same in all respects except for the introduction of batch normalization as detailed in 3.
We show the learning curves in Figure 3(a). BN-LSTM converges faster and generalizes better than
the LSTM baseline. Figure 3(b) shows the generalization of our model to longer sequences. We
observe that using the population statistics improves generalization performance, which conﬁrms
that repeating the last population statistic (cf. Section 3) is a viable strategy. In table 2 we report
the performance of our best models (early-stopped on validation performance) on the Penn Treebank


key: chroma:document
string_value:
the performance of our best models (early-stopped on validation performance) on the Penn Treebank
test sequence. Follow up works havd since improved the state of the art (Krueger et al., 2016; Chung
et al., 2016; Ha et al., 2016).
5.3 T EXT8
We evaluate our model on a second character-level language modeling task on the much larger
text8 dataset (Mahoney, 2009). This dataset is derived from Wikipedia and consists of a sequence
of 100M characters including only alphabetical characters and spaces. We follow Mikolov et al.
(2012); Zhang et al. (2016) and use the ﬁrst 90M characters for training, the next 5M for validation
and the ﬁnal 5M characters for testing. We train on nonoverlapping sequences of length 180.
Both our baseline and batch-normalized models are LSTMs with 2000 units, trained to predict the
next character using a softmax classiﬁer on the hidden state ht. We use stochastic gradient descent
on minibatches of size 128, with gradient clipping at 1.0 and step rule determined by Adam (Kingma
& Ba, 2014) with learning rate 0.001. All weight matrices were initialized to be orthogonal.
6
Published as a conference paper at ICLR 2017


key: chroma:document
string_value:
& Ba, 2014) with learning rate 0.001. All weight matrices were initialized to be orthogonal.
6
Published as a conference paper at ICLR 2017
We early-stop on validation performance and report the test performance of the resulting model in
table 3. We observe that BN-LSTM obtains a signiﬁcant performance improvement over the LSTM
baseline. Chung et al. (2016) has since improved on our performance.
Model text8
td-LSTM (Zhang et al., 2016) 1.63
HF-MRNN (Mikolov et al., 2012) 1.54
skipping RNN (Pachitariu & Sahani, 2013) 1.48
LSTM (ours) 1.43
BN-LSTM (ours) 1.36
HM-LSTM (Chung et al., 2016) 1.29
Table 3: Bits-per-character on the text8 test sequence.
5.4 T EACHING MACHINES TO READ AND COMPREHEND
Recently, Hermann et al. (2015) introduced a set of challenging benchmarks for natural language
processing, along with neural network architectures to address them. The tasks involve reading


key: chroma:document
string_value:
Recently, Hermann et al. (2015) introduced a set of challenging benchmarks for natural language
processing, along with neural network architectures to address them. The tasks involve reading
real news articles and answering questions about their content. Their principal model, the Attentive Reader, is a recurrent neural network that invokes an attention mechanism to locate relevant
information in the document. Such models are notoriously hard to optimize and yet increasingly
popular.
To demonstrate the generality and practical applicability of our proposal, we apply batch normalization in the Attentive Reader model and show that this drastically improves training.
We evaluate several variants. The ﬁrst variant, referred to as BN-LSTM, consists of the vanilla Attentive Reader model with the LSTM simply replaced by our BN-LSTM reparameterization. The
second variant, termed BN-everywhere, is exactly like the ﬁrst, except that we also introduce batch
normalization into the attention computations, normalizing each term going into the tanh nonlinearities.
Our third variant, BN-e*, is like BN-everywhere, but improved to more carefully handle variablelength sequences. Throughout this experiment we followed the common practice of padding each
batch of variable-length data with zeros. However, this biases the batch mean and variance of xt


key: chroma:document
string_value:
batch of variable-length data with zeros. However, this biases the batch mean and variance of xt
toward zero. We address this effect using sequencewise normalization of the inputs as proposed
by Laurent et al. (2016); Amodei et al. (2015). That is, we share statistics over time for normalization
0 2000 4000 6000 8000 10000 12000 14000 16000
training steps1.41.61.82.02.22.4bits per characterLSTM
BN-LSTM
(a) Performance in bits-per-character on length100 subsequences of the Penn Treebank validation
sequence during training.
100 200 300 400 500 600 700 800 900 1000
sequence length1.321.341.361.381.401.421.441.46mean bits per characterLSTM
BN-LSTM, population statistics
BN-LSTM, batch statistics(b) Generalization to longer subsequences of Penn
Treebank using population statistics. The subsequences are taken from the test sequence.
Figure 3: Penn Treebank evaluation
7
Published as a conference paper at ICLR 2017
0 100 200 300 400 500 600 700 800
training steps (thousands)0.00.20.40.60.81.0error rateLSTM train
BN-LSTM train
BN-everywhere train
BN-e* train


key: chroma:document
string_value:
BN-LSTM train
BN-everywhere train
BN-e* train
BN-e** train
LSTM valid
BN-LSTM valid
BN-everywhere valid
BN-e* valid
BN-e** valid
(a) Error rate on the validation set for the Attentive Reader models on a variant of the CNN QA
task (Hermann et al., 2015). As detailed in Appendix C, the theoretical lower bound on the error
rate on this task is 43%.
0 50 100 150 200 250 300 350 400
training steps (thousands)0.10.20.30.40.50.60.70.80.91.0error rateLSTM train
BN-e** train
LSTM valid
BN-e** valid(b) Error rate on the validation set on the full CNN
QA task from Hermann et al. (2015).
Figure 4: Training curves on the CNN question-answering tasks.
of the input terms Wxxt, but notfor the recurrent terms Whhtor the cell output ct. Doing so avoids
many issues involving degenerate statistics due to input sequence padding.
Our fourth and ﬁnal variant BN-e** is like BN-e* but bidirectional. The main difﬁculty in adapting


key: chroma:document
string_value:
Our fourth and ﬁnal variant BN-e** is like BN-e* but bidirectional. The main difﬁculty in adapting
to bidirectional models also involves padding. Padding poses no problem as long as it is properly
ignored (by not updating the hidden states based on padded regions of the input). However to
perform the reverse application of a bidirectional model, it is common to simply reverse the padded
sequences, thus moving the padding to the front. This causes similar problems as were observed
on the sequential MNIST task (Section 5.1): the hidden states will not diverge during the initial
timesteps and hence their variance will be severely underestimated. To get around this, we reverse
only the unpadded portion of the input sequences and leave the padding in place.
See Appendix C for hyperparameters and task details.
Figure 4(a) shows the learning curves for the different variants of the attentive reader. BN-LSTM
trains dramatically faster than the LSTM baseline. BN-everywhere in turn shows a signiﬁcant improvement over BN-LSTM. In addition, both BN-LSTM and BN-everywhere show a generalization


key: chroma:document
string_value:
beneﬁt over the baseline. The validation curves have minima of 50.3%, 49.5% and 50.0% for the
baseline, BN-LSTM and BN-everywhere respectively. We emphasize that these results were obtained without any tweaking – all we did was to introduce batch normalization.
BN-e* and BN-e** converge faster yet, and reach lower minima: 47.1% and 43.9% respectively.
Model CNN valid CNN test
Attentive Reader (Hermann et al., 2015) 38.4 37.0
LSTM (ours) 45.5 45.0
BN-e** (ours) 37.9 36.3
Table 4: Error rates on the CNN question-answering task Hermann et al. (2015).
We train and evaluate our best model, BN-e**, on the full task from (Hermann et al., 2015). On
this dataset we had to reduce the number of hidden units to 120 to avoid severe overﬁtting. Training
curves for BN-e** and a vanilla LSTM are shown in Figure 4(b). Table 4 reports performances of
the early-stopped models.
8
Published as a conference paper at ICLR 2017
6 C ONCLUSION


key: chroma:document
string_value:
the early-stopped models.
8
Published as a conference paper at ICLR 2017
6 C ONCLUSION
Contrary to previous ﬁndings by Laurent et al. (2016); Amodei et al. (2015), we have demonstrated
that batch-normalizing the hidden states of recurrent neural networks greatly improves optimization. Indeed, doing so yields beneﬁts similar to those of batch normalization in feed-forward neural
networks: our proposed BN-LSTM trains faster and generalizes better on a variety of tasks including language modeling and question-answering. We have argued that proper initialization of
the batch normalization parameters is crucial, and suggest that previous difﬁculties (Laurent et al.,
2016; Amodei et al., 2015) were due in large part to improper initialization. Finally, we have shown
our model to apply to complex settings involving variable-length data, bidirectionality and highly
nonlinear attention mechanisms.
ACKNOWLEDGEMENTS
The authors would like to acknowledge the following agencies for research funding and computing
support: the Nuance Foundation, Samsung, NSERC, Calcul Québec, Compute Canada, the Canada
Research Chairs and CIFAR. Experiments were carried out using the Theano (Team et al., 2016) and


key: chroma:document
string_value:
Research Chairs and CIFAR. Experiments were carried out using the Theano (Team et al., 2016) and
the Blocks and Fuel (van Merriënboer et al., 2015) libraries for scientiﬁc computing. We thank David
Krueger, Saizheng Zhang, Ishmael Belghazi and Yoshua Bengio for discussions and suggestions.
REFERENCES
D. Amodei et al. Deep speech 2: End-to-end speech recognition in english and mandarin.
arXiv:1512.02595 , 2015.
M. Arjovsky, A. Shah, and Y . Bengio. Unitary evolution recurrent neural networks.
arXiv:1511.06464 , 2015.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv:1607.06450 ,
2016.
D. Bahdanau, K. Cho, and Y . Bengio. Neural machine translation by jointly learning to align and
translate. ICLR , 2015.
Y . Bengio, P. Simard, and P. Frasconi. Learning long-term dependencies with gradient descent is
difﬁcult. Neural Networks, IEEE Transactions on , 1994.


key: chroma:document
string_value:
difﬁcult. Neural Networks, IEEE Transactions on , 1994.
K. Cho, B. Van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y . Bengio.
Learning phrase representations using rnn encoder-decoder for statistical machine translation.
arXiv:1406.1078 , 2014.
Junyoung Chung, Sungjin Ahn, and Yoshua Bengio. Hierarchical multiscale recurrent neural networks. arXiv:1609.01704 , 2016.
A. Graves. Generating sequences with recurrent neural networks. arXiv:1308.0850 , 2013.
David Ha, Andrew Dai, and Quoc V Le. Hypernetworks. arXiv:1609.09106 , 2016.
K. M. Hermann, T. Kocisky, E. Grefenstette, L. Espeholt, W. Kay, M. Suleyman, and P. Blunsom.
Teaching machines to read and comprehend. In NIPS , 2015.
S. Hochreiter. Untersuchungen zu dynamischen neuronalen netzen. Master’s thesis , 1991.


key: chroma:document
string_value:
S. Hochreiter. Untersuchungen zu dynamischen neuronalen netzen. Master’s thesis , 1991.
S. Hochreiter and J Schmidhuber. Long short-term memory. Neural computation , 1997.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by
reducing internal covariate shift. abs/1502.03167, 2015.
D. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv:1412.6980 , 2014.
D Krueger and R. Memisevic. Regularizing rnns by stabilizing activations. ICLR , 2016.
9
Published as a conference paper at ICLR 2017
David Krueger, Tegan Maharaj, János Kramár, Mohammad Pezeshki, Nicolas Ballas, Nan Rosemary Ke, Anirudh Goyal, Yoshua Bengio, Hugo Larochelle, and Aaron Courville. Zoneout:
Regularizing rnns by randomly preserving hidden activations. arXiv:1606.01305 , 2016.
C. Laurent, G. Pereyra, P. Brakel, Y . Zhang, and Y . Bengio. Batch normalized recurrent neural
networks. ICASSP , 2016.


key: chroma:document
string_value:
C. Laurent, G. Pereyra, P. Brakel, Y . Zhang, and Y . Bengio. Batch normalized recurrent neural
networks. ICASSP , 2016.
Quoc V Le, N. Jaitly, and G. Hinton. A simple way to initialize recurrent networks of rectiﬁed linear
units. arXiv:1504.00941 , 2015.
Qianli Liao and Tomaso Poggio. Bridging the gaps between residual learning, recurrent neural
networks and visual cortex. arXiv:1604.03640 , 2016.
M. Mahoney. Large text compression benchmark. 2009.
M. P. Marcus, M. Marcinkiewicz, and B. Santorini. Building a large annotated corpus of english:
The penn treebank. Comput. Linguist. , 1993.
J. Martens and I. Sutskever. Learning recurrent neural networks with hessian-free optimization. In
ICML , 2011.
T. Mikolov, I. Sutskever, A. Deoras, H. Le, S. Kombrink, and J. Cernocky. Subword language
modeling with neural networks. preprint , 2012.
Yann Ollivier. Persistent contextual neural networks for learning symbolic data sequences. CoRR ,


key: chroma:document
string_value:
modeling with neural networks. preprint , 2012.
Yann Ollivier. Persistent contextual neural networks for learning symbolic data sequences. CoRR ,
abs/1306.0514, 2013.
Marius Pachitariu and Maneesh Sahani. Regularization and nonlinearities for neural language models: when are they needed? arXiv:1301.5650 , 2013.
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difﬁculty of training recurrent neural
networks. arXiv:1211.5063 , 2012.
H. Shimodaira. Improving predictive inference under covariate shift by weighting the log-likelihood
function. Journal of statistical planning and inference , 2000.
The Theano Development Team et al. Theano: A Python framework for fast computation of mathematical expressions. arXiv e-prints , abs/1605.02688, May 2016.
T. Tieleman and G. Hinton. Lecture 6.5—RmsProp: Divide the gradient by a running average of its
recent magnitude. COURSERA: Neural Networks for Machine Learning, 2012.


key: chroma:document
string_value:
recent magnitude. COURSERA: Neural Networks for Machine Learning, 2012.
Bart van Merriënboer, Dzmitry Bahdanau, Vincent Dumoulin, Dmitriy Serdyuk, David WardeFarley, Jan Chorowski, and Yoshua Bengio. Blocks and fuel: Frameworks for deep learning.
CoRR , abs/1506.00619, 2015. URL http://arxiv.org/abs/1506.00619 .
K. Xu, J. Ba, R. Kiros, A. Courville, R. Salakhutdinov, R. Zemel, and Y . Bengio. Show, attend and
tell: Neural image caption generation with visual attention. arXiv:1502.03044 , 2015.
L. Yao, A. Torabi, K. Cho, N. Ballas, C. Pal, H. Larochelle, and A. Courville. Describing videos by
exploiting temporal structure. In ICCV , 2015.
S. Zhang, Y . Wu, T. Che, Z. Lin, R. Memisevic, R. Salakhutdinov, and Y . Bengio. Architectural
complexity measures of recurrent neural networks. arXiv:1602.08210 , 2016.
10
Published as a conference paper at ICLR 2017


key: chroma:document
string_value:
complexity measures of recurrent neural networks. arXiv:1602.08210 , 2016.
10
Published as a conference paper at ICLR 2017
A C ONVERGENCE OF POPULATION STATISTICS
0.20
0.15
0.10
0.05
0.000.050.100.150.20mean of recurrent term
0.20
0.15
0.10
0.05
0.000.050.100.150.20mean of cell state
0 10 20 30 40 50
time steps0.00.51.01.52.0variance of recurrent term
0 10 20 30 40 50
time steps0.0010.0020.0030.0040.0050.0060.0070.0080.009variance of cell state
Figure 5: Convergence of population statistics to stationary distributions on the Penn Treebank task.
The horizontal axis denotes RNN time. Each curve corresponds to a single hidden unit. Only a
random subset of units is shown. See Section 3 for discussion.
B S ENSITIVITY TO INITIALIZATION OF 
In Section 4 we investigated the effect of initial 
on gradient ﬂow. To show the practical implications of this, we performed several experiments on the pMNIST and Penn Treebank benchmarks.
The resulting performances are shown in Figure 6.


key: chroma:document
string_value:
The resulting performances are shown in Figure 6.
ThepMNIST training curves conﬁrm that higher initial values of 
are detrimental to the optimization of the model. For the Penn Treebank task however, the effect is gone.
We believe this is explained by the difference in the nature of the two tasks. For pMNIST, the model
absorbs the input sequence and only at the end of the sequence does it make a prediction on which it
receives feedback. Learning from this feedback requires propagating the gradient all the way back
through the sequence.
In the Penn Treebank task on the other hand, the model makes a prediction at each timestep. At
each step of the backward pass, a fresh learning signal is added to the backpropagated gradient.
Essentially, the model is able to get off the ground by picking up short-term dependencies. This fails
onpMNIST wich is dominated by long-term dependencies (Arjovsky et al., 2015).
C T EACHING MACHINES TO READ AND COMPREHEND : TASK SETUP
We evaluate the models on the question answering task using the CNN corpus (Hermann et al.,
2015), with placeholders for the named entities. We follow a similar preprocessing pipeline as Hermann et al. (2015). During training, we randomly sample the examples with replacement and shufﬂe


key: chroma:document
string_value:
the order of the placeholders in each text inside the minibatch. We use a vocabulary of 65829 words.
We deviate from Hermann et al. (2015) in order to save computation: we use only the 4 most relevant
sentences from the description, as identiﬁed by a string matching procedure. Both the training and
validation sets are preprocessed in this way. Due to imprecision this heuristic sometimes strips the
11
Published as a conference paper at ICLR 2017
0 10000 20000 30000 40000 50000
training steps0.00.51.01.52.02.5cross entropyPermuted MNIST train
gamma 0.10
gamma 0.30
gamma 0.50
gamma 0.70
gamma 1.00
0 10000 20000 30000 40000 50000
training steps0.00.51.01.52.02.5cross entropyPermuted MNIST valid
gamma 0.10
gamma 0.30
gamma 0.50
gamma 0.70
gamma 1.00
0 5000 10000 15000
training steps0.800.850.900.951.001.051.10bits per characterPTB train
gamma 0.10
gamma 0.30
gamma 0.50
gamma 0.70


key: chroma:document
string_value:
gamma 0.10
gamma 0.30
gamma 0.50
gamma 0.70
gamma 1.00
0 5000 10000 15000
training steps1.001.021.041.061.081.10bits per characterPTB valid
gamma 0.10
gamma 0.30
gamma 0.50
gamma 0.70
gamma 1.00
Figure 6: Training curves on pMNIST and Penn Treebank for various initializations of 
.
answers from the passage, putting an upper bound of 57% on the validation accuracy that can be
achieved.
For the reported performances, the ﬁrst three models (LSTM, BN-LSTM and BN-everywhere) are
trained using the exact same hyperparameters, which were chosen because they work well for the
baseline. The hidden state is composed of 240 units. We use stochastic gradient descent on minibatches of size 64, with gradient clipping at 10 and step rule determined by Adam (Kingma & Ba,
2014) with learning rate 8105.
For BN-e* and BN-e**, we use the same hyperparameters except that we reduce the learning rate to
8104and the minibatch size to 40.


key: chroma:document
string_value:
8104and the minibatch size to 40.
D H YPERPARAMETER SEARCHES
Table 5 reports hyperparameter values that were tried in the experiments.
(a) MNIST and pMNIST
Learning rate: 1e-2, 1e-3, 1e-4
RMSProp momentum: 0.5, 0.9
Hidden state size: 100, 200, 400
Initial
: 1e-1, 3e-1, 5e-1, 7e-1, 1.0(b) Penn Treebank
Learning rate: 1e-1, 1e-2, 2e-2, 1e-3
Hidden state size: 800, 1000, 1200, 1500, 2000
Batch size: 32, 64, 100, 128
Initial
: 1e-1, 3e-1, 5e-1, 7e-1, 1.0
(c) Text8
Learning rate: 1e-1, 1e-2, 1e-3
Hidden state size: 500, 1000, 2000, 4000(d) Attentive Reader
Learning rate: 8e-3, 8e-4, 8e-5, 8e-6
Hidden state size: 60, 120, 240, 280
Table 5: Hyperparameter values that have been explored in the experiments.


key: chroma:document
string_value:
Hidden state size: 60, 120, 240, 280
Table 5: Hyperparameter values that have been explored in the experiments.
For MNIST and pMNIST, the hyperparameters were varied independently. For Penn Treebank, we
performed a full grid search on learning rate and hidden state size, and later performed a sensitivity
12
Published as a conference paper at ICLR 2017
analysis on the batch size and initial 
. For the text8 task and the experiments with the Attentive
Reader, we carried out a grid search on the learning rate and hidden state size.
The same values were tried for both the baseline and our BN-LSTM. In each case, our reported
results are those of the model with the best validation performance.
13


key: chroma:document
string_value:
Convolutional Random Walk Networks for Semantic Image Segmentation
Gedas Bertasius1, Lorenzo Torresani2, Stella X. Yu3, Jianbo Shi1
1University of Pennsylvania,2Dartmouth College,3UC Berkeley ICSI
{gberta,jshi}@seas.upenn.edu lt@dartmouth.edu stella.yu@berkeley.edu
Abstract
Most current semantic segmentation methods rely on
fully convolutional networks (FCNs). However, their use of
large receptive ﬁelds and many pooling layers cause low
spatial resolution inside the deep layers. This leads to
predictions with poor localization around the boundaries.
Prior work has attempted to address this issue by postprocessing predictions with CRFs or MRFs. But such models often fail to capture semantic relationships between objects, which causes spatially disjoint predictions. To overcome these problems, recent methods integrated CRFs or
MRFs into an FCN framework. The downside of these new
models is that they have much higher complexity than traditional FCNs, which renders training and testing more challenging.
In this work we introduce a simple, yet effective Convolutional Random Walk Network (RWN) that addresses the
issues of poor boundary localization and spatially fragmented predictions with very little increase in model complexity. Our proposed RWN jointly optimizes the objectives


key: chroma:document
string_value:
issues of poor boundary localization and spatially fragmented predictions with very little increase in model complexity. Our proposed RWN jointly optimizes the objectives
of pixelwise afﬁnity and semantic segmentation. It combines these two objectives via a novel random walk layer
that enforces consistent spatial grouping in the deep layers
of the network. Our RWN is implemented using standard
convolution and matrix multiplication. This allows an easy
integration into existing FCN frameworks and it enables
end-to-end training of the whole network via standard backpropagation. Our implementation of RWN requires just 131
additional parameters compared to the traditional FCNs,
and yet it consistently produces an improvement over the
FCNs on semantic segmentation and scene labeling.
1. Introduction
Fully convolutional networks (FCNs) were ﬁrst introduced in [20] where they were shown to yield signiﬁcant
improvements in semantic image segmentation. Adopting the FCN approach, many subsequent methods have
achieved even better performance [6, 27, 8, 12, 17, 6, 27, 8,
19, 23]. However, traditional FCN-based methods tend to
suffer from several limitations. Large receptive ﬁelds in the
Input
 DeepLab [6]
 DeepLab-CRF [6]


key: chroma:document
string_value:
suffer from several limitations. Large receptive ﬁelds in the
Input
 DeepLab [6]
 DeepLab-CRF [6]
Figure 1: Examples illustrating shortcomings of prior semantic segmentation methods. Segments produced by
FCNs are poorly localized around object boundaries, while
Dense-CRF produce spatially-disjoint object segments.
convolutional layers and the presence of pooling layers lead
to low spatial resolution in the deepest FCN layers. As a result, their predicted segments tend to be blobby and lack ﬁne
object boundary details. We report in Fig. 1 some examples
illustrating typical poor localization of objects in the outputs
of FCNs. Recently, Chen at al. [6] addressed this issue by
applying a Dense-CRF post-processing step [15] on top of
coarse FCN segmentations. However, such approaches often fail to accurately capture semantic relationships between
objects and lead to spatially fragmented segmentations (see
an example in the last column of Fig. 1).
To address these problems several recent methods integrated CRFs or MRFs directly into the FCN framework [27, 19, 17, 5]. However, these new models typically involve (1) a large number of parameters, (2) complex loss functions requiring specialized model training or
(3) recurrent layers, which make training and testing more


key: chroma:document
string_value:
(3) recurrent layers, which make training and testing more
complicated. We summarize the most prominent of these
approaches and their model complexities in Table 1.
We note that we do not claim that using complex loss
functions always makes the model overly complex and too
difﬁcult to use. If a complex loss is integrated into an FCN
framework such that the FCN can still be trained in a standard fashion, and produce better results than using standard losses, then such a model is beneﬁcial. However, in
the context of prior segmentation methods [27, 19, 17, 5],
such complex losses often require: 1) modifying the network structure (casting CNN into an RNN) [27, 5], or 2)
using a complicated multi-stage learning scheme, where
different layers are optimized during a different training
stage [19, 17]. Due to such complex training procedures,
1arXiv:1605.07681v3  [cs.CV]  8 May 2017
[6] [5] [23] [27] [19] RWN
requires post-processing? 3777 7 7
uses complex loss? 7773 3 7
requires recurrent layers? 7373 7 7
model size (in MB) 79 79 961 514 >1000 79


key: chroma:document
string_value:
uses complex loss? 7773 3 7
requires recurrent layers? 7373 7 7
model size (in MB) 79 79 961 514 >1000 79
Table 1: Summary of recent semantic segmentation models. For each model, we report whether it requires: (1) CRF
post-processing, (2) complex loss functions, or (3) recurrent layers. We also list the size of the model (using the
size of Caffe [13] models in MB). We note that unlike prior
methods, our RWN does not need post-processing, it is implemented using standard layers and loss functions, and it
also has a compact model.
which are adapted for speciﬁc tasks and datasets, these
models can be quite difﬁcult to adapt for new tasks and
datasets, which is disadvantageous.
Inspired by random walk methods [21, 3, 24], in this
work we introduce a simple, yet effective alternative to traditional FCNs: a Convolutional Random Walk Network
(RWN) that combines the strengths of FCNs and random walk methods. Our model addresses the issues of
(1) poor localization around the boundaries suffered by
FCNs and (2) spatially disjoint segments produced by dense
CRFs. Additionally, unlike recent semantic segmentation


key: chroma:document
string_value:
(1) poor localization around the boundaries suffered by
FCNs and (2) spatially disjoint segments produced by dense
CRFs. Additionally, unlike recent semantic segmentation
approaches [23, 27, 19, 17], our RWN does so without signiﬁcantly increasing the complexity of the model.
Our proposed RWN jointly optimizes (1) pixelwise afﬁnity and (2) semantic segmentation learning objectives that
are linked via a novel random walk layer, which enforces
spatial consistency in the deepest layers of the network. The
random walk layer is implemented via matrix multiplication. As a result, RWN seamlessly integrates both afﬁnity
and segmentation branches, and can be jointly trained endto-end via standard back-propagation with minimal modiﬁcations to the existing FCN framework. Additionally, our
implementation of RWN requires only 131additional parameters. Thus, the effective complexity of our model is
the same as the complexity of traditional FCNs (see Table 1). We compare our approach to several variants of the
DeepLab semantic segmentation system [6, 7], and show
that our proposed RWN consistently produces better performance over these baselines for the tasks of semantic segmentation and scene labeling.
2. Related Work
The recent introduction of fully convolutional networks


key: chroma:document
string_value:
that our proposed RWN consistently produces better performance over these baselines for the tasks of semantic segmentation and scene labeling.
2. Related Work
The recent introduction of fully convolutional networks
(FCNs) [20] has led to remarkable advances in semantic
segmentation. However, due to the large receptive ﬁelds
and many pooling layers, segments predicted by FCNs tend
to be blobby and lack ﬁne object boundary details. Recently
there have been several attempts to address these problems.
These approaches can be divided into several groups.The work in [4, 6, 25, 14, 4] used FCN predictions as
unary potentials in a separate globalization model that reﬁnes the segments using similarity cues based on regions
or boundaries. One disadvantage of these methods is that
the learning of the unary potentials and the training of the
globalization model are completely disjoint. As a result,
these methods often fail to capture semantic relationships
between objects, which produces segmentation results that
are spatially disjoint (see the right side of Fig. 1).
To address these issues, several recent methods [27, 19,
17] have proposed to integrate a CRF or a MRF into the network, thus enabling end-to-end training of the joint model.


key: chroma:document
string_value:
17] have proposed to integrate a CRF or a MRF into the network, thus enabling end-to-end training of the joint model.
However, the merging of these two models leads to a dramatic increase in complexity and number of parameters.
For instance, the method in [27], requires to cast the original
FCN into a Recurrent Neural Network (RNN), which renders the model much bigger in size (see Table 1). A recent
method [5] jointly predicts boundaries and segmentations,
and then combines them using a recurrent layer, which also
requires complex modiﬁcations to the existing FCN framework.
The work in [19] proposes to use local convolutional layers, which leads to a signiﬁcantly larger number of parameters. Similarly, the method in [17] proposes to model unary
and pairwise potentials by separate multi-scale branches.
This leads to a network with at least twice as many parameters as the traditional FCN and a much more complex multistage training procedure.
In addition to the above methods, it is worth mentioning
deconvolutional networks [23, 12], which use deconvolution and unpooling layers to recover ﬁne object details from
the coarse FCN predictions. However, in order to effectively recover ﬁne details one must employ nearly as many


key: chroma:document
string_value:
the coarse FCN predictions. However, in order to effectively recover ﬁne details one must employ nearly as many
deconvolutional layers as the number of convolutional layers, which yields a large growth in number of parameters
(see Table 1).
Unlike these prior methods, our implementation of RWN
needs only 131additional parameters over the base FCN.
These additional parameters represent only 0:0008% of the
total number of parameters in the network. In addition, our
RWN uses standard convolution and matrix multiplication.
Thus, it does not need to incorporate complicated loss functions or new complex layers [27, 5, 19, 17]. Finally, unlike the methods in [4, 6, 25, 14] that predict and reﬁne the
segmentations disjointly, our RWN model jointly optimizes
pixel afﬁnity and semantic segmentation in an end-to-end
fashion. Our experiments show that this leads to spatially
smoother segmentations.
3. Background
Random Graph Walks. Random walks are one of
the most widely known and used methods in graph theory [21]. Most notably, the concept of random walks led to
FC8 ActivationsDeepLab FCN Architecture
RGB ImageSparse Pairwise L1 Distances    Sparse Pixel    Similarity Matrix1 x 1 x k conv
   Ground Truth   Pixel Similarities


key: chroma:document
string_value:
RGB ImageSparse Pairwise L1 Distances    Sparse Pixel    Similarity Matrix1 x 1 x k conv
   Ground Truth   Pixel Similarities
 Exp
   Ground Truth     Segmentation
      Segmentation            Random Walk Layerˆy=AfAffinity Learning BranchPairwise L1  Distances
n⇥n⇥mRow-Normalization:n⇥n⇥3
f2Rn2⇥mA2Rn2⇥n2W2Rn2⇥n2F2Rn2⇥n2⇥kFigure 2: The architecture of our proposed Random Walk Network (RWN) (best viewed in color). Our RWN consists
of two branches: (1) one branch devoted to the segmentation predictions , and (2) another branch predicting pixel-level
afﬁnities. These two branches are then merged via a novel random walk layer that encourages spatially smooth segmentation
predictions. The entire RWN is jointly optimized end-to-end via a standard back-propagation algorithm.
the development of PageRank [24] and Personalized PageRank [3], which are widely used for many applications. Let
G= (V;E)denote an undirected graph with a set of verticesVand a set of edges E. Then a random walk in


key: chroma:document
string_value:
G= (V;E)denote an undirected graph with a set of verticesVand a set of edges E. Then a random walk in
such graph can be characterized by the transition probabilities between its vertices. Let Wbe a symmetric nn
afﬁnity matrix, where ndenotes the number of nodes in
the graph and where Wij2[0;1]denotes how similar the
nodesiandjare. In the context of a semantic segmentation problem, each pixel in the image can be viewed as
a separate node in the graph, where the similarity between
two nodes can be evaluated according to some metric (e.g.
color or texture similarity etc). Then let Dindicate a diagonalnnmatrix, which stores the degree values for each
node:Dii=Pn
j=1Wijfor alljexcepti=j. Then, we can
express our random walk transition matrix as A=D1W.
Given this setup, we want to model how the information
in the graph spreads if we start at a particular node, and
perform a random walk in this graph. Let ytbe an1vector denoting a node distribution at time t. In the context of
the PageRank algorithm, ytmay indicate the rank estimates
associated with each of the nWeb pages at time t. Then,


key: chroma:document
string_value:
the PageRank algorithm, ytmay indicate the rank estimates
associated with each of the nWeb pages at time t. Then,
according to the random walk theory, we can spread the
rank information in the graph by performing a one-step random walk. This process can be expressed as yt+1=Ayt,
whereyt+1denotes a newly obtained rank distribution after
one random walk step, the matrix Acontains the random
walk transition probabilities, and ytis the rank distribution
at time step t. Thus, we can observe that the information
among the nodes can be diffused, by simply multiplying the
random walk transition probability matrix A, with the rank
distribution ytat a particular time t. This process can be
repeated multiple times, until convergence is reached. For a
more detailed survey please see [21, 24].
Difference from MRF/CRF Approaches. CRFs andMRFs have been widely used in structured prediction problems [16]. Recently, CRFs and MRFs have also been integrated into the fully convolutional network framework for
semantic segmentation [27, 19, 17]. We want to stress that
while the goals of CRF/MRF and random walk methods are
the same (i.e. to globally propagate information in the graph
structures), the mechanism to achieve this objective is very


key: chroma:document
string_value:
the same (i.e. to globally propagate information in the graph
structures), the mechanism to achieve this objective is very
different in these two approaches. While MRFs and CRFs
typically employ graphs with a ﬁxed grid structure (e.g., one
where each node is connected to its four closest neighbors),
random walk methods are much more ﬂexible and can implement any arbitrary graph structure via the afﬁnity matrix
speciﬁcation. Thus, since our proposed RWN is based on
random walks, it can employ any arbitrary graph structure,
which can be beneﬁcial as different problems may require
different graph structures.
Additionally, to globally propagate information among
the nodes, MRFs and CRFs need to employ approximate
inference techniques, because exact inference tends to be
intractable in graphs with a grid structure. Integrating such
approximate inference techniques into the steps of FCN
training and prediction can be challenging and may require
lots of domain-speciﬁc modiﬁcations. In comparison, random walk methods globally propagate information among
the nodes via a simple matrix multiplication. Not only is
the matrix multiplication efﬁcient and exact, but it is also
easy to integrate into the traditional FCN framework for
both training and prediction schemes. Additionally, due to


key: chroma:document
string_value:
the matrix multiplication efﬁcient and exact, but it is also
easy to integrate into the traditional FCN framework for
both training and prediction schemes. Additionally, due to
the use of standard convolution and matrix multiplication
operations, our RWN can be trivially trained via standard
back-propagation in an end-to-end fashion.
4. Convolutional Random Walk Networks
In this work, our goal is to integrate a random walk process into the FCN architecture to encourage coherent semantic segmentation among pixels that are similar to each
other. Such a process introduces an explicit grouping mechanism, which should be beneﬁcial in addressing the issues
of (1) poor localization around the boundaries, and (2) spatially fragmented segmentations.
A schematic illustration of our proposed RWN architecture is presented in Fig. 2. Our RWN is a network composed of two branches: (1) one branch that predicts semantic segmentation potentials, and (2) another branch devoted
to predicting pixel-level afﬁnities. These two branches are
merged via a novel random walk layer that encourages spatially coherent semantic segmentation. The entire RWN can
be jointly optimized end-to-end. We now describe each of
the components of the RWN architecture in more detail.
4.1. Semantic Segmentation Branch
For the semantic segmentation branch, we present results


key: chroma:document
string_value:
the components of the RWN architecture in more detail.
4.1. Semantic Segmentation Branch
For the semantic segmentation branch, we present results
for several variants of the DeepLab segmentation systems,
including DeepLab-LargeFOV [6], DeepLab-attention [7],
and DeepLab-v2, which is one of the top performing segmentation systems. DeepLab-largeFOV is a fully convolutional adaptation of the VGG [26] architecture, which contains 16convolutional layers. DeepLab-attention [7], is a
multi-scale VGG based network, for which each multi-scale
branch focuses on a speciﬁc part of the image. Finally,
DeepLab-v2 is a multi-scale network based on the residual network [11] implementation. We note that even though
we use a DeepLab architecture in our experiments, other architectures such as [2] and many others could be integrated
into our framework.
4.2. Pixel-Level Afﬁnity Branch
To learn the pairwise pixel-level afﬁnities, we employ a
separate afﬁnity learning branch with its own learning objective (See Fig. 2). The afﬁnity branch is connected with
the inputnn3RGB image, and low-level conv1_1and


key: chroma:document
string_value:
the inputnn3RGB image, and low-level conv1_1and
conv1_2layers. The feature maps corresponding to these
layers arennin width and height but they have a different number of channels ( 3;64;and64respectively). Let k
be the total number of afﬁnity learning parameters (in our
casek= 3 + 64 + 64 = 131 ). Then, let Fbe a sparse
n2n2kmatrix that stores L1distances between each
pixel and all of its neighbors within a radius R, according
to each channel. Note that the distances are notsummed
up across the kchannels, but instead they are computed and
stored separately for each channel. The resulting matrix F
is then used as an input to the afﬁnity branch, as shown in
Figure 2.
The afﬁnity branch consists of a 11kconvolutional
layer and an exponential layer. The output of the exponential layer is then attached to the Euclidean loss layer and is
optimized to predict the ground truth pixel afﬁnities, which
are obtained from the original semantic segmentation anInput
 DeepLab_v2
 RWN_v2
Figure 3: A ﬁgure illustrating the segmentation results of


key: chroma:document
string_value:
are obtained from the original semantic segmentation anInput
 DeepLab_v2
 RWN_v2
Figure 3: A ﬁgure illustrating the segmentation results of
our RWN and the DeepLab-v2 network. Note that RWN
produced segmentations are spatially smoother and produce
less false positive predictions than the DeepLab-v2 system.
notations. Speciﬁcally, we set the ground truth afﬁnity between two pixels to 1if the pixels share the same semantic
label and have distance less than Rfrom each other. Note
thatF, which is used as an input to the afﬁnity branch, is a
sparse matrix, as only a small fraction of all the entries in F
are populated with non-zero values. The rest of the entries
are ignored during the computation.
Also note that we only use features from RGB, conv1_1
andconv1_2layers, because they are not affected by pooling, and thus, preserve the original spatial resolution. We
also experimented with using features from deeper FCN
layers such as fc6, and fc7. However, we observed that
features from deeper layers are highly correlated to the
predicted semantic segmentation unary potentials, which
causes redundancy and little improvement in the segmentation performance. We also experimented with using more


key: chroma:document
string_value:
features from deeper layers are highly correlated to the
predicted semantic segmentation unary potentials, which
causes redundancy and little improvement in the segmentation performance. We also experimented with using more
than one convolutional layer in the afﬁnity learning branch,
but observed that additional layers provide negligible improvements in accuracy.
4.3. Random Walk Layer
To integrate the semantic segmentation potentials and
our learned pixel-level afﬁnities, we introduce a novel random walk layer, which propagates the semantic segmentation information based on the learned afﬁnities. The random
walk layer is connected to the two bottom layers: (1) the fc8
layer containing the semantic segmentation potentials, and
(2) the afﬁnity layer that outputs a sparse n2n2random
walk transition matrix A. Then, let fdenote the activation values from the fc8layer, reshaped to the dimensions
ofn2m, wheren2refers to the number of pixels, and
mis the number of object classes in the dataset. A single
random walk layer implements one step of the random walk
Method aero bike bird boat bottle bus car cat chair cow table dog horse mbike person plant sheep sofa train tv mean overall


key: chroma:document
string_value:
random walk layer implements one step of the random walk
Method aero bike bird boat bottle bus car cat chair cow table dog horse mbike person plant sheep sofa train tv mean overall
DeepLab-largeFOV 79.8 71.5 78.9 70.9 72.1 87.9 81.2 85.7 46.9 80.9 56.5 82.6 77.9 79.3 80.1 64.4 77.6 52.7 80.3 70.0 73.8 76.0
RWN-largeFOV 81.6 72.1 82.3 72.0 75.4 89.1 82.5 87.4 49.1 83.6 57.9 84.8 80.7 80.2 81.2 65.7 79.7 55.5 81.5 74.0 75.8 77.9
DeepLab-attention 83.4 76.0 83.0 74.2 77.6 91.6 85.2 89.1 54.4 86.1 62.9 86.7 83.8 84.2 82.4 70.2 84.7 61.0 84.8 77.9 79.0 80.5


key: chroma:document
string_value:
RWN-attention 84.7 76.6 85.5 74.0 79.0 92.4 85.6 90.0 55.6 87.4 63.5 88.2 85.0 84.8 83.4 70.1 85.9 62.6 85.1 79.3 79.9 81.5
DeepLab-v2 85.5 50.6 86.9 74.4 82.7 93.1 88.4 91.9 62.1 89.7 71.5 90.3 86.2 86.3 84.6 75.1 87.6 72.2 87.8 81.3 81.4 83.4
RWN-v2 86.0 50.0 88.4 73.5 83.9 93.4 88.6 92.5 63.9 90.9 72.6 90.9 87.3 86.9 85.7 75.0 89.0 74.0 88.1 82.3 82.1 84.3
Table 2: Semantic segmentation results on the SBD dataset according to the per-pixel Intersection over Union evaluation
metric. From the results, we observe that our proposed RWN consistently outperforms DeepLab-LargeFOV , DeepLabattention, and DeepLab-v2 baselines.


key: chroma:document
string_value:
metric. From the results, we observe that our proposed RWN consistently outperforms DeepLab-LargeFOV , DeepLabattention, and DeepLab-v2 baselines.
process, which can be performed as ^y=Af, where ^yindicates the diffused segmentation predictions, and Adenotes
the random walk transition matrix.
The random walk layer is then attached to the softmax
loss layer, and is optimized to predict ground truth semantic
segmentations. One of the advantages of our proposed random walk layer is that it is implemented as a matrix multiplication, which makes it possible to back-propagate the
gradients to both (1) the afﬁnity branch and (2) the segmentation branch. Let the softmax-loss gradient be an n2m
matrix@L
@^y, wheren2is the number of pixels in the fc8layer,
andmis the number of predicted object classes. Then the
gradients, which are back-propagated to the semantic segmentation branch are computed as@L
@f=AT@L
@^y, whereAT
is the transposed random walk transition matrix. Also, the
gradients, that are back-propagated to the afﬁnity branch
are computed as@L
@A=@L


key: chroma:document
string_value:
gradients, that are back-propagated to the afﬁnity branch
are computed as@L
@A=@L
@^yfT, wherefTis amn2matrix
that contains transposed activation values from the fc8layer
of the segmentation branch. We note that@L
@Ais asparse
n2n2matrix, which means that the above matrix multiplication only considers the pixel pairs that correspond to
the non-zero entries in the random walk transition matrix A.
4.4. Random Walk Prediction at Testing
In the previous subsection, we mentioned that the prediction in the random walk layer can be done via a simple
matrix multiplication operation ^y=Af, whereAdenotes
the random walk transition matrix, and fdepicts the activation values from the fc8layer. Typically, we want to apply
multiple random walk steps until convergence is reached.
However, we also do not want to deviate too much from our
initial segmentation predictions, in case the random walk
transition matrix is not perfectly accurate, which is a reasonable expectation. Thus, our prediction scheme needs to
balance two effects: (1) propagating the segmentation information across the nodes using a random walk transition
matrix, and (2) not deviating too much from the initial segmentation.


key: chroma:document
string_value:
balance two effects: (1) propagating the segmentation information across the nodes using a random walk transition
matrix, and (2) not deviating too much from the initial segmentation.
This tradeoff between the two quantities is very similar to the idea behind MRF and CRF models, which try to
minimize an energy formed by unary and pairwise terms.
However, as discussed earlier, MRF and CRF methods tend
to use 1) grid structure graphs and 2) various approximateinference techniques to propagate segmentation information globally. In comparison, our random walk approach
is advantageous because it can use 1) any arbitrary graph
structure and 2) an exact matrix multiplication operation to
achieve the same goal.
Let us ﬁrst denote our segmentation prediction after t+1
random walk steps as ^yt+1. Then our general prediction
scheme can be written as:
^yt+1=A^yt+ (1)f (1)
wheredenotes a parameter [0;1]that controls the
tradeoff between (1) diffusing segmentation information
along the connections of a random walk transition matrix
and (2) not deviating too much from initial segmentation
values (i.e. the outputs of the last layer of the FCN). Let us
now initialize ^y0to contain the output values from the fc8


key: chroma:document
string_value:
values (i.e. the outputs of the last layer of the FCN). Let us
now initialize ^y0to contain the output values from the fc8
layer, which we denoted with f. Then we can write our prediction equation by substituting the recurrent expressions:
^yt+1= (A)t+1f+ (1)tX
i=0(A)if (2)
Now, because we want to apply our random walk procedure until convergence we set t=1. Then, because
our random walk transition matrix is stochastic we know
thatlimt!1(A)t+1= 0 . Furthermore, we can write
St=Pt
i=0Ai=I+A+A2+:::+At, whereIis
an identity matrix, and where Stdenotes a partial sum of
random walk transitions until iteration t. We can then write
StASt=IAt+1, which implies:
(IA)St=IAt+1(3)
From our previous derivation, we already know that
limt!1(A)t+1= 0, which implies that
S1= (IA)1(4)
Thus, our ﬁnal prediction equation, which corresponds
to applying repeated random walk steps until convergence,


key: chroma:document
string_value:
S1= (IA)1(4)
Thus, our ﬁnal prediction equation, which corresponds
to applying repeated random walk steps until convergence,
can be written as
^y1= (IA)1f (5)
Input
 DeepLab_v2-CRF
 RWN_v2
Figure 4: Comparison of segmentation results produced by
our RWN versus the DeepLab-v2-CRF system. It can be noticed that, despite not using any post-processing steps, our
RWN predicts ﬁne object details (e.g., bike wheels or plane
wings) more accurately than DeepLab-v2-CRF, which fails
to capture some these object parts.
In practice, the random walk transition matrix Ais pretty
large, and inverting it is impractical. To deal with this problem, we shrink the matrix (IA)using a simple and efﬁcient technique presented in [1], and then invert it to compute the ﬁnal segmentation. In the experimental section, we
show that such a prediction scheme produces solid results
and is still pretty efﬁcient ( 1 second per image). We also
note that we use this prediction scheme only during testing. During training we employ a scheme that uses a single


key: chroma:document
string_value:
note that we use this prediction scheme only during testing. During training we employ a scheme that uses a single
random walk step (but with a much larger radius), which is
faster. We explain this procedure in the next subsection.
4.5. Implementation Details
We jointly train our RWN in an end-to-end fashion for
2000 iterations, with a learning rate of 105,0:9momentum, the weight decay of 5105, and 15samples per batch.
For the RWN model, we set the tradeoff parameter to
0:01. During testing we set the random walk connectivity
radiusR= 5 and apply the random walk procedure until convergence. However, during training we set R= 40 ,
and apply a single random walk step. This training strategy
works well because increasing the radius size eliminates the
need for multiple random walk steps, which speeds up the
training. However, using R= 5 and applying an inﬁnite
number of random walk steps until convergence still yields
slightly better results (see study in 5.4), so we use it during testing. For all of our experiments, we use a Caffe library [13]. During training, we also employ data augmentation techniques such as cropping, and mirroring.
5. Experimental Results


key: chroma:document
string_value:
5. Experimental Results
In this section, we present our results for semantic segmentation on the SBD [10] dataset, which contains objects
and their per-pixel labels for 20Pascal VOC classes (excluding the background class). We also include scene label-Method mean IOU overall IOU
DeepLab-largeFOV-CRF 75.7 77.7
RWN-largeFOV 75.8 77.9
DeepLab-attention-CRF 79.9 81.6
RWN-attention 79.9 81.5
DeepLab-v2-CRF 81.9 84.2
RWN-v2 82.1 84.3
DeepLab-DT 76.6 78.7
RWN 76.7 78.8
Table 3: Quantitative comparison between our RWN model
and several variants of the DeepLab system that use a dense
CRF or a domain-transfer (DT) ﬁlter for post-processing.
These results suggest that our RWN acts as an effective
globalization scheme, since it produces results that are
similar or even better than the results achieved by postprocessing the DeepLab outputs with a CRF or DT.
ing results on the commonly used Stanford background [9]
and Sift Flow [18] datasets. We evaluate our segmentation results on these tasks using the standard metric of the


key: chroma:document
string_value:
ing results on the commonly used Stanford background [9]
and Sift Flow [18] datasets. We evaluate our segmentation results on these tasks using the standard metric of the
intersection-over-union (IOU) averaged per pixels across all
the classes from each dataset. We also include the classagnostic overall pixel intersection-over-union score, which
measures the per-pixel IOU across all classes.
We experiment with several variants of the DeepLab
system [6, 7] as our main baselines throughout our experiments: DeepLab-LargeFOV [6], DeepLab-attention [7],
and DeepLab-v2.
Our evaluations provide evidence for four conclusions:
In subsections 5.1, 5.2, we demonstrate that our proposed RWN outperforms DeepLab baselines for both
semantic segmentation, and scene labeling tasks.
In subsection 5.1, we demonstrate that, compared to
the dense CRF approaches, RWN predicts segmentations that are spatially smoother.
In Subsection 5.3, we show that our approach is more
efﬁcient than the denseCRF inference.
Finally, in Subsection, 5.4, we demonstrate that our
random walk layer is beneﬁcial and that our model is
ﬂexible to use different graph structures.


key: chroma:document
string_value:
random walk layer is beneﬁcial and that our model is
ﬂexible to use different graph structures.
5.1. Semantic Segmentation Task
Standard Evaluation. In Table 2, we present semantic
segmentation results on the Pascal SBD dataset [10], which
contains 8055 training and 2857 testing images. These results indicate that RWN consistently outperforms all three
of the DeepLab baselines. In Figure 3, we also compare
qualitative segmentation results of a DeepLab-v2 network
and our RWN model. We note that the RWN segmentations contain fewer false positive predictions and are also
spatially smoother across the object regions.
0 2 4 6 8 10 12 14 16 18 200.20.250.30.350.40.450.50.55
Trimap Width (in Pixels)Pixel Classification Error (%)Localization Around the Boundaries Error
  
DeepLab
RWNFigure 5: Localization error around the object boundaries
within a trimap. Compared to the DeepLab system (blue),
our RWN (red) achieves lower segmentation error around
object boundaries for all trimap widths.
Furthermore, in Table 3, we present experiments where
we compare RWN with models using dense CRFs [15] to
post-process the predictions of DeepLab systems. We also


key: chroma:document
string_value:
Furthermore, in Table 3, we present experiments where
we compare RWN with models using dense CRFs [15] to
post-process the predictions of DeepLab systems. We also
include DeepLab-DT [5], which uses domain-transfer ﬁltering to reﬁne the segmentations inside an FCN. Based on
these results, we observe that, despite not using any postprocessing, our RWN produces results similar to or even
better than the DeepLab models employing post-processing.
These results indicate that RWN can be used as a globalization mechanism to ensure spatial coherence in semantic
segmentation predictions. In Figure 4 we present qualitative results where we compare the ﬁnal segmentation predictions of RWN and the DeepLab-v2-CRF system. Based
on these qualitative results, we observe that RWN captures
more accurately the ﬁne details of the objects, such as the
bike wheels, or plane wings. The DeepLab-v2-CRF system
misses some of these object parts.
Localization Around the Boundaries. Earlier we
claimed that due to the use of large receptive ﬁelds and
many pooling layers, FCNs tend to produce blobby segmentations that lack ﬁne object boundary details. We want


key: chroma:document
string_value:
many pooling layers, FCNs tend to produce blobby segmentations that lack ﬁne object boundary details. We want
to show that our RWN produces more accurate segmentations around object boundaries the traditional FCNs. Thus,
adopting the practice from [15], we evaluate segmentation
accuracy around object boundaries. We do so by counting
the relative number of misclassiﬁed pixels within a narrow
band (“trimap”) surrounding the ground truth object boundaries. We present these results in Figure 5. The results show
that RWN achieves higher segmentation accuracy than the
DeepLab (DL) system for all trimap widths considered in
this test.
Spatial Smoothness. We also argued that applying the
dense CRF [15] as a post-processing technique often leads
to spatially fragmented segmentations (see the right side of
Fig. 1). How can we evaluate whether a given method produces spatially smooth or spatially fragmented segmenta-Method MF AP
DeepLab-largeFOV-CRF 0.676 0.457
RWN-largeFOV 0.703 0.494
DeepLab-attention-CRF 0.722 0.521
RWN-attention 0.747 0.556
DeepLab-v2-CRF 0.763 0.584


key: chroma:document
string_value:
RWN-attention 0.747 0.556
DeepLab-v2-CRF 0.763 0.584
RWN-v2 0.773 0.595
Table 4: Quantitative comparison of spatial segmentation
smoothness. We extract the boundaries from the predicted
segmentations and evaluate them against ground truth object boundaries using max F-score (MF) and average precision (AP) metrics. These results suggest that RWN segmentations are spatially smoother than the DeepLab-CRF
segmentations across all baselines.
tions? Intuitively, spatially fragmented segmentations will
produce many false boundaries that do not correspond to
actual object boundaries. Thus, to test the spatial smoothness of a given segmentation, we extract the boundaries
from the segmentation and then compare these boundaries
against ground truth object boundaries using the standard
maximum F-score (MF) and average precision (AP) metrics, as done in the popular BSDS benchmark [22]. We perform this experiment on the Pascal SBD dataset and present
these results in Table 4. We can see that the boundaries extracted from the RWN segmentations yield better MF and
AP results compared to the boundaries extracted from the
different variants of the DeepLab-CRF system. Thus, these
results suggest that RWN produces spatially smoother segmentations than DeepLab-CRF.
5.2. Scene Labeling


key: chroma:document
string_value:
results suggest that RWN produces spatially smoother segmentations than DeepLab-CRF.
5.2. Scene Labeling
We also tested our RWN on the task of scene labeling
using two popular datasets: Stanford Background [9] and
Sift Flow [18]. Stanford Background is a relatively small
dataset for scene labeling. It contains 715images, which
we randomly split into 600training images and 115testing images. In contrast, the Sift Flow dataset contains 2489
training examples and 201testing images. For all of our
experiments, we use the DeepLab-largeFOV [6] architecture since it is smaller and more efﬁcient to train and test.
To evaluate scene labeling results, we use the overall IOU
evaluation metric which is a commonly used metric for this
task. In Table 5, we present our scene labeling results on
both of these datasets. Our results indicate that our RWN
method outperforms the DeepLab baseline by 2:57%, and
2:54% on these two datasets, respectively.
5.3. Runtime Comparisons
We also include the runtime comparison of our RWN approach versus the denseCRF inference. We note that using a single core of a 2:7GHz Intel Core i7 processor, the
denseCRF inference requires 3.301 seconds per image on
Input Image
Iteration 0Iteration 50


key: chroma:document
string_value:
denseCRF inference requires 3.301 seconds per image on
Input Image
Iteration 0Iteration 50
Figure 6: A ﬁgure illustrating how the probability predictions change as we apply more random walk steps. Note
that the RWN predictions become more reﬁned and better localized around the object boundaries as more random
walk steps are applied.
average on a Pascal SBD dataset. In comparison, a single
iteration of a random walk, which is simply a sparse matrix
multiplication, takes 0.032 seconds on average on the same
Pascal SBD dataset. A DeepLab_v2 post-processed with
denseCRF achieves 81:9%IOU score on this same Pascal
SBD dataset. In comparison, RWN_v2 with a single random walk iteration and with R=40 (radius) achieves 82:2%
IOU, which is both more accurate and more than 100times
more efﬁcient than the denseCRF inference.
5.4. Ablation Experiments
Optimal Number of Random Walk Steps. In Figure 7,
we illustrate how the IOU accuracy changes when we use a
different number of random walk steps. We observe that the
segmentation accuracy keeps increasing as we apply more
random walk steps, and that it reaches its peak performance
when the random walk process converges, which indicates


key: chroma:document
string_value:
segmentation accuracy keeps increasing as we apply more
random walk steps, and that it reaches its peak performance
when the random walk process converges, which indicates
the effectiveness of our random walk step procedure. In
Figure 6, we also illustrate how the predicted object segmentation probabilities change as we apply more random
walk steps. We observe that the object boundaries become
much better localized as more iterations of random walk are
applied.
Radius Size. To analyze the effect of a radius size in
the RWN architecture, we test alternative versions of our
model with different radii sizes. Our results indicate, that
the RWN model produces similar results with different radii
in the interval of R > 3andR < 20if the random walk
step process is applied until convergence. We also note
that, if we select R= 40 , and apply a random walk step
only once , we can achieve the segmentation accuracy of
75:5%and77:6%according to the two evaluation metrics,
0 5 10 15 20 25 30 40 50 60 inf0.7350.740.7450.750.7550.760.765
Number of Random Walk StepsIOU AccuracyAccuracy versus the Number of Random Walk Steps


key: chroma:document
string_value:
Number of Random Walk StepsIOU AccuracyAccuracy versus the Number of Random Walk Steps
  
Mean−class IOUFigure 7: IOU accuracy as a function of the number of random walk steps. From this plot we observe that the segmentation accuracy keeps improving as we apply more random walk steps and that it reaches its peak when the random
walk process converges.
DeepLab-largeFOV RWN-largeFOV
Stanford-BG 65.74 68.31
Sift-Flow 67.31 69.85
Table 5: Scene labeling results on the Stanford Background
and Sift-Flow datasets measured according to the overall
IOU evaluation metric. We use a DeepLab-largeFOV network as base model, and show that our RWN yields better
results on both of these scene labeling datasets.
respectively. In comparison, choosing R= 5 and applying random walk until convergence yields the accuracies of
75:8%and77:9%, which is slightly better. However, note
that selecting R= 40 , and applying multiple random walk
steps does not yield any improvement in segmentation accuracy. These experiments show the ﬂexibility of our model
compared to the MRF or CRF models, which typically use
graphs with a ﬁxed grid structure. Our model has the ability
to use different graph structures depending on the problem.
6. Conclusion


key: chroma:document
string_value:
graphs with a ﬁxed grid structure. Our model has the ability
to use different graph structures depending on the problem.
6. Conclusion
In this work, we introduced Random Walk Networks
(RWNs), and showed that, compared to traditional fully
convolutional networks (FCNs), they produce improved accuracy for the same model complexity. Our RWN addresses
the issues of 1) poor localization around the segmentation
boundaries and 2) spatially disjoint segmentations. Additionally, our implementation of RWN uses only 131additional learnable parameters ( 0:0008% of the original number of the parameters in the network) and it can be easily
integrated into the standard FCN learning framework for a
joint end-to-end training. Finally, RWN provides a more
efﬁcient alternative to the denseCRF approaches.
Our future work includes experimenting with alternative RWN architectures and applying RWN to new domains
such as language processing or speech recognition.
References
[1] Pablo Arbelaez, J. Pont-Tuset, Jon Barron, F. Marqués, and
Jitendra Malik. Multiscale combinatorial grouping. In Computer Vision and Pattern Recognition (CVPR) , 2014. 6
[2] Vijay Badrinarayanan, Ankur Handa, and Roberto Cipolla.


key: chroma:document
string_value:
[2] Vijay Badrinarayanan, Ankur Handa, and Roberto Cipolla.
Segnet: A deep convolutional encoder-decoder architecture
for robust semantic pixel-wise labelling. arXiv preprint
arXiv:1505.07293 , 2015. 4
[3] Bahman Bahmani, Abdur Chowdhury, and Ashish Goel. Fast
incremental and personalized pagerank. Proc. VLDB Endow., 4(3):173–184, December 2010. 2, 3
[4] Gedas Bertasius, Jianbo Shi, and Lorenzo Torresani. Semantic segmentation with boundary neural ﬁelds. In The
IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , June 2016. 2
[5] Liang-Chieh Chen, Jonathan T. Barron, George Papandreou,
Kevin Murphy, and Alan L. Yuille. Semantic image segmentation with task-speciﬁc edge detection using cnns and a
discriminatively trained domain transform. CVPR , 2016. 1,
2, 7
[6] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos,


key: chroma:document
string_value:
2, 7
[6] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos,
Kevin Murphy, and Alan L. Yuille. Semantic image segmentation with deep convolutional nets and fully. In ICLR , 2015.
1, 2, 4, 6, 7
[7] Liang-Chieh Chen, Yi Yang, Jiang Wang, Wei Xu, and
Alan L. Yuille. Attention to scale: Scale-aware semantic
image segmentation. CVPR , 2016. 2, 4, 6
[8] Jifeng Dai, Kaiming He, and Jian Sun. Boxsup: Exploiting
bounding boxes to supervise convolutional networks for semantic segmentation. In The IEEE International Conference
on Computer Vision (ICCV) , December 2015. 1
[9] S. Gould, R. Fulton, and D. Koller. Decomposing a scene
into geometric and semantically consistent regions. In Proceedings of the International Conference on Computer Vision (ICCV) , 2009. 6, 7
[10] Bharath Hariharan, Pablo Arbelaez, Lubomir Bourdev,
Subhransu Maji, and Jitendra Malik. Semantic contours from
inverse detectors. In International Conference on Computer
Vision (ICCV) , 2011. 6


key: chroma:document
string_value:
Subhransu Maji, and Jitendra Malik. Semantic contours from
inverse detectors. In International Conference on Computer
Vision (ICCV) , 2011. 6
[11] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
Deep residual learning for image recognition. In The IEEE
Conference on Computer Vision and Pattern Recognition
(CVPR) , June 2016. 4
[12] Seunghoon Hong, Hyeonwoo Noh, and Bohyung Han. Decoupled deep neural network for semi-supervised semantic
segmentation. In NIPS) , December 2015. 1, 2
[13] Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey
Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama,
and Trevor Darrell. Caffe: Convolutional architecture for fast
feature embedding. arXiv preprint arXiv:1408.5093 , 2014.
2, 6
[14] Iasonas Kokkinos. Surpassing humans in boundary detection
using deep learning. CoRR , abs/1511.07386, 2015. 2[15] Philipp Krähenbühl and Vladlen Koltun. Efﬁcient inference
in fully connected crfs with gaussian edge potentials. In


key: chroma:document
string_value:
in fully connected crfs with gaussian edge potentials. In
J. Shawe-Taylor, R.S. Zemel, P.L. Bartlett, F. Pereira, and
K.Q. Weinberger, editors, Advances in Neural Information
Processing Systems 24 , pages 109–117. Curran Associates,
Inc., 2011. 1, 7
[16] John D. Lafferty, Andrew McCallum, and Fernando C. N.
Pereira. Conditional random ﬁelds: Probabilistic models for
segmenting and labeling sequence data. In Proceedings of
the Eighteenth International Conference on Machine Learning, ICML ’01, pages 282–289, San Francisco, CA, USA,
2001. Morgan Kaufmann Publishers Inc. 3
[17] Guosheng Lin, Chunhua Shen, Ian D. Reid, and Anton
van den Hengel. Efﬁcient piecewise training of deep
structured models for semantic segmentation. CoRR ,
abs/1504.01013, 2015. 1, 2, 3
[18] Ce Liu, Jenny Yuen, and Antonio Torralba. Nonparametric
Scene Parsing via Label Transfer , pages 207–236. Springer
International Publishing, Cham, 2016. 6, 7


key: chroma:document
string_value:
Scene Parsing via Label Transfer , pages 207–236. Springer
International Publishing, Cham, 2016. 6, 7
[19] Ziwei Liu, Xiaoxiao Li, Ping Luo, Chen Change Loy, and
Xiaoou Tang. Semantic image segmentation via deep parsing
network. In ICCV , 2015. 1, 2, 3
[20] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully
convolutional networks for semantic segmentation. In The
IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , June 2015. 1, 2
[21] Laszlo Lovasz. Random walks on graphs: A survey, 1993.
2, 3
[22] D. Martin, C. Fowlkes, D. Tal, and J. Malik. A database
of human segmented natural images and its application to
evaluating segmentation algorithms and measuring ecological statistics. In Proc. 8th Int’l Conf. Computer Vision , volume 2, pages 416–423, July 2001. 7
[23] Hyeonwoo Noh, Seunghoon Hong, and Bohyung Han.
Learning deconvolution network for semantic segmentation.
InComputer Vision (ICCV), 2015 IEEE International Conference on , 2015. 1, 2
[24] Lawrence Page, Sergey Brin, Rajeev Motwani, and Terry


key: chroma:document
string_value:
InComputer Vision (ICCV), 2015 IEEE International Conference on , 2015. 1, 2
[24] Lawrence Page, Sergey Brin, Rajeev Motwani, and Terry
Winograd. The pagerank citation ranking: Bringing order
to the web. Technical Report 1999-66, Stanford InfoLab,
November 1999. Previous number = SIDL-WP-1999-0120.
2, 3
[25] Xiaojuan Qi, Jianping Shi, Shu Liu, Renjie Liao, and Jiaya Jia. Semantic segmentation with object clique potential.
InThe IEEE International Conference on Computer Vision
(ICCV) , December 2015. 2
[26] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. CoRR ,
abs/1409.1556, 2014. 4
[27] Shuai Zheng, Sadeep Jayasumana, Bernardino RomeraParedes, Vibhav Vineet, Zhizhong Su, Dalong Du, Chang
Huang, and Philip Torr. Conditional random ﬁelds as recurrent neural networks. In International Conference on Computer Vision (ICCV) , 2015. 1, 2, 3


key: chroma:document
string_value:
Learning to learn by gradient descent
by gradient descent
Marcin Andrychowicz1, Misha Denil1, Sergio Gómez Colmenarejo1, Matthew W. Hoffman1,
David Pfau1, Tom Schaul1, Brendan Shillingford1;2, Nando de Freitas1;2;3
1Google DeepMind2University of Oxford3Canadian Institute for Advanced Research
marcin.andrychowicz@gmail.com
{mdenil,sergomez,mwhoffman,pfau,schaul}@google.com
brendan.shillingford@cs.ox.ac.uk ,nandodefreitas@google.com
Abstract
The move from hand-designed features to learned features in machine learning has
been wildly successful. In spite of this, optimization algorithms are still designed
by hand. In this paper we show how the design of an optimization algorithm can be
cast as a learning problem, allowing the algorithm to learn to exploit structure in
the problems of interest in an automatic way. Our learned algorithms, implemented
by LSTMs, outperform generic, hand-designed competitors on the tasks for which
they are trained, and also generalize well to new tasks with similar structure. We
demonstrate this on a number of tasks, including simple convex problems, training
neural networks, and styling images with neural art.


key: chroma:document
string_value:
demonstrate this on a number of tasks, including simple convex problems, training
neural networks, and styling images with neural art.
1 Introduction
Frequently, tasks in machine learning can be expressed as the problem of optimizing an objective
functionf()deﬁned over some domain 2. The goal in this case is to ﬁnd the minimizer
= arg min2f(). While any method capable of minimizing this objective function can be
applied, the standard approach for differentiable functions is some form of gradient descent, resulting
in a sequence of updates
t+1=ttrf(t):
The performance of vanilla gradient descent, however, is hampered by the fact that it only makes use
of gradients and ignores second-order information. Classical optimization techniques correct this
behavior by rescaling the gradient step using curvature information, typically via the Hessian matrix
of second-order partial derivatives—although other choices such as the generalized Gauss-Newton
matrix or Fisher information matrix are possible.
Much of the modern work in optimization is based around designing update rules tailored to speciﬁc
classes of problems, with the types of problems of interest differing between different research


key: chroma:document
string_value:
Much of the modern work in optimization is based around designing update rules tailored to speciﬁc
classes of problems, with the types of problems of interest differing between different research
communities. For example, in the deep learning community we have seen a proliferation of optimization methods specialized for high-dimensional, non-convex optimization problems. These include
momentum [Nesterov, 1983, Tseng, 1998], Rprop [Riedmiller and Braun, 1993], Adagrad [Duchi
et al., 2011], RMSprop [Tieleman and Hinton, 2012], and ADAM [Kingma and Ba, 2015]. More
focused methods can also be applied when more structure of the optimization problem is known
[Martens and Grosse, 2015]. In contrast, communities who focus on sparsity tend to favor very
different approaches [Donoho, 2006, Bach et al., 2012]. This is even more the case for combinatorial
optimization for which relaxations are often the norm [Nemhauser and Wolsey, 1988].
30th Conference on Neural Information Processing Systems (NIPS 2016), Barcelona, Spain.arXiv:1606.04474v2  [cs.NE]  30 Nov 2016
optimizeroptimizeeparameter updates
error signalFigure 1: The optimizer (left) is provided with
performance of the optimizee (right) and proposes


key: chroma:document
string_value:
optimizeroptimizeeparameter updates
error signalFigure 1: The optimizer (left) is provided with
performance of the optimizee (right) and proposes
updates to increase the optimizee’s performance.
[photos: Bobolas, 2009, Maley, 2011]This industry of optimizer design allows different communities to create optimization methods which exploit structure in their problems
of interest at the expense of potentially poor
performance on problems outside of that scope.
Moreover the No Free Lunch Theorems for Optimization [Wolpert and Macready, 1997] show
that in the setting of combinatorial optimization,
no algorithm is able to do better than a random
strategy in expectation. This suggests that specialization to a subclass of problems is in fact
theonly way that improved performance can be
achieved in general.
In this work we take a different tack and instead
propose to replace hand-designed update rules
with a learned update rule, which we call the optimizerg, speciﬁed by its own set of parameters
. This results in updates to the optimizee fof
the form
t+1=t+gt(rf(t);): (1)
A high level view of this process is shown in Figure 1. In what follows we will explicitly model


key: chroma:document
string_value:
A high level view of this process is shown in Figure 1. In what follows we will explicitly model
the update rule gusing a recurrent neural network (RNN) which maintains its own state and hence
dynamically updates as a function of its iterates.
1.1 Transfer learning and generalization
The goal of this work is to develop a procedure for constructing a learning algorithm which performs
well on a particular class of optimization problems. Casting algorithm design as a learning problem
allows us to specify the class of problems we are interested in through example problem instances.
This is in contrast to the ordinary approach of characterizing properties of interesting problems
analytically and using these analytical insights to design learning algorithms by hand.
It is informative to consider the meaning of generalization in this framework. In ordinary statistical
learning we have a particular function of interest, whose behavior is constrained through a data set of
example function evaluations. In choosing a model we specify a set of inductive biases about how
we think the function of interest should behave at points we have not observed, and generalization
corresponds to the capacity to make predictions about the behavior of the target function at novel
points. In our setting the examples are themselves problem instances , which means generalization
corresponds to the ability to transfer knowledge between different problems. This reuse of problem


key: chroma:document
string_value:
corresponds to the capacity to make predictions about the behavior of the target function at novel
points. In our setting the examples are themselves problem instances , which means generalization
corresponds to the ability to transfer knowledge between different problems. This reuse of problem
structure is commonly known as transfer learning , and is often treated as a subject in its own right.


key: chroma:document
string_value:
corresponds to the ability to transfer knowledge between different problems. This reuse of problem
structure is commonly known as transfer learning , and is often treated as a subject in its own right.
However, by taking a meta-learning perspective, we can cast the problem of transfer learning as one
of generalization, which is much better studied in the machine learning community.
One of the great success stories of deep-learning is that we can rely on the ability of deep networks to
generalize to new examples by learning interesting sub-structures. In this work we aim to leverage
this generalization power, but also to lift it from simple supervised learning to the more general
setting of optimization.
1.2 A brief history and related work
The idea of using learning to learn ormeta-learning to acquire knowledge or inductive biases has a
long history [Thrun and Pratt, 1998]. More recently, Lake et al. [2016] have argued forcefully for
its importance as a building block in artiﬁcial intelligence. Similarly, Santoro et al. [2016] frame
multi-task learning as generalization, however unlike our approach they directly train a base learner
rather than a training algorithm. In general these ideas involve learning which occurs at two different
time scales: rapid learning within tasks and more gradual, meta learning across many different tasks.


key: chroma:document
string_value:
rather than a training algorithm. In general these ideas involve learning which occurs at two different
time scales: rapid learning within tasks and more gradual, meta learning across many different tasks.
Perhaps the most general approach to meta-learning is that of Schmidhuber [1992, 1993]—building
on work from [Schmidhuber, 1987]—which considers networks that are able to modify their own
weights. Such a system is differentiable end-to-end, allowing both the network and the learning
2
algorithm to be trained jointly by gradient descent with few restrictions. However this generality
comes at the expense of making the learning rules very difﬁcult to train. Alternatively, the work
of Schmidhuber et al. [1997] uses the Success Story Algorithm to modify its search strategy rather
than gradient descent; a similar approach has been recently taken in Daniel et al. [2016] which uses
reinforcement learning to train a controller for selecting step-sizes.
Bengio et al. [1990, 1995] propose to learn updates which avoid back-propagation by using simple
parametric rules. In relation to the focus of this paper the work of Bengio et al. could be characterized
aslearning to learn without gradient descent by gradient descent . The work of Runarsson and


key: chroma:document
string_value:
aslearning to learn without gradient descent by gradient descent . The work of Runarsson and
Jonsson [2000] builds upon this work by replacing the simple rule with a neural network.
Cotter and Conwell [1990], and later Younger et al. [1999], also show ﬁxed-weight recurrent neural
networks can exhibit dynamic behavior without need to modify their network weights. Similarly this
has been shown in a ﬁltering context [e.g. Feldkamp and Puskorius, 1998], which is directly related
to simple multi-timescale optimizers [Sutton, 1992, Schraudolph, 1999].
Finally, the work of Younger et al. [2001] and Hochreiter et al. [2001] connects these different threads
of research by allowing for the output of backpropagation from one network to feed into an additional
learning network, with both networks trained jointly. Our approach to meta-learning builds on this
work by modifying the network architecture of the optimizer in order to scale this approach to larger
neural-network optimization problems.
2 Learning to learn with recurrent neural networks
In this work we consider directly parameterizing the optimizer. As a result, in a slight abuse of notation
we will write the ﬁnal optimizee parameters (f;)as a function of the optimizer parameters and


key: chroma:document
string_value:
we will write the ﬁnal optimizee parameters (f;)as a function of the optimizer parameters and
the function in question. We can then ask the question: What does it mean for an optimizer to be
good? Given a distribution of functions fwe will write the expected loss as
L() =Efh
f
(f;)i
: (2)
As noted earlier, we will take the update steps gtto be the output of a recurrent neural network m,
parameterized by , whose state we will denote explicitly with ht. Next, while the objective function
in(2)depends only on the ﬁnal parameter value, for training the optimizer it will be convenient to
have an objective that depends on the entire trajectory of optimization, for some horizon T,
L() =Ef"TX
t=1wtf(t)#
where t+1=t+gt;
gt
ht+1
=m(rt;ht;):(3)
Herewt2R0are arbitrary weights associated with each time-step and we will also use the notation
rt=rf(t). This formulation is equivalent to (2)whenwt=1[t=T], but later we will describe


key: chroma:document
string_value:
rt=rf(t). This formulation is equivalent to (2)whenwt=1[t=T], but later we will describe
why using different weights can prove useful.
We can minimize the value of L()using gradient descent on . The gradient estimate @L()=@ can
be computed by sampling a random function fand applying backpropagation to the computational
graph in Figure 2. We allow gradients to ﬂow along the solid edges in the graph, but gradients
along the dashed edges are dropped. Ignoring gradients along the dashed edges amounts to making
the assumption that the gradients of the optimizee do not depend on the optimizer parameters, i.e.
@rt
@= 0. This assumption allows us to avoid computing second derivatives of f.
Examining the objective in (3)we see that the gradient is non-zero only for terms where wt6= 0. If
we usewt=1[t=T]to match the original problem, then gradients of trajectory preﬁxes are zero
and only the ﬁnal optimization step provides information for training the optimizer. This renders
Backpropagation Through Time (BPTT) inefﬁcient. We solve this problem by relaxing the objective


key: chroma:document
string_value:
Backpropagation Through Time (BPTT) inefﬁcient. We solve this problem by relaxing the objective
such thatwt>0at intermediate points along the trajectory. This changes the objective function, but
allows us to train the optimizer on partial trajectories. For simplicity, in all our experiments we use
wt= 1for everyt.
3
OptimizeeOptimizert-2t-1t
mmm+++ft-1ftft-2∇t-2∇t-1∇tht-2ht-1htht+1gt-1gtθt-2θt-1θtθt+1gt-2Figure 2: Computational graph used for computing the gradient of the optimizer.
2.1 Coordinatewise LSTM optimizer
One challenge in applying RNNs in our setting is that we want to be able to optimize at least tens of
thousands of parameters. Optimizing at this scale with a fully connected RNN is not feasible as it
would require a huge hidden state and an enormous number of parameters. To avoid this difﬁculty we
will use an optimizer mwhich operates coordinatewise on the parameters of the objective function,
similar to other common update rules like RMSprop and ADAM. This coordinatewise network


key: chroma:document
string_value:
will use an optimizer mwhich operates coordinatewise on the parameters of the objective function,
similar to other common update rules like RMSprop and ADAM. This coordinatewise network
architecture allows us to use a very small network that only looks at a single coordinate to deﬁne the
optimizer and share optimizer parameters across different parameters of the optimizee.
Different behavior on each coordinate is achieved by using separate activations for each objective
function parameter. In addition to allowing us to use a small network for this optimizer, this setup has
the nice effect of making the optimizer invariant to the order of parameters in the network, since the
same update rule is used independently on each coordinate.
LSTM1fLSTMn∇1θ1θn++
…∇n………
…
Figure 3: One step of an LSTM optimizer. All
LSTMs have shared parameters, but separate hidden states.We implement the update rule for each coordinate using a two-layer Long Short Term Memory
(LSTM) network [Hochreiter and Schmidhuber,
1997], using the now-standard forget gate architecture. The network takes as input the optimizee gradient for a single coordinate as well
as the previous hidden state and outputs the update for the corresponding optimizee parameter.
We will refer to this architecture, illustrated in


key: chroma:document
string_value:
as the previous hidden state and outputs the update for the corresponding optimizee parameter.
We will refer to this architecture, illustrated in
Figure 3, as an LSTM optimizer.
The use of recurrence allows the LSTM to learn
dynamic update rules which integrate information from the history of gradients, similar to
momentum. This is known to have many desirable properties in convex optimization [see e.g.
Nesterov, 1983] and in fact many recent learning procedures—such as ADAM—use momentum in
their updates.
Preprocessing and postprocessing Optimizer inputs and outputs can have very different magnitudes depending on the class of function being optimized, but neural networks usually work robustly
only for inputs and outputs which are neither very small nor very large. In practice rescaling inputs
and outputs of an LSTM optimizer using suitable constants (shared across all timesteps and functions
f) is sufﬁcient to avoid this problem. In Appendix A we propose a different method of preprocessing
inputs to the optimizer inputs which is more robust and gives slightly better performance.
4
20 40 60 80 10010-1100101LossQuadratics
ADAM
RMSprop
SGD
NAG
LSTM
20 40 60 80 100
Step100MNIST


key: chroma:document
string_value:
ADAM
RMSprop
SGD
NAG
LSTM
20 40 60 80 100
Step100MNIST
120 140 160 180 200MNIST, 200 stepsFigure 4: Comparisons between learned and hand-crafted optimizers performance. Learned optimizers are shown with solid lines and hand-crafted optimizers are shown with dashed lines. Units for the
yaxis in the MNIST plots are logits. Left: Performance of different optimizers on randomly sampled
10-dimensional quadratic functions. Center: the LSTM optimizer outperforms standard methods
training the base network on MNIST. Right: Learning curves for steps 100-200 by an optimizer
trained to optimize for 100 steps (continuation of center plot).
3 Experiments
In all experiments the trained optimizers use two-layer LSTMs with 20 hidden units in each layer.
Each optimizer is trained by minimizing Equation 3 using truncated BPTT as described in Section 2.
The minimization is performed using ADAM with a learning rate chosen by random search.
We use early stopping when training the optimizer in order to avoid overﬁtting the optimizer. After
each epoch (some ﬁxed number of learning steps) we freeze the optimizer parameters and evaluate its
performance. We pick the best optimizer (according to the ﬁnal validation loss) and report its average


key: chroma:document
string_value:
performance. We pick the best optimizer (according to the ﬁnal validation loss) and report its average
performance on a number of freshly sampled test problems.
We compare our trained optimizers with standard optimizers used in Deep Learning: SGD, RMSprop,
ADAM, and Nesterov’s accelerated gradient (NAG). For each of these optimizer and each problem
we tuned the learning rate, and report results with the rate that gives the best ﬁnal error for each
problem. When an optimizer has more parameters than just a learning rate (e.g. decay coefﬁcients for
ADAM) we use the default values from the optim package in Torch7. Initial values of all optimizee
parameters were sampled from an IID Gaussian distribution.
3.1 Quadratic functions
In this experiment we consider training an optimizer on a simple class of synthetic 10-dimensional
quadratic functions. In particular we consider minimizing functions of the form
f() =kWyk2
2
for different 10x10 matrices Wand 10-dimensional vectors ywhose elements are drawn from an IID
Gaussian distribution. Optimizers were trained by optimizing random functions from this family and
tested on newly sampled functions from the same distribution. Each function was optimized for 100


key: chroma:document
string_value:
Gaussian distribution. Optimizers were trained by optimizing random functions from this family and
tested on newly sampled functions from the same distribution. Each function was optimized for 100
steps and the trained optimizers were unrolled for 20 steps. We have not used any preprocessing, nor
postprocessing.
Learning curves for different optimizers, averaged over many functions, are shown in the left plot of
Figure 4. Each curve corresponds to the average performance of one optimization algorithm on many
test functions; the solid curve shows the learned optimizer performance and dashed curves show
the performance of the standard baseline optimizers. It is clear the learned optimizers substantially
outperform the baselines in this setting.
3.2 Training a small neural network on MNIST
In this experiment we test whether trainable optimizers can learn to optimize a small neural network
on MNIST, and also explore how the trained optimizers generalize to functions beyond those they
were trained on. To this end, we train the optimizer to optimize a base network and explore a series
of modiﬁcations to the network architecture and training procedure at test time.
5
20 40 60 80 100100LossMNIST, 40 units
ADAM
RMSprop
SGD
NAG
LSTM
20 40 60 80 100
StepsMNIST, 2 layers


key: chroma:document
string_value:
ADAM
RMSprop
SGD
NAG
LSTM
20 40 60 80 100
StepsMNIST, 2 layers
20 40 60 80 100MNIST, ReLUFigure 5: Comparisons between learned and hand-crafted optimizers performance. Units for the
yaxis are logits. Left: Generalization to the different number of hidden units (40 instead of 20).
Center: Generalization to the different number of hidden layers (2 instead of 1). This optimization
problem is very hard, because the hidden layers are very narrow. Right: Training curves for an MLP
with 20 hidden units using ReLU activations. The LSTM optimizer was trained on an MLP with
sigmoid activations.
5 20 35 5010-1100Final Loss
LSTM
5 20 35 50
Hidden units
ADAM
5 20 35 50
NAG
Layers
1
2
Figure 6: Systematic study of ﬁnal MNIST performance as the optimizee architecture is varied,
using sigmoid non-linearities. The vertical dashed line in the left-most plot denotes the architecture
at which the LSTM is trained and the horizontal line shows the ﬁnal performance of the trained
optimizer in this setting.
In this setting the objective function f()is the cross entropy of a small MLP with parameters .


key: chroma:document
string_value:
optimizer in this setting.
In this setting the objective function f()is the cross entropy of a small MLP with parameters .
The values of fas well as the gradients @f()=@are estimated using random minibatches of 128
examples. The base network is an MLP with one hidden layer of 20 units using a sigmoid activation
function. The only source of variability between different runs is the initial value 0and randomness
in minibatch selection. Each optimization was run for 100 steps and the trained optimizers were
unrolled for 20 steps. We used input preprocessing described in Appendix A and rescaled the outputs
of the LSTM by the factor 0:1.
Learning curves for the base network using different optimizers are displayed in the center plot of
Figure 4. In this experiment NAG, ADAM, and RMSprop exhibit roughly equivalent performance the
LSTM optimizer outperforms them by a signiﬁcant margin. The right plot in Figure 4 compares the
performance of the LSTM optimizer if it is allowed to run for 200 steps, despite having been trained
to optimize for 100 steps. In this comparison we re-used the LSTM optimizer from the previous
experiment, and here we see that the LSTM optimizer continues to outperform the baseline optimizers
on this task.


key: chroma:document
string_value:
experiment, and here we see that the LSTM optimizer continues to outperform the baseline optimizers
on this task.
Generalization to different architectures Figure 5 shows three examples of applying the LSTM
optimizer to train networks with different architectures than the base network on which it was trained.
The modiﬁcations are (from left to right) (1) an MLP with 40 hidden units instead of 20, (2) a
network with two hidden layers instead of one, and (3) a network using ReLU activations instead of
sigmoid. In the ﬁrst two cases the LSTM optimizer generalizes well, and continues to outperform
the hand-designed baselines despite operating outside of its training regime. However, changing
the activation function to ReLU makes the dynamics of the learning procedure sufﬁciently different
that the learned optimizer is no longer able to generalize. Finally, in Figure 6 we show the results
of systematically varying the tested architecture; for the LSTM results we again used the optimizer
trained using 1 layer of 20 units and sigmoid non-linearities. Note that in this setting where the
6
200 400 600 800 1000100LossCIFAR-10
200 400 600 800 1000
Step100CIFAR-5
200 400 600 800 100010-1CIFAR-2


key: chroma:document
string_value:
200 400 600 800 1000
Step100CIFAR-5
200 400 600 800 100010-1CIFAR-2
ADAM
RMSprop
SGD
NAG
LSTM
LSTM-subFigure 7: Optimization performance on the CIFAR-10 dataset and subsets. Shown on the left is the
LSTM optimizer versus various baselines trained on CIFAR-10 and tested on a held-out test set. The
two plots on the right are the performance of these optimizers on subsets of the CIFAR labels. The
additional optimizer LSTM-sub has been trained only on the heldout labels and is hence transferring
to a completely novel dataset.
test-set problems are similar enough to those in the training set we see even better generalization than
the baseline optimizers.
3.3 Training a convolutional network on CIFAR-10
Next we test the performance of the trained neural optimizers on optimizing classiﬁcation performance
for the CIFAR-10 dataset [Krizhevsky, 2009]. In these experiments we used a model with both
convolutional and feed-forward layers. In particular, the model used for these experiments includes
three convolutional layers with max pooling followed by a fully-connected layer with 32 hidden units;


key: chroma:document
string_value:
three convolutional layers with max pooling followed by a fully-connected layer with 32 hidden units;
all non-linearities were ReLU activations with batch normalization.
The coordinatewise network decomposition introduced in Section 2.1—and used in the previous
experiment—utilizes a single LSTM architecture with shared weights, but separate hidden states,
for each optimizee parameter. We found that this decomposition was not sufﬁcient for the model
architecture introduced in this section due to the differences between the fully connected and convolutional layers. Instead we modify the optimizer by introducing two LSTMs: one proposes parameter
updates for the fully connected layers and the other updates the convolutional layer parameters. Like
the previous LSTM optimizer we still utilize a coordinatewise decomposition with shared weights
and individual hidden states, however LSTM weights are now shared only between parameters of the
same type (i.e. fully-connected vs. convolutional).
The performance of this trained optimizer compared against the baseline techniques is shown in
Figure 7. The left-most plot displays the results of using the optimizer to ﬁt a classiﬁer on a held-out
test set. The additional two plots on the right display the performance of the trained optimizer on


key: chroma:document
string_value:
test set. The additional two plots on the right display the performance of the trained optimizer on
modiﬁed datasets which only contain a subset of the labels, i.e. the CIFAR-2 dataset only contains
data corresponding to 2 of the 10 labels. Additionally we include an optimizer LSTM-sub which was
only trained on the held-out labels.
In all these examples we can see that the LSTM optimizer learns much more quickly than the baseline
optimizers, with signiﬁcant boosts in performance for the CIFAR-5 and especially CIFAR-2 datsets.
We also see that the optimizers trained only on a disjoint subset of the data is hardly effected by this
difference and transfers well to the additional dataset.
3.4 Neural Art
The recent work on artistic style transfer using convolutional networks, or Neural Art [Gatys et al.,
2015], gives a natural testbed for our method, since each content and style image pair gives rise to a
different optimization problem. Each Neural Art problem starts from a content image ,c, and a style
image ,s, and is given by
f() =Lcontent (c;) +Lstyle(s;) +
Lreg()


key: chroma:document
string_value:
image ,s, and is given by
f() =Lcontent (c;) +Lstyle(s;) +
Lreg()
The minimizer of fis the styled image . The ﬁrst two terms try to match the content and style of
the styled image to that of their ﬁrst argument, and the third term is a regularizer that encourages
smoothness in the styled image. Details can be found in [Gatys et al., 2015].
7
20 40 60 80 100 120
StepLossNeural art, training resolution
20 40 60 80 100 120
StepDouble resolution
ADAM
RMSprop
SGD
NAG
LSTMFigure 8: Optimization curves for Neural Art. Content images come from the test set, which was not
used during the LSTM optimizer training. Note: the y-axis is in log scale and we zoom in on the
interesting portion of this plot. Left: Applying the training style at the training resolution. Right:
Applying the test style at double the training resolution.
Figure 9: Examples of images styled using the LSTM optimizer. Each triple consists of the content
image (left), style (right) and image generated by the LSTM optimizer (center). Left: The result of


key: chroma:document
string_value:
image (left), style (right) and image generated by the LSTM optimizer (center). Left: The result of
applying the training style at the training resolution to a test image. Right: The result of applying a
new style to a test image at double the resolution on which the optimizer was trained.
We train optimizers using only 1 style and 1800 content images taken from ImageNet [Deng et al.,
2009]. We randomly select 100 content images for testing and 20 content images for validation of
trained optimizers. We train the optimizer on 64x64 content images from ImageNet and one ﬁxed
style image. We then test how well it generalizes to a different style image and higher resolution
(128x128). Each image was optimized for 128 steps and trained optimizers were unrolled for 32
steps. Figure 9 shows the result of styling two different images using the LSTM optimizer. The
LSTM optimizer uses inputs preprocessing described in Appendix A and no postprocessing. See
Appendix C for additional images.
Figure 8 compares the performance of the LSTM optimizer to standard optimization algorithms. The
LSTM optimizer outperforms all standard optimizers if the resolution and style image are the same
as the ones on which it was trained. Moreover, it continues to perform very well when both the
resolution and style are changed at test time.


key: chroma:document
string_value:
as the ones on which it was trained. Moreover, it continues to perform very well when both the
resolution and style are changed at test time.
Finally, in Appendix B we qualitatively examine the behavior of the step directions generated by the
learned optimizer.
4 Conclusion
We have shown how to cast the design of optimization algorithms as a learning problem, which
enables us to train optimizers that are specialized to particular classes of functions. Our experiments
have conﬁrmed that learned neural optimizers compare favorably against state-of-the-art optimization
methods used in deep learning. We witnessed a remarkable degree of transfer, with for example the
LSTM optimizer trained on 12,288 parameter neural art tasks being able to generalize to tasks with
49,152 parameters, different styles, and different content images all at the same time. We observed
similar impressive results when transferring to different architectures in the MNIST task.
The results on the CIFAR image labeling task show that the LSTM optimizers outperform handengineered optimizers when transferring to datasets drawn from the same data distribution.
References
F. Bach, R. Jenatton, J. Mairal, and G. Obozinski. Optimization with sparsity-inducing penalties. Foundations
and Trends in Machine Learning , 4(1):1–106, 2012.
8


key: chroma:document
string_value:
and Trends in Machine Learning , 4(1):1–106, 2012.
8
S. Bengio, Y . Bengio, and J. Cloutier. On the search for new learning rules for ANNs. Neural Processing Letters ,
2(4):26–30, 1995.
Y . Bengio, S. Bengio, and J. Cloutier. Learning a synaptic learning rule . Université de Montréal, Département
d’informatique et de recherche opérationnelle, 1990.
Y . Bengio, N. Boulanger-Lewandowski, and R. Pascanu. Advances in optimizing recurrent networks. In
International Conference on Acoustics, Speech and Signal Processing , pages 8624–8628. IEEE, 2013.
F. Bobolas. brain-neurons, 2009. URL https://www.flickr.com/photos/fbobolas/3822222947 . Creative Commons Attribution-ShareAlike 2.0 Generic.
N. E. Cotter and P. R. Conwell. Fixed-weight networks can learn. In International Joint Conference on Neural
Networks , pages 553–559, 1990.
C. Daniel, J. Taylor, and S. Nowozin. Learning step size controllers for robust neural network training. In


key: chroma:document
string_value:
Networks , pages 553–559, 1990.
C. Daniel, J. Taylor, and S. Nowozin. Learning step size controllers for robust neural network training. In
Association for the Advancement of Artiﬁcial Intelligence , 2016.
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database.
InComputer Vision and Pattern Recognition , pages 248–255. IEEE, 2009.
D. L. Donoho. Compressed sensing. Transactions on Information Theory , 52(4):1289–1306, 2006.
J. Duchi, E. Hazan, and Y . Singer. Adaptive subgradient methods for online learning and stochastic optimization.
Journal of Machine Learning Research , 12:2121–2159, 2011.
L. A. Feldkamp and G. V . Puskorius. A signal processing framework based on dynamic neural networks
with application to problems in adaptation, ﬁltering, and classiﬁcation. Proceedings of the IEEE , 86(11):
2259–2277, 1998.
L. A. Gatys, A. S. Ecker, and M. Bethge. A neural algorithm of artistic style. arXiv Report 1508.06576, 2015.


key: chroma:document
string_value:
L. A. Gatys, A. S. Ecker, and M. Bethge. A neural algorithm of artistic style. arXiv Report 1508.06576, 2015.
A. Graves, G. Wayne, and I. Danihkela. Neural Turing machines. arXiv Report 1410.5401, 2014.
S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation , 9(8):1735–1780, 1997.
S. Hochreiter, A. S. Younger, and P. R. Conwell. Learning to learn using gradient descent. In International
Conference on Artiﬁcial Neural Networks , pages 87–94. Springer, 2001.
D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In International Conference on Learning
Representations , 2015.
A. Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
B. M. Lake, T. D. Ullman, J. B. Tenenbaum, and S. J. Gershman. Building machines that learn and think like
people. arXiv Report 1604.00289, 2016.


key: chroma:document
string_value:
people. arXiv Report 1604.00289, 2016.
T. Maley. neuron, 2011. URL https://www.flickr.com/photos/taylortotz101/6280077898 . Creative
Commons Attribution 2.0 Generic.
J. Martens and R. Grosse. Optimizing neural networks with Kronecker-factored approximate curvature. In
International Conference on Machine Learning , pages 2408–2417, 2015.
G. L. Nemhauser and L. A. Wolsey. Integer and combinatorial optimization . John Wiley & Sons, 1988.
Y . Nesterov. A method of solving a convex programming problem with convergence rate o (1/k2). In Soviet
Mathematics Doklady , volume 27, pages 372–376, 1983.
J. Nocedal and S. Wright. Numerical optimization . Springer Science & Business Media, 2006.
M. Riedmiller and H. Braun. A direct adaptive method for faster backpropagation learning: The RPROP
algorithm. In International Conference on Neural Networks , pages 586–591, 1993.
T. P. Runarsson and M. T. Jonsson. Evolution and design of distributed learning rules. In IEEE Symposium on


key: chroma:document
string_value:
T. P. Runarsson and M. T. Jonsson. Evolution and design of distributed learning rules. In IEEE Symposium on
Combinations of Evolutionary Computation and Neural Networks , pages 59–63. IEEE, 2000.
A. Santoro, S. Bartunov, M. Botvinick, D. Wierstra, and T. Lillicrap. Meta-learning with memory-augmented
neural networks. In International Conference on Machine Learning , 2016.
J. Schmidhuber. Evolutionary principles in self-referential learning; On learning how to learn: The meta-meta-...
hook. PhD thesis, Institut f. Informatik, Tech. Univ. Munich, 1987.
J. Schmidhuber. Learning to control fast-weight memories: An alternative to dynamic recurrent networks.
Neural Computation , 4(1):131–139, 1992.
J. Schmidhuber. A neural network that embeds its own meta-levels. In International Conference on Neural
Networks , pages 407–412. IEEE, 1993.
J. Schmidhuber, J. Zhao, and M. Wiering. Shifting inductive bias with success-story algorithm, adaptive levin
search, and incremental self-improvement. Machine Learning , 28(1):105–130, 1997.


key: chroma:document
string_value:
search, and incremental self-improvement. Machine Learning , 28(1):105–130, 1997.
N. N. Schraudolph. Local gain adaptation in stochastic gradient descent. In International Conference on
Artiﬁcial Neural Networks , volume 2, pages 569–574, 1999.
R. S. Sutton. Adapting bias by gradient descent: An incremental version of delta-bar-delta. In Association for
the Advancement of Artiﬁcial Intelligence , pages 171–176, 1992.
S. Thrun and L. Pratt. Learning to learn . Springer Science & Business Media, 1998.
T. Tieleman and G. Hinton. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent
magnitude. COURSERA: Neural Networks for Machine Learning , 4:2, 2012.
P. Tseng. An incremental gradient (-projection) method with momentum term and adaptive stepsize rule. Journal
on Optimization , 8(2):506–531, 1998.
D. H. Wolpert and W. G. Macready. No free lunch theorems for optimization. Transactions on Evolutionary
Computation , 1(1):67–82, 1997.
9


key: chroma:document
string_value:
Computation , 1(1):67–82, 1997.
9
A. S. Younger, P. R. Conwell, and N. E. Cotter. Fixed-weight on-line learning. Transactions on Neural Networks ,
10(2):272–283, 1999.
A. S. Younger, S. Hochreiter, and P. R. Conwell. Meta-learning with backpropagation. In International Joint
Conference on Neural Networks , 2001.
10
0 64 128−8−24
0 64 128−2616
LSTM
ADAM
SGDFigure 10: Updates proposed by different optimizers at different optimization steps for two different
coordinates.
A Gradient preprocessing
One potential challenge in training optimizers is that different input coordinates (i.e. the gradients
w.r.t. different optimizee parameters) can have very different magnitudes. This is indeed the case e.g.
when the optimizee is a neural network and different parameters correspond to weights in different
layers. This can make training an optimizer difﬁcult, because neural networks naturally disregard
small variations in input signals and concentrate on bigger input values.
To this aim we propose to preprocess the optimizer’s inputs. One solution would be to give the


key: chroma:document
string_value:
small variations in input signals and concentrate on bigger input values.
To this aim we propose to preprocess the optimizer’s inputs. One solution would be to give the
optimizer (log(jrj);sgn(r))as an input, whereris the gradient in the current timestep. This has a
problem that log(jrj)diverges forr! 0. Therefore, we use the following preprocessing formula
rk!(
log(jrj)
p;sgn(r)
ifjrjep
(1;epr) otherwise
wherep>0is a parameter controlling how small gradients are disregarded (we use p= 10 in all our
experiments).
We noticed that just rescaling all inputs by an appropriate constant instead also works ﬁne, but the
proposed preprocessing seems to be more robust and gives slightly better results on some problems.
B Visualizations
Visualizing optimizers is inherently difﬁcult because their proposed updates are functions of the full
optimization trajectory. In this section we try to peek into the decisions made by the LSTM optimizer,
trained on the neural art task.
Histories of updates We select a single optimizee parameter (one color channel of one pixel in the


key: chroma:document
string_value:
trained on the neural art task.
Histories of updates We select a single optimizee parameter (one color channel of one pixel in the
styled image) and trace the updates proposed to this coordinate by the LSTM optimizer over a single
trajectory of optimization. We also record the updates that would have been proposed by both SGD
and ADAM if they followed the same trajectory of iterates. Figure 10 shows the trajectory of updates
for two different optimizee parameters. From the plots it is clear that the trained optimizer makes
bigger updates than SGD and ADAM. It is also visible that it uses some kind of momentum, but its
updates are more noisy than those proposed by ADAM which may be interpreted as having a shorter
time-scale momentum.
Proposed update as a function of current gradient Another way to visualize the optimizer
behavior is to look at the proposed update gtfor a single coordinate as a function of the current
gradient evaluation rt. We follow the same procedure as in the previous experiment, and visualize
the proposed updates for a few selected time steps.
These results are shown in Figures 11–13. In these plots, the x-axis is the current value of the gradient
for the chosen coordinate, and the y-axis shows the update that each optimizer would propose should
the corresponding gradient value be observed. The history of gradient observations is the same for all


key: chroma:document
string_value:
for the chosen coordinate, and the y-axis shows the update that each optimizer would propose should
the corresponding gradient value be observed. The history of gradient observations is the same for all
methods and follows the trajectory of the LSTM optimizer.
11
The shape of this function for the LSTM optimizer is often step-like, which is also the case for
ADAM. Surprisingly the step is sometimes in the opposite direction as for ADAM, i.e. the bigger the
gradient, the bigger the update.
C Neural Art
Shown below are additional examples of images styled using the LSTM optimizer. Each triple
consists of the content image (left), style (right) and image generated by the LSTM optimizer (center).
D Information sharing between coordinates
In previous sections we considered a coordinatewise architecture, which corresponds by analogy
to a learned version of RMSprop or ADAM. Although diagonal methods are quite effective in
practice, we can also consider learning more sophisticated optimizers that take the correlations
between coordinates into effect. To this end, we introduce a mechanism allowing different LSTMs to
communicate with each other.
Global averaging cells
The simplest solution is to designate a subset of the cells in each LSTM layer for communication.
These cells operate like normal LSTM cells, but their outgoing activations are averaged at each step


key: chroma:document
string_value:
These cells operate like normal LSTM cells, but their outgoing activations are averaged at each step
across all coordinates. These global averaging cells (GACs) are sufﬁcient to allow the networks to
implement L2 gradient clipping [Bengio et al., 2013] assuming that each LSTM can compute the
square of the gradient. This architecture is denoted as an LSTM+GAC optimizer.
NTM-BFGS optimizer
We also consider augmenting the LSTM+GAC architecture with an external memory that is shared
between coordinates. Such a memory, if appropriately designed could allow the optimizer to learn
algorithms similar to (low-memory) approximations to Newton’s method, e.g. (L-)BFGS [see Nocedal
and Wright, 2006]. The reason for this interpretation is that such methods can be seen as a set of
independent processes working coordinatewise, but communicating through the inverse Hessian
approximation stored in the memory. We designed a memory architecture that, in theory, allows the
12
Step 1−10010
Step 2 Step 3 Step 4
Step 5−10010
Step 6 Step 7 Step 8
Step 9−10010
Step 10 Step 11 Step 12
Step 13−10010
Step 14 Step 15 Step 16
Step 17−10010


key: chroma:document
string_value:
Step 6 Step 7 Step 8
Step 9−10010
Step 10 Step 11 Step 12
Step 13−10010
Step 14 Step 15 Step 16
Step 17−10010
Step 18 Step 19 Step 20
Step 21−10010
Step 22 Step 23 Step 24
Step 25−10010
Step 26 Step 27 Step 28
−400 0 400
Step 29−10010
−400 0 400
Step 30−400 0 400
Step 31−400 0 400
Step 32Figure 11: The proposed update direction for a single coordinate over 32 steps.
13
Step 1−10010
Step 2 Step 3 Step 4
Step 5−10010
Step 6 Step 7 Step 8
Step 9−10010
Step 10 Step 11 Step 12
Step 13−10010
Step 14 Step 15 Step 16
Step 17−10010
Step 18 Step 19 Step 20
Step 21−10010
Step 22 Step 23 Step 24
Step 25−10010
Step 26 Step 27 Step 28
−400 0 400
Step 29−10010
−400 0 400
Step 30−400 0 400
Step 31−400 0 400
Step 32Figure 12: The proposed update direction for a single coordinate over 32 steps.
14
Step 1−10010
Step 2 Step 3 Step 4
Step 5−10010
Step 6 Step 7 Step 8
Step 9−10010


key: chroma:document
string_value:
14
Step 1−10010
Step 2 Step 3 Step 4
Step 5−10010
Step 6 Step 7 Step 8
Step 9−10010
Step 10 Step 11 Step 12
Step 13−10010
Step 14 Step 15 Step 16
Step 17−10010
Step 18 Step 19 Step 20
Step 21−10010
Step 22 Step 23 Step 24
Step 25−10010
Step 26 Step 27 Step 28
−400 0 400
Step 29−10010
−400 0 400
Step 30−400 0 400
Step 31−400 0 400
Step 32Figure 13: The proposed update direction for a single coordinate over 32 steps.
15
LSTMLSTMLSTMLSTMLSTMLSTMLSTMr1MLSTMr2r3…………rni1i2i3…………inx=
LSTMLSTMLSTMLSTMLSTMLSTMLSTMLSTMa1a2a3…………anb1b2b3…………bnOuterProductMtΔMtMt+1+=Figure 14: Left: NTM-BFGS read operation. Right: NTM-BFGS write operation.
network to simulate (L-)BFGS, motivated by the approximate Newton method BFGS, named for
Broyden, Fletcher, Goldfarb, and Shanno. We call this architecture an NTM-BFGS optimizer , because


key: chroma:document
string_value:
Broyden, Fletcher, Goldfarb, and Shanno. We call this architecture an NTM-BFGS optimizer , because
its use of external memory is similar to the Neural Turing Machine [Graves et al., 2014]. The pivotal
differences between our construction and the NTM are (1) our memory allows only low-rank updates;
(2) the controller (including read/write heads) operates coordinatewise.
In BFGS an explicit estimate of the full (inverse) Hessian is built up from the sequence of observed
gradients. We can write a skeletonized version of the BFGS algorithm, using Mtto represent the
inverse Hessian approximation at iteration t, as follows
gt= read(Mt;t)
t+1=t+gt
Mt+1= write(Mt;t;gt):
Here we have packed up all of the details of the BFGS algorithm into the suggestively named read
andwrite operations, which operate on the inverse Hessian approximation Mt. In BFGS these
operations have speciﬁc forms, for example read(Mt;t) =Mtrf(t)is a speciﬁc matrix-vector
multiplication and the BFGS write operation corresponds to a particular low-rank update of Mt.


key: chroma:document
string_value:
multiplication and the BFGS write operation corresponds to a particular low-rank update of Mt.
In this work we preserve the structure of the BFGS updates, but discard their particular form. More
speciﬁcally the read operation remains a matrix-vector multiplication but the form of the vector
used is learned. Similarly, the write operation remains a low-rank update, but the vectors involved
are also learned. Conveniently, this structure of interaction with a large dynamically updated state
corresponds in a fairly direct way to the architecture of a Neural Turing Machine (NTM), where Mt
corresponds to the NTM memory [Graves et al., 2014].
Our NTM-BFGS optimizer uses an LSTM+GAC as a controller; however, instead of producing the
update directly we attach one or more read and write heads to the controller. Each read head produces
a read vector rtwhich is combined with the memory to produce a read result itwhich is fed back
into the controller at the following time step. Each write head produces two outputs, a left write
vectoratand a right write vector bt. The two write vectors are used to update the memory state by
accumulating their outer product. The read and write operation for a single head is diagrammed in
Figure 14 and the way read and write heads are attached to the controller is depicted in Figure 15.


key: chroma:document
string_value:
Figure 14 and the way read and write heads are attached to the controller is depicted in Figure 15.
In can be shown that NTM-BFGS with one read head and 3 write heads can simulate inverse Hessian
BFGS assuming that the controller can compute arbitrary (coordinatewise) functions and have access
to 2 GACs.
NTM-L-BFGS optimizer
In cases where memory is constrained we can follow the example of L-BFGS and maintain a low
rank approximation of the full memory ( vis. inverse Hessian). The simplest way to do this is to store a
sliding history of the left and right write vectors, allowing us to form the matrix vector multiplication
required by the read operation efﬁciently.
16
Controller         ReadWriteMtrtht+1Mt+1rt+1itatbt∇thtLSTM1LSTM2…gt
LSTMk∇kikakbkrkhkhkgkFigure 15: Left: Interaction between the controller and the external memory in NTM-BFGS. The
controller is composed of replicated coordinatewise LSTMs (possibly with GACs), but the read and
write operations are global across all coordinates. Right: A single LSTM for the kth coordinate in


key: chroma:document
string_value:
write operations are global across all coordinates. Right: A single LSTM for the kth coordinate in
the NTM-BFGS controller. Note that here we have dropped the time index tto simplify notation.
17


key: chroma:document
string_value:
Layer Normalization
Jimmy Lei Ba
University of Toronto
jimmy@psi.toronto.eduJamie Ryan Kiros
University of Toronto
rkiros@cs.toronto.eduGeoffrey E. Hinton
University of Toronto
and Google Inc.
hinton@cs.toronto.edu
Abstract
Training state-of-the-art, deep neural networks is computationally expensive. One
way to reduce the training time is to normalize the activities of the neurons. A
recently introduced technique called batch normalization uses the distribution of
the summed input to a neuron over a mini-batch of training cases to compute a
mean and variance which are then used to normalize the summed input to that
neuron on each training case. This signiﬁcantly reduces the training time in feedforward neural networks. However, the effect of batch normalization is dependent
on the mini-batch size and it is not obvious how to apply it to recurrent neural networks. In this paper, we transpose batch normalization into layer normalization by
computing the mean and variance used for normalization from all of the summed
inputs to the neurons in a layer on a single training case. Like batch normalization,
we also give each neuron its own adaptive bias and gain which are applied after
the normalization but before the non-linearity. Unlike batch normalization, layer


key: chroma:document
string_value:
we also give each neuron its own adaptive bias and gain which are applied after
the normalization but before the non-linearity. Unlike batch normalization, layer
normalization performs exactly the same computation at training and test times.
It is also straightforward to apply to recurrent neural networks by computing the
normalization statistics separately at each time step. Layer normalization is very
effective at stabilizing the hidden state dynamics in recurrent networks. Empirically, we show that layer normalization can substantially reduce the training time
compared with previously published techniques.
1 Introduction
Deep neural networks trained with some version of Stochastic Gradient Descent have been shown
to substantially outperform previous approaches on various supervised learning tasks in computer
vision [Krizhevsky et al., 2012] and speech processing [Hinton et al., 2012]. But state-of-the-art
deep neural networks often require many days of training. It is possible to speed-up the learning
by computing gradients for different subsets of the training cases on different machines or splitting
the neural network itself over many machines [Dean et al., 2012], but this can require a lot of communication and complex software. It also tends to lead to rapidly diminishing returns as the degree
of parallelization increases. An orthogonal approach is to modify the computations performed in


key: chroma:document
string_value:
the neural network itself over many machines [Dean et al., 2012], but this can require a lot of communication and complex software. It also tends to lead to rapidly diminishing returns as the degree
of parallelization increases. An orthogonal approach is to modify the computations performed in
the forward pass of the neural net to make learning easier. Recently, batch normalization [Ioffe and


key: chroma:document
string_value:
the forward pass of the neural net to make learning easier. Recently, batch normalization [Ioffe and
Szegedy, 2015] has been proposed to reduce training time by including additional normalization
stages in deep neural networks. The normalization standardizes each summed input using its mean
and its standard deviation across the training data. Feedforward neural networks trained using batch
normalization converge faster even with simple SGD. In addition to training time improvement, the
stochasticity from the batch statistics serves as a regularizer during training.
Despite its simplicity, batch normalization requires running averages of the summed input statistics. In feed-forward networks with ﬁxed depth, it is straightforward to store the statistics separately
for each hidden layer. However, the summed inputs to the recurrent neurons in a recurrent neural network (RNN) often vary with the length of the sequence so applying batch normalization to
RNNs appears to require different statistics for different time-steps. Furthermore, batch normaliza-arXiv:1607.06450v1  [stat.ML]  21 Jul 2016
tion cannot be applied to online learning tasks or to extremely large distributed models where the
minibatches have to be small.
This paper introduces layer normalization, a simple normalization method to improve the training
speed for various neural network models. Unlike batch normalization, the proposed method directly


key: chroma:document
string_value:
This paper introduces layer normalization, a simple normalization method to improve the training
speed for various neural network models. Unlike batch normalization, the proposed method directly
estimates the normalization statistics from the summed inputs to the neurons within a hidden layer
so the normalization does not introduce any new dependencies between training cases. We show that
layer normalization works well for RNNs and improves both the training time and the generalization
performance of several existing RNN models.
2 Background
A feed-forward neural network is a non-linear mapping from a input pattern xto an output vector
y. Consider the lthhidden layer in a deep feed-forward, neural network, and let albe the vector
representation of the summed inputs to the neurons in that layer. The summed inputs are computed
through a linear projection with the weight matrix Wland the bottom-up inputs hlgiven as follows:
al
i=wl
i>hlhl+1
i=f(al
i+bl
i) (1)
wheref()is an element-wise non-linear function and wl
iis the incoming weights to the ithhidden
units andbl
iis the scalar bias parameter. The parameters in the neural network are learnt using
gradient-based optimization algorithms with the gradients being computed by back-propagation.


key: chroma:document
string_value:
units andbl
iis the scalar bias parameter. The parameters in the neural network are learnt using
gradient-based optimization algorithms with the gradients being computed by back-propagation.
One of the challenges of deep learning is that the gradients with respect to the weights in one layer
are highly dependent on the outputs of the neurons in the previous layer especially if these outputs
change in a highly correlated way. Batch normalization [Ioffe and Szegedy, 2015] was proposed
to reduce such undesirable “covariate shift”. The method normalizes the summed inputs to each
hidden unit over the training cases. Speciﬁcally, for the ithsummed input in the lthlayer, the batch
normalization method rescales the summed inputs according to their variances under the distribution
of the data
al
i=gl
i
l
i
al
il
i
l
i=E
xP(x)
al
i
l
i=r
E
xP(x)h
al
il
i2i
(2)
where al


key: chroma:document
string_value:
l
i=r
E
xP(x)h
al
il
i2i
(2)
where al
iis normalized summed inputs to the ithhidden unit in the lthlayer andgiis a gain parameter scaling the normalized activation before the non-linear activation function. Note the expectation
is under the whole training data distribution. It is typically impractical to compute the expectations
in Eq. (2) exactly, since it would require forward passes through the whole training dataset with the
current set of weights. Instead, andare estimated using the empirical samples from the current
mini-batch. This puts constraints on the size of a mini-batch and it is hard to apply to recurrent
neural networks.
3 Layer normalization
We now consider the layer normalization method which is designed to overcome the drawbacks of
batch normalization.
Notice that changes in the output of one layer will tend to cause highly correlated changes in the
summed inputs to the next layer, especially with ReLU units whose outputs can change by a lot.
This suggests the “covariate shift” problem can be reduced by ﬁxing the mean and the variance of
the summed inputs within each layer. We, thus, compute the layer normalization statistics over all
the hidden units in the same layer as follows:
l=1


key: chroma:document
string_value:
the summed inputs within each layer. We, thus, compute the layer normalization statistics over all
the hidden units in the same layer as follows:
l=1
HHX
i=1al
il=vuut1
HHX
i=1
al
il2(3)
whereHdenotes the number of hidden units in a layer. The difference between Eq. (2) and Eq. (3)
is that under layer normalization, all the hidden units in a layer share the same normalization terms
and, but different training cases have different normalization terms. Unlike batch normalization,
layer normaliztion does not impose any constraint on the size of a mini-batch and it can be used in
the pure online regime with batch size 1.
2
3.1 Layer normalized recurrent neural networks
The recent sequence to sequence models [Sutskever et al., 2014] utilize compact recurrent neural
networks to solve sequential prediction problems in natural language processing. It is common
among the NLP tasks to have different sentence lengths for different training cases. This is easy to
deal with in an RNN because the same weights are used at every time-step. But when we apply batch
normalization to an RNN in the obvious way, we need to to compute and store separate statistics for


key: chroma:document
string_value:
normalization to an RNN in the obvious way, we need to to compute and store separate statistics for
each time step in a sequence. This is problematic if a test sequence is longer than any of the training
sequences. Layer normalization does not have such problem because its normalization terms depend
only on the summed inputs to a layer at the current time-step. It also has only one set of gain and
bias parameters shared over all time-steps.
In a standard RNN, the summed inputs in the recurrent layer are computed from the current input
xtand previous vector of hidden states ht1which are computed as at=Whhht1+Wxhxt. The
layer normalized recurrent layer re-centers and re-scales its activations using the extra normalization
terms similar to Eq. (3):
ht=fhg
t
att
+bi
t=1
HHX
i=1at
it=vuut1
HHX
i=1(at
it)2(4)
whereWhhis the recurrent hidden to hidden weights and Wxhare the bottom up input to hidden
weights.is the element-wise multiplication between two vectors. bandgare deﬁned as the bias
and gain parameters of the same dimension as ht.


key: chroma:document
string_value:
weights.is the element-wise multiplication between two vectors. bandgare deﬁned as the bias
and gain parameters of the same dimension as ht.
In a standard RNN, there is a tendency for the average magnitude of the summed inputs to the recurrent units to either grow or shrink at every time-step, leading to exploding or vanishing gradients. In
a layer normalized RNN, the normalization terms make it invariant to re-scaling all of the summed
inputs to a layer, which results in much more stable hidden-to-hidden dynamics.
4 Related work
Batch normalization has been previously extended to recurrent neural networks [Laurent et al., 2015,
Amodei et al., 2015, Cooijmans et al., 2016]. The previous work [Cooijmans et al., 2016] suggests
the best performance of recurrent batch normalization is obtained by keeping independent normalization statistics for each time-step. The authors show that initializing the gain parameter in the
recurrent batch normalization layer to 0.1 makes signiﬁcant difference in the ﬁnal performance of
the model. Our work is also related to weight normalization [Salimans and Kingma, 2016]. In
weight normalization, instead of the variance, the L2 norm of the incoming weights is used to


key: chroma:document
string_value:
weight normalization, instead of the variance, the L2 norm of the incoming weights is used to
normalize the summed inputs to a neuron. Applying either weight normalization or batch normalization using expected statistics is equivalent to have a different parameterization of the original
feed-forward neural network. Re-parameterization in the ReLU network was studied in the Pathnormalized SGD [Neyshabur et al., 2015]. Our proposed layer normalization method, however, is
not a re-parameterization of the original neural network. The layer normalized model, thus, has
different invariance properties than the other methods, that we will study in the following section.
5 Analysis
In this section, we investigate the invariance properties of different normalization schemes.
5.1 Invariance under weights and data transformations
The proposed layer normalization is related to batch normalization and weight normalization. Although, their normalization scalars are computed differently, these methods can be summarized as
normalizing the summed inputs aito a neuron through the two scalars and. They also learn an
adaptive bias band gaingfor each neuron after the normalization.
hi=f(gi
i(aii) +bi) (5)
Note that for layer normalization and batch normalization, andis computed according to Eq. 2


key: chroma:document
string_value:
i(aii) +bi) (5)
Note that for layer normalization and batch normalization, andis computed according to Eq. 2
and 3. In weight normalization, is 0, and=kwk2.
3
Weight matrix Weight matrix Weight vector Dataset Dataset Single training case
re-scaling re-centering re-scaling re-scaling re-centering re-scaling
Batch norm Invariant No Invariant Invariant Invariant No
Weight norm Invariant No Invariant No No No
Layer norm Invariant Invariant No Invariant No Invariant
Table 1: Invariance properties under the normalization methods.
Table 1 highlights the following invariance results for three normalization methods.
Weight re-scaling and re-centering: First, observe that under batch normalization and weight
normalization, any re-scaling to the incoming weights wiof a single neuron has no effect on the
normalized summed inputs to a neuron. To be precise, under batch and weight normalization, if
the weight vector is scaled by , the two scalar andwill also be scaled by . The normalized
summed inputs stays the same before and after scaling. So the batch and weight normalization are


key: chroma:document
string_value:
summed inputs stays the same before and after scaling. So the batch and weight normalization are
invariant to the re-scaling of the weights. Layer normalization, on the other hand, is not invariant
to the individual scaling of the single weight vectors. Instead, layer normalization is invariant to
scaling of the entire weight matrix and invariant to a shift to all of the incoming weights in the
weight matrix. Let there be two sets of model parameters ,0whose weight matrices WandW0
differ by a scaling factor and all of the incoming weights in W0are also shifted by a constant
vector 
, that isW0=W+1
>. Under layer normalization, the two models effectively compute
the same output:
h0=f(g
0(W0x0) +b) =f(g
0
(W+1
>)x0
+b)
=f(g
(Wx) +b) =h: (6)
Notice that if normalization is only applied to the input before the weights, the model will not be
invariant to re-scaling and re-centering of the weights.
Data re-scaling and re-centering: We can show that all the normalization methods are invariant


key: chroma:document
string_value:
Data re-scaling and re-centering: We can show that all the normalization methods are invariant
to re-scaling the dataset by verifying that the summed inputs of neurons stays constant under the
changes. Furthermore, layer normalization is invariant to re-scaling of individual training cases,
because the normalization scalars andin Eq. (3) only depend on the current input data. Let x0
be a new data point obtained by re-scaling xby. Then we have,
h0
i=f(gi
0
w>
ix00
+bi) =f(gi

w>
ix
+bi) =hi: (7)
It is easy to see re-scaling individual data points does not change the model’s prediction under layer
normalization. Similar to the re-centering of the weight matrix in layer normalization, we can also
show that batch normalization is invariant to re-centering of the dataset.
5.2 Geometry of parameter space during learning
We have investigated the invariance of the model’s prediction under re-centering and re-scaling of
the parameters. Learning, however, can behave very differently under different parameterizations,


key: chroma:document
string_value:
the parameters. Learning, however, can behave very differently under different parameterizations,
even though the models express the same underlying function. In this section, we analyze learning
behavior through the geometry and the manifold of the parameter space. We show that the normalization scalar can implicitly reduce learning rate and makes learning more stable.
5.2.1 Riemannian metric
The learnable parameters in a statistical model form a smooth manifold that consists of all possible
input-output relations of the model. For models whose output is a probability distribution, a natural
way to measure the separation of two points on this manifold is the Kullback-Leibler divergence
between their model output distributions. Under the KL divergence metric, the parameter space is a
Riemannian manifold.
The curvature of a Riemannian manifold is entirely captured by its Riemannian metric, whose
quadratic form is denoted as ds2. That is the inﬁnitesimal distance in the tangent space at a point
in the parameter space. Intuitively, it measures the changes in the model output from the parameter
space along a tangent direction. The Riemannian metric under KL was previously studied [Amari,
1998] and was shown to be well approximated under second order Taylor expansion using the Fisher
4
information matrix:
ds2= D KL


key: chroma:document
string_value:
1998] and was shown to be well approximated under second order Taylor expansion using the Fisher
4
information matrix:
ds2= D KL
P(yjx;)kP(yjx;+)
1
2>F(); (8)
F() = E
xP(x);yP(yjx)"
@logP(yjx;)
@@logP(yjx;)
@>#
; (9)
where,is a small change to the parameters. The Riemannian metric above presents a geometric
view of parameter spaces. The following analysis of the Riemannian metric provides some insight
into how normalization methods could help in training neural networks.
5.2.2 The geometry of normalized generalized linear models
We focus our geometric analysis on the generalized linear model. The results from the following
analysis can be easily applied to understand deep neural networks with block-diagonal approximation to the Fisher information matrix, where each block corresponds to the parameters for a single
neuron.
A generalized linear model (GLM) can be regarded as parameterizing an output distribution from
the exponential family using a weight vector wand bias scalar b. To be consistent with the previous


key: chroma:document
string_value:
A generalized linear model (GLM) can be regarded as parameterizing an output distribution from
the exponential family using a weight vector wand bias scalar b. To be consistent with the previous
sections, the log likelihood of the GLM can be written using the summed inputs aas the following:
logP(yjx;w;b) =(a+b)y(a+b)
+c(y;); (10)
E[yjx] =f(a+b) =f(w>x+b);Var[yjx] =f0(a+b); (11)
where,f()is the transfer function that is the analog of the non-linearity in neural networks, f0()
is the derivative of the transfer function, ()is a real valued function and c()is the log partition function. is a constant that scales the output variance. Assume a H-dimensional output
vector y= [y1;y2;;yH]is modeled using Hindependent GLMs and logP(yjx;W;b) =PH
i=1logP(yijx;wi;bi). LetWbe the weight matrix whose rows are the weight vectors of the


key: chroma:document
string_value:
i=1logP(yijx;wi;bi). LetWbe the weight matrix whose rows are the weight vectors of the
individual GLMs, bdenote the bias vector of length Handvec()denote the Kronecker vector operator. The Fisher information matrix for the multi-dimensional GLM with respect to its parameters
= [w>
1;b1;;w>
H;bH]>= vec([W;b]>)is simply the expected Kronecker product of the data
features and the output covariance matrix:
F() = E
xP(x)Cov[yjx]
2

xx>x
x>1
: (12)
We obtain normalized GLMs by applying the normalization methods to the summed inputs ain
the original model through and. Without loss of generality, we denote Fas the Fisher information matrix under the normalized multi-dimensional GLM with the additional gain parameters
= vec([W;b;g]>):
F() =2
664F11 F1H
.........
FH1 FHH3
775;Fij=E
xP(x)2
664Cov[yi; yjjx]


key: chroma:document
string_value:
FH1 FHH3
775;Fij=E
xP(x)2
664Cov[yi; yjjx]
22
664gigj
iji>
jigi
iigi(ajj)
ij
>
jgj
j1ajj
j
>
jgj(aii)
ijaii
i(aii)(ajj)
ij3
7753
775
(13)
i=x@i
@wiaii
i@i
@wi: (14)
Implicit learning rate reduction through the growth of the weight vector: Notice that, comparing to standard GLM, the block Fijalong the weight vector widirection is scaled by the gain
parameters and the normalization scalar i. If the norm of the weight vector wigrows twice as large,
even though the model’s output remains the same, the Fisher information matrix will be different.
The curvature along the widirection will change by a factor of1
2because the iwill also be twice


key: chroma:document
string_value:
The curvature along the widirection will change by a factor of1
2because the iwill also be twice
as large. As a result, for the same parameter update in the normalized model, the norm of the weight
vector effectively controls the learning rate for the weight vector. During learning, it is harder to
change the orientation of the weight vector with large norm. The normalization methods, therefore,
5
050100150200250300
iteration x 30034353637383940414243mean Recall@1Image Retrieval (Validation)
Order-Embedding + LN
Order-Embedding(a) Recall@1
050100150200250300
iteration x 3007172737475767778mean Recall@5Image Retrieval (Validation)
Order-Embedding + LN
Order-Embedding (b) Recall@5
050100150200250300
iteration x 30084858687888990mean Recall@10Image Retrieval (Validation)
Order-Embedding + LN
Order-Embedding (c) Recall@10
Figure 1: Recall@K curves using order-embeddings with and without layer normalization.
MSCOCO
Caption Retrieval Image Retrieval


key: chroma:document
string_value:
Figure 1: Recall@K curves using order-embeddings with and without layer normalization.
MSCOCO
Caption Retrieval Image Retrieval
Model R@1 R@5 R@10 Mean rR@1 R@5 R@10 Mean r
Sym [Vendrov et al., 2016] 45.4 88.7 5.8 36.3 85.8 9.0
OE [Vendrov et al., 2016] 46.7 88.9 5.7 37.9 85.9 8.1
OE (ours) 46.6 79.3 89.1 5.2 37.8 73.6 85.7 7.9
OE + LN 48.5 80.6 89.8 5.1 38.9 74.3 86.3 7.6
Table 2: Average results across 5 test splits for caption and image retrieval. R@K is Recall@K
(high is good). Mean ris the mean rank (low is good). Sym corresponds to the symmetric baseline
while OE indicates order-embeddings.
have an implicit “early stopping” effect on the weight vectors and help to stabilize learning towards
convergence.
Learning the magnitude of incoming weights: In normalized models, the magnitude of the incoming weights is explicitly parameterized by the gain parameters. We compare how the model output


key: chroma:document
string_value:
convergence.
Learning the magnitude of incoming weights: In normalized models, the magnitude of the incoming weights is explicitly parameterized by the gain parameters. We compare how the model output
changes between updating the gain parameters in the normalized GLM and updating the magnitude
of the equivalent weights under original parameterization during learning. The direction along the
gain parameters in Fcaptures the geometry for the magnitude of the incoming weights. We show
that Riemannian metric along the magnitude of the incoming weights for the standard GLM is scaled
by the norm of its input, whereas learning the gain parameters for the batch normalized and layer
normalized models depends only on the magnitude of the prediction error. Learning the magnitude
of incoming weights in the normalized model is therefore, more robust to the scaling of the input
and its parameters than in the standard model. See Appendix for detailed derivations.
6 Experimental results
We perform experiments with layer normalization on 6 tasks, with a focus on recurrent neural networks: image-sentence ranking, question-answering, contextual language modelling, generative
modelling, handwriting sequence generation and MNIST classiﬁcation. Unless otherwise noted,
the default initialization of layer normalization is to set the adaptive gains to 1and the biases to 0in
the experiments.
6.1 Order embeddings of images and language


key: chroma:document
string_value:
the default initialization of layer normalization is to set the adaptive gains to 1and the biases to 0in
the experiments.
6.1 Order embeddings of images and language
In this experiment, we apply layer normalization to the recently proposed order-embeddings model
of Vendrov et al. [2016] for learning a joint embedding space of images and sentences. We follow
the same experimental protocol as Vendrov et al. [2016] and modify their publicly available code
to incorporate layer normalization1which utilizes Theano [Team et al., 2016]. Images and sentences from the Microsoft COCO dataset [Lin et al., 2014] are embedded into a common vector
space, where a GRU [Cho et al., 2014] is used to encode sentences and the outputs of a pre-trained
VGG ConvNet [Simonyan and Zisserman, 2015] (10-crop) are used to encode images. The orderembedding model represents images and sentences as a 2-level partial ordering and replaces the
cosine similarity scoring function used in Kiros et al. [2014] with an asymmetric one.
1https://github.com/ivendrov/order-embedding
6
0 100 200 300 400 500 600 700 800
training steps (thousands)0.40.50.60.70.80.91.0validation error rateAttentive reader


key: chroma:document
string_value:
0 100 200 300 400 500 600 700 800
training steps (thousands)0.40.50.60.70.80.91.0validation error rateAttentive reader
LSTM
BN-LSTM
BN-everywhere
LN-LSTMFigure 2: Validation curves for the attentive reader model. BN results are taken from [Cooijmans
et al., 2016].
We trained two models: the baseline order-embedding model as well as the same model with layer
normalization applied to the GRU. After every 300 iterations, we compute Recall@K (R@K) values
on a held out validation set and save the model whenever R@K improves. The best performing
models are then evaluated on 5 separate test sets, each containing 1000 images and 5000 captions,
for which the mean results are reported. Both models use Adam [Kingma and Ba, 2014] with the
same initial hyperparameters and both models are trained using the same architectural choices as
used in Vendrov et al. [2016]. We refer the reader to the appendix for a description of how layer
normalization is applied to GRU.
Figure 1illustrates the validation curves of the models, with and without layer normalization. We
plot R@1, R@5 and R@10 for the image retrieval task. We observe that layer normalization offers


key: chroma:document
string_value:
plot R@1, R@5 and R@10 for the image retrieval task. We observe that layer normalization offers
a per-iteration speedup across all metrics and converges to its best validation model in 60% of the
time it takes the baseline model to do so. In Table 2, the test set results are reported from which we
observe that layer normalization also results in improved generalization over the original model. The
results we report are state-of-the-art for RNN embedding models, with only the structure-preserving
model of Wang et al. [2016] reporting better results on this task. However, they evaluate under
different conditions (1 test set instead of the mean over 5) and are thus not directly comparable.
6.2 Teaching machines to read and comprehend
In order to compare layer normalization to the recently proposed recurrent batch normalization
[Cooijmans et al., 2016], we train an unidirectional attentive reader model on the CNN corpus both
introduced by Hermann et al. [2015]. This is a question-answering task where a query description
about a passage must be answered by ﬁlling in a blank. The data is anonymized such that entities


key: chroma:document
string_value:
about a passage must be answered by ﬁlling in a blank. The data is anonymized such that entities
are given randomized tokens to prevent degenerate solutions, which are consistently permuted during training and evaluation. We follow the same experimental protocol as Cooijmans et al. [2016]
and modify their public code to incorporate layer normalization2which uses Theano [Team et al.,
2016]. We obtained the pre-processed dataset used by Cooijmans et al. [2016] which differs from
the original experiments of Hermann et al. [2015] in that each passage is limited to 4 sentences.
In Cooijmans et al. [2016], two variants of recurrent batch normalization are used: one where BN
is only applied to the LSTM while the other applies BN everywhere throughout the model. In our
experiment, we only apply layer normalization within the LSTM.
The results of this experiment are shown in Figure 2. We observe that layer normalization not only
trains faster but converges to a better validation result over both the baseline and BN variants. In
Cooijmans et al. [2016], it is argued that the scale parameter in BN must be carefully chosen and is
set to 0.1 in their experiments. We experimented with layer normalization for both 1.0 and 0.1 scale


key: chroma:document
string_value:
set to 0.1 in their experiments. We experimented with layer normalization for both 1.0 and 0.1 scale
initialization and found that the former model performed signiﬁcantly better. This demonstrates that
layer normalization is not sensitive to the initial scale in the same way that recurrent BN is.3
6.3 Skip-thought vectors
Skip-thoughts [Kiros et al., 2015] is a generalization of the skip-gram model [Mikolov et al., 2013]
for learning unsupervised distributed sentence representations. Given contiguous text, a sentence is
2https://github.com/cooijmanstim/Attentive_reader/tree/bn
3We only produce results on the validation set, as in the case of Cooijmans et al. [2016]
7
5 10 15 20
iteration x 5000082.082.583.083.584.084.585.085.586.0Pearson x 100
Skip-Thoughts + LN
Skip-Thoughts
Original(a) SICK(r)
5 10 15 20
iteration x 500002728293031323334MSE x 100
Skip-Thoughts + LN
Skip-Thoughts
Original (b) SICK(MSE)
5 10 15 20


key: chroma:document
string_value:
Skip-Thoughts + LN
Skip-Thoughts
Original (b) SICK(MSE)
5 10 15 20
iteration x 5000070727476788082Accuracy
Skip-Thoughts + LN
Skip-Thoughts
Original (c) MR
5 10 15 20
iteration x 5000074767880828486Accuracy
Skip-Thoughts + LN
Skip-Thoughts
Original
(d) CR
5 10 15 20
iteration x 5000090.090.591.091.592.092.593.093.594.094.5Accuracy
Skip-Thoughts + LN
Skip-Thoughts
Original (e) SUBJ
5 10 15 20
iteration x 50000838485868788899091Accuracy
Skip-Thoughts + LN
Skip-Thoughts
Original (f) MPQA
Figure 3: Performance of skip-thought vectors with and without layer normalization on downstream
tasks as a function of training iterations. The original lines are the reported results in [Kiros et al.,
2015]. Plots with error use 10-fold cross validation. Best seen in color.
Method SICK( r) SICK() SICK( MSE ) MR CR SUBJ MPQA


key: chroma:document
string_value:
Method SICK( r) SICK() SICK( MSE ) MR CR SUBJ MPQA
Original [Kiros et al., 2015] 0.848 0.778 0.287 75.5 79.3 92.1 86.9
Ours 0.842 0.767 0.298 77.3 81.8 92.6 87.9
Ours + LN 0.854 0.785 0.277 79.5 82.6 93.4 89.0
Ours + LN y 0.858 0.788 0.270 79.4 83.1 93.7 89.3
Table 3: Skip-thoughts results. The ﬁrst two evaluation columns indicate Pearson and Spearman correlation, the third is mean squared error and the remaining indicate classiﬁcation accuracy. Higher is
better for all evaluations except MSE. Our models were trained for 1M iterations with the exception
of (y) which was trained for 1 month (approximately 1.7M iterations)
encoded with a encoder RNN and decoder RNNs are used to predict the surrounding sentences.
Kiros et al. [2015] showed that this model could produce generic sentence representations that


key: chroma:document
string_value:
Kiros et al. [2015] showed that this model could produce generic sentence representations that
perform well on several tasks without being ﬁne-tuned. However, training this model is timeconsuming, requiring several days of training in order to produce meaningful results.
In this experiment we determine to what effect layer normalization can speed up training. Using the
publicly available code of Kiros et al. [2015]4, we train two models on the BookCorpus dataset [Zhu
et al., 2015]: one with and one without layer normalization. These experiments are performed with
Theano [Team et al., 2016]. We adhere to the experimental setup used in Kiros et al. [2015], training
a 2400-dimensional sentence encoder with the same hyperparameters. Given the size of the states
used, it is conceivable layer normalization would produce slower per-iteration updates than without.
However, we found that provided CNMeM5is used, there was no signiﬁcant difference between the
two models. We checkpoint both models after every 50,000 iterations and evaluate their performance
on ﬁve tasks: semantic-relatedness (SICK) [Marelli et al., 2014], movie review sentiment (MR)
[Pang and Lee, 2005], customer product reviews (CR) [Hu and Liu, 2004], subjectivity/objectivity


key: chroma:document
string_value:
[Pang and Lee, 2005], customer product reviews (CR) [Hu and Liu, 2004], subjectivity/objectivity
classiﬁcation (SUBJ) [Pang and Lee, 2004] and opinion polarity (MPQA) [Wiebe et al., 2005]. We
plot the performance of both models for each checkpoint on all tasks to determine whether the
performance rate can be improved with LN.
The experimental results are illustrated in Figure 3. We observe that applying layer normalization
results both in speedup over the baseline as well as better ﬁnal results after 1M iterations are performed as shown in Table 3. We also let the model with layer normalization train for a total of a
month, resulting in further performance gains across all but one task. We note that the performance
4https://github.com/ryankiros/skip-thoughts
5https://github.com/NVIDIA/cnmem
8
100101102103
Updates x 200−900−800−700−600−500−400−300−200−1000Negative Log LikelihoodBaseline test
Baseline train
LN test
LN trainFigure 5: Handwriting sequence generation model negative log likelihood with and without layer
normalization. The models are trained with mini-batch size of 8 and sequence length of 500,


key: chroma:document
string_value:
LN trainFigure 5: Handwriting sequence generation model negative log likelihood with and without layer
normalization. The models are trained with mini-batch size of 8 and sequence length of 500,
differences between the original reported results and ours are likely due to the fact that the publicly
available code does not condition at each timestep of the decoder, where the original model does.
6.4 Modeling binarized MNIST using DRAW
0 20 40 60 80 100
Epoch80859095100Test Variational BoundBaseline
WN
LN
Figure 4: DRAW model test negative log likelihood with and without
layer normalization.We also experimented with the generative modeling on the
MNIST dataset. Deep Recurrent Attention Writer (DRAW)
[Gregor et al., 2015] has previously achieved the state-of-theart performance on modeling the distribution of MNIST digits. The model uses a differential attention mechanism and
a recurrent neural network to sequentially generate pieces of
an image. We evaluate the effect of layer normalization on
a DRAW model using 64 glimpses and 256 LSTM hidden
units. The model is trained with the default setting of Adam
[Kingma and Ba, 2014] optimizer and the minibatch size of
128. Previous publications on binarized MNIST have used
various training protocols to generate their datasets. In this


key: chroma:document
string_value:
128. Previous publications on binarized MNIST have used
various training protocols to generate their datasets. In this
experiment, we used the ﬁxed binarization from Larochelle
and Murray [2011]. The dataset has been split into 50,000
training, 10,000 validation and 10,000 test images.
Figure 4 shows the test variational bound for the ﬁrst 100
epoch. It highlights the speedup beneﬁt of applying layer normalization that the layer normalized DRAW converges almost twice as fast than the baseline model.
After 200 epoches, the baseline model converges to a variational log likelihood of 82.36 nats on the
test data and the layer normalization model obtains 82.09 nats.
6.5 Handwriting sequence generation
The previous experiments mostly examine RNNs on NLP tasks whose lengths are in the range of 10
to 40. To show the effectiveness of layer normalization on longer sequences, we performed handwriting generation tasks using the IAM Online Handwriting Database [Liwicki and Bunke, 2005].
IAM-OnDB consists of handwritten lines collected from 221 different writers. When given the input
character string, the goal is to predict a sequence of x and y pen co-ordinates of the corresponding


key: chroma:document
string_value:
character string, the goal is to predict a sequence of x and y pen co-ordinates of the corresponding
handwriting line on the whiteboard. There are, in total, 12179 handwriting line sequences. The input
string is typically more than 25 characters and the average handwriting line has a length around 700.
We used the same model architecture as in Section (5.2) of Graves [2013]. The model architecture
consists of three hidden layers of 400 LSTM cells, which produce 20 bivariate Gaussian mixture
components at the output layer, and a size 3 input layer. The character sequence was encoded with
one-hot vectors, and hence the window vectors were size 57. A mixture of 10 Gaussian functions
was used for the window parameters, requiring a size 30 parameter vector. The total number of
weights was increased to approximately 3.7M. The model is trained using mini-batches of size 8
and the Adam [Kingma and Ba, 2014] optimizer.
The combination of small mini-batch size and very long sequences makes it important to have very
stable hidden dynamics. Figure 5 shows that layer normalization converges to a comparable log
likelihood as the baseline model but is much faster.
9
0 10 20 30 40 50 60
Epoch10-710-610-510-410-310-210-1100Train NLL
BatchNorm bz128


key: chroma:document
string_value:
9
0 10 20 30 40 50 60
Epoch10-710-610-510-410-310-210-1100Train NLL
BatchNorm bz128
Baseline bz128
LayerNorm bz128
0 10 20 30 40 50 60
Epoch0.0050.0100.0150.0200.025Test Err.
BatchNorm bz128
Baseline bz128
LayerNorm bz128
0 10 20 30 40 50 60
Epoch10-710-610-510-410-310-210-1100Train NLL
LayerNorm bz4
Baseline bz4
BatchNorm bz4
0 10 20 30 40 50 60
Epoch0.0050.0100.0150.0200.025Test Err.
LayerNorm bz4
Baseline bz4
BatchNorm bz4Figure 6: Permutation invariant MNIST 784-1000-1000-10 model negative log likelihood and test
error with layer normalization and batch normalization. (Left) The models are trained with batchsize of 128. (Right) The models are trained with batch-size of 4.
6.6 Permutation invariant MNIST
In addition to RNNs, we investigated layer normalization in feed-forward networks. We show how


key: chroma:document
string_value:
6.6 Permutation invariant MNIST
In addition to RNNs, we investigated layer normalization in feed-forward networks. We show how
layer normalization compares with batch normalization on the well-studied permutation invariant
MNIST classiﬁcation problem. From the previous analysis, layer normalization is invariant to input
re-scaling which is desirable for the internal hidden layers. But this is unnecessary for the logit
outputs where the prediction conﬁdence is determined by the scale of the logits. We only apply
layer normalization to the fully-connected hidden layers that excludes the last softmax layer.
All the models were trained using 55000 training data points and the Adam [Kingma and Ba, 2014]
optimizer. For the smaller batch-size, the variance term for batch normalization is computed using
the unbiased estimator. The experimental results from Figure 6 highlight that layer normalization is
robust to the batch-sizes and exhibits a faster training convergence comparing to batch normalization
that is applied to all layers.
6.7 Convolutional Networks
We have also experimented with convolutional neural networks. In our preliminary experiments, we
observed that layer normalization offers a speedup over the baseline model without normalization,
but batch normalization outperforms the other methods. With fully connected layers, all the hidden


key: chroma:document
string_value:
observed that layer normalization offers a speedup over the baseline model without normalization,
but batch normalization outperforms the other methods. With fully connected layers, all the hidden
units in a layer tend to make similar contributions to the ﬁnal prediction and re-centering and rescaling the summed inputs to a layer works well. However, the assumption of similar contributions
is no longer true for convolutional neural networks. The large number of the hidden units whose
receptive ﬁelds lie near the boundary of the image are rarely turned on and thus have very different
statistics from the rest of the hidden units within the same layer. We think further research is needed
to make layer normalization work well in ConvNets.
7 Conclusion
In this paper, we introduced layer normalization to speed-up the training of neural networks. We
provided a theoretical analysis that compared the invariance properties of layer normalization with
batch normalization and weight normalization. We showed that layer normalization is invariant to
per training-case feature shifting and scaling.
Empirically, we showed that recurrent neural networks beneﬁt the most from the proposed method
especially for long sequences and small mini-batches.
Acknowledgments
This research was funded by grants from NSERC, CFI, and Google.
10
References


key: chroma:document
string_value:
especially for long sequences and small mini-batches.
Acknowledgments
This research was funded by grants from NSERC, CFI, and Google.
10
References
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classiﬁcation with deep convolutional neural
networks. In NIPS , 2012.
Geoffrey Hinton, Li Deng, Dong Yu, George E Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, Andrew Senior,
Vincent Vanhoucke, Patrick Nguyen, Tara N Sainath, et al. Deep neural networks for acoustic modeling in
speech recognition: The shared views of four research groups. IEEE , 2012.
Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Mark Mao, Andrew Senior, Paul Tucker,
Ke Yang, Quoc V Le, et al. Large scale distributed deep networks. In NIPS , 2012.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing
internal covariate shift. ICML , 2015.
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In
Advances in neural information processing systems , pages 3104–3112, 2014.


key: chroma:document
string_value:
Advances in neural information processing systems , pages 3104–3112, 2014.
C´esar Laurent, Gabriel Pereyra, Phil ´emon Brakel, Ying Zhang, and Yoshua Bengio. Batch normalized recurrent
neural networks. arXiv preprint arXiv:1510.01378 , 2015.
Dario Amodei, Rishita Anubhai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Jingdong Chen,
Mike Chrzanowski, Adam Coates, Greg Diamos, et al. Deep speech 2: End-to-end speech recognition in
english and mandarin. arXiv preprint arXiv:1512.02595 , 2015.
Tim Cooijmans, Nicolas Ballas, C ´esar Laurent, and Aaron Courville. Recurrent batch normalization. arXiv
preprint arXiv:1603.09025 , 2016.
Tim Salimans and Diederik P Kingma. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. arXiv preprint arXiv:1602.07868 , 2016.
Behnam Neyshabur, Ruslan R Salakhutdinov, and Nati Srebro. Path-sgd: Path-normalized optimization in deep


key: chroma:document
string_value:
Behnam Neyshabur, Ruslan R Salakhutdinov, and Nati Srebro. Path-sgd: Path-normalized optimization in deep
neural networks. In Advances in Neural Information Processing Systems , pages 2413–2421, 2015.
Shun-Ichi Amari. Natural gradient works efﬁciently in learning. Neural computation , 1998.
Ivan Vendrov, Ryan Kiros, Sanja Fidler, and Raquel Urtasun. Order-embeddings of images and language.
ICLR , 2016.
The Theano Development Team, Rami Al-Rfou, Guillaume Alain, Amjad Almahairi, Christof Angermueller,
Dzmitry Bahdanau, Nicolas Ballas, Fr ´ed´eric Bastien, Justin Bayer, Anatoly Belikov, et al. Theano: A python
framework for fast computation of mathematical expressions. arXiv preprint arXiv:1605.02688 , 2016.
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll ´ar, and
C Lawrence Zitnick. Microsoft coco: Common objects in context. ECCV , 2014.


key: chroma:document
string_value:
C Lawrence Zitnick. Microsoft coco: Common objects in context. ECCV , 2014.
Kyunghyun Cho, Bart Van Merri ¨enboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger
Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical
machine translation. EMNLP , 2014.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition.
ICLR , 2015.
Ryan Kiros, Ruslan Salakhutdinov, and Richard S Zemel. Unifying visual-semantic embeddings with multimodal neural language models. arXiv preprint arXiv:1411.2539 , 2014.
D. Kingma and J. L. Ba. Adam: a method for stochastic optimization. ICLR , 2014. arXiv:1412.6980.
Liwei Wang, Yin Li, and Svetlana Lazebnik. Learning deep structure-preserving image-text embeddings.
CVPR , 2016.
Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman,


key: chroma:document
string_value:
CVPR , 2016.
Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman,
and Phil Blunsom. Teaching machines to read and comprehend. In NIPS , 2015.
Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja
Fidler. Skip-thought vectors. In NIPS , 2015.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efﬁcient estimation of word representations in
vector space. arXiv preprint arXiv:1301.3781 , 2013.
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja
Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading
books. In ICCV , 2015.
Marco Marelli, Luisa Bentivogli, Marco Baroni, Raffaella Bernardi, Stefano Menini, and Roberto Zamparelli.
Semeval-2014 task 1: Evaluation of compositional distributional semantic models on full sentences through


key: chroma:document
string_value:
Semeval-2014 task 1: Evaluation of compositional distributional semantic models on full sentences through
semantic relatedness and textual entailment. SemEval-2014 , 2014.
11
Bo Pang and Lillian Lee. Seeing stars: Exploiting class relationships for sentiment categorization with respect
to rating scales. In ACL, pages 115–124, 2005.
Minqing Hu and Bing Liu. Mining and summarizing customer reviews. In Proceedings of the tenth ACM
SIGKDD international conference on Knowledge discovery and data mining , 2004.
Bo Pang and Lillian Lee. A sentimental education: Sentiment analysis using subjectivity summarization based
on minimum cuts. In ACL, 2004.
Janyce Wiebe, Theresa Wilson, and Claire Cardie. Annotating expressions of opinions and emotions in language. Language resources and evaluation , 2005.
K. Gregor, I. Danihelka, A. Graves, and D. Wierstra. DRAW: a recurrent neural network for image generation.
arXiv:1502.04623, 2015.
Hugo Larochelle and Iain Murray. The neural autoregressive distribution estimator. In AISTATS , volume 6,
page 622, 2011.
Marcus Liwicki and Horst Bunke. Iam-ondb-an on-line english sentence database acquired from handwritten


key: chroma:document
string_value:
page 622, 2011.
Marcus Liwicki and Horst Bunke. Iam-ondb-an on-line english sentence database acquired from handwritten
text on a whiteboard. In ICDAR , 2005.
Alex Graves. Generating sequences with recurrent neural networks. arXiv preprint arXiv:1308.0850 , 2013.
12
Supplementary Material
Application of layer normalization to each experiment
This section describes how layer normalization is applied to each of the papers’ experiments. For
notation convenience, we deﬁne layer normalization as a function mapping LN:RD!RDwith
two set of adaptive parameters, gains and biases :
LN(z;;) =(z)
+; (15)
=1
DDX
i=1zi;  =vuut1
DDX
i=1(zi)2; (16)
where,ziis theithelement of the vector z.
Teaching machines to read and comprehend and handwriting sequence generation
The basic LSTM equations used for these experiment are given by:
0
B@ft
it
ot
gt1
CA=Whht1+Wxxt+b (17)


key: chroma:document
string_value:
0
B@ft
it
ot
gt1
CA=Whht1+Wxxt+b (17)
ct=(ft)ct1+(it)tanh(gt) (18)
ht=(ot)tanh(ct) (19)
The version that incorporates layer normalization is modiﬁed as follows:
0
B@ft
it
ot
gt1
CA=LN(Whht1;1;1) +LN(Wxxt;2;2) +b (20)
ct=(ft)ct1+(it)tanh(gt) (21)
ht=(ot)tanh(LN(ct;3;3)) (22)
where i;iare the additive and multiplicative parameters, respectively. Each iis initialized to a
vector of zeros and each iis initialized to a vector of ones.
Order embeddings and skip-thoughts
These experiments utilize a variant of gated recurrent unit which is deﬁned as follows:

zt
rt
=Whht1+Wxxt (23)


key: chroma:document
string_value:

zt
rt
=Whht1+Wxxt (23)
^ht=tanh(Wxt+(rt)(Uht1)) (24)
ht= (1(zt))ht1+(zt)^ht (25)
Layer normalization is applied as follows:

zt
rt
=LN(Whht1;1;1) +LN(Wxxt;2;2) (26)
^ht=tanh(LN(Wxt;3;3) +(rt)LN(Uht1;4;4)) (27)
ht= (1(zt))ht1+(zt)^ht (28)
just as before, iis initialized to a vector of zeros and each iis initialized to a vector of ones.
13
Modeling binarized MNIST using DRAW
The layer norm is only applied to the output of the LSTM hidden states in this experiment:
The version that incorporates layer normalization is modiﬁed as follows:0
B@ft
it
ot
gt1
CA=Whht1+Wxxt+b (29)


key: chroma:document
string_value:
B@ft
it
ot
gt1
CA=Whht1+Wxxt+b (29)
ct=(ft)ct1+(it)tanh(gt) (30)
ht=(ot)tanh(LN(ct;;)) (31)
where ;are the additive and multiplicative parameters, respectively. is initialized to a vector
of zeros and is initialized to a vector of ones.
Learning the magnitude of incoming weights
We now compare how gradient descent updates changing magnitude of the equivalent weights between the normalized GLM and original parameterization. The magnitude of the weights are explicitly parameterized using the gain parameter in the normalized model. Assume there is a gradient
update that changes norm of the weight vectors by g. We can project the gradient updates to the
weight vector for the normal GLM. The KL metric, ie how much the gradient update changes the
model prediction, for the normalized model depends only on the magnitude of the prediction error.
Speciﬁcally,
under batch normalization:
ds2=1
2vec([0;0;g]>)>F(vec([W;b;g]>) vec([0;0;g]>) =1
2>
gE


key: chroma:document
string_value:
2>
gE
xP(x)Cov[yjx]
2
g:
(32)
Under layer normalization:
ds2=1
2vec([0;0;g]>)>F(vec([W;b;g]>) vec([0;0;g]>)
=1
2>
g1
2E
xP(x)2
664Cov(y1;y1jx)(a1)2
2 Cov(y1;yHjx)(a1)(aH)
2
.........
Cov(yH;y1jx)(aH)(a1)
2 Cov(yH;yHjx)(aH)2
23
775g
(33)
Under weight normalization:
ds2=1
2vec([0;0;g]>)>F(vec([W;b;g]>) vec([0;0;g]>)
=1
2>
g1
2E
xP(x)2
6664Cov(y1;y1jx)a2
1
kw1k2


key: chroma:document
string_value:
g1
2E
xP(x)2
6664Cov(y1;y1jx)a2
1
kw1k2
2 Cov(y1;yHjx)a1aH
kw1k2kwHk2
.........
Cov(yH;y1jx)aHa1
kwHk2kw1k2 Cov(yH;yHjx)a2
H
kwHk2
23
7775g:(34)
Whereas, the KL metric in the standard GLM is related to its activities ai=w>
ix, that is depended
on both its current weights and input data. We project the gradient updates to the gain parameter gi
of theithneuron to its weight vector as giwi
kwik2in the standard GLM model:
1
2vec([giwi
kwik2;0;gjwj
kwik2;0]>)>F([w>
i;bi;w>
j;bj]>) vec([giwi
kwik2;0;gjwj
kwjk2;0]>)
=gigj
22E
xP(x)


key: chroma:document
string_value:
kwik2;0;gjwj
kwjk2;0]>)
=gigj
22E
xP(x)
Cov(yi;yjjx)aiaj
kwik2kwjk2
(35)
The batch normalized and layer normalized models are therefore more robust to the scaling of the
input and its parameters than the standard model.
14


key: chroma:document
string_value:
Published as a conference paper at ICLR 2017
HIERARCHICAL MULTISCALE
RECURRENT NEURAL NETWORKS
Junyoung Chung, Sungjin Ahn & Yoshua Bengio
Département d’informatique et de recherche opérationnelle
Université de Montréal
{junyoung.chung,sungjin.ahn,yoshua.bengio}@umontreal.ca
ABSTRACT
Learning both hierarchical and temporal representation has been among the longstanding challenges of recurrent neural networks. Multiscale recurrent neural
networks have been considered as a promising approach to resolve this issue, yet
there has been a lack of empirical evidence showing that this type of models can
actually capture the temporal dependencies by discovering the latent hierarchical
structure of the sequence. In this paper, we propose a novel multiscale approach,
called the hierarchical multiscale recurrent neural network, that can capture the
latent hierarchical structure in the sequence by encoding the temporal dependencies
with different timescales using a novel update mechanism. We show some evidence
that the proposed model can discover underlying hierarchical structure in the
sequences without using explicit boundary information. We evaluate our proposed
model on character-level language modelling and handwriting sequence generation.
1 I NTRODUCTION
One of the key principles of learning in deep neural networks as well as in the human brain is to obtain


key: chroma:document
string_value:
1 I NTRODUCTION
One of the key principles of learning in deep neural networks as well as in the human brain is to obtain
a hierarchical representation with increasing levels of abstraction (Bengio, 2009; LeCun et al., 2015;
Schmidhuber, 2015). A stack of representation layers, learned from the data in a way to optimize
the target task, make deep neural networks entertain advantages such as generalization to unseen
examples (Hoffman et al., 2013), sharing learned knowledge among multiple tasks, and discovering
disentangling factors of variation (Kingma & Welling, 2013). The remarkable recent successes of
the deep convolutional neural networks are particularly based on this ability to learn hierarchical
representation for spatial data (Krizhevsky et al., 2012). For modelling temporal data, the recent
resurgence of recurrent neural networks (RNN) has led to remarkable advances (Mikolov et al., 2010;
Graves, 2013; Cho et al., 2014; Sutskever et al., 2014; Vinyals et al., 2015). However, unlike the
spatial data, learning both hierarchical and temporal representation has been among the long-standing
challenges of RNNs in spite of the fact that hierarchical multiscale structures naturally exist in many


key: chroma:document
string_value:
spatial data, learning both hierarchical and temporal representation has been among the long-standing
challenges of RNNs in spite of the fact that hierarchical multiscale structures naturally exist in many
temporal data (Schmidhuber, 1991; Mozer, 1993; El Hihi & Bengio, 1995; Lin et al., 1996; Koutník
et al., 2014).
A promising approach to model such hierarchical and temporal representation is the multiscale
RNNs (Schmidhuber, 1992; El Hihi & Bengio, 1995; Koutník et al., 2014). Based on the observation
that high-level abstraction changes slowly with temporal coherency while low-level abstraction
has quickly changing features sensitive to the precise local timing (El Hihi & Bengio, 1995), the
multiscale RNNs group hidden units into multiple modules of different timescales. In addition to
the fact that the architecture ﬁts naturally to the latent hierarchical structures in many temporal data,
the multiscale approach provides the following advantages that resolve some inherent problems
of standard RNNs: (a) computational efﬁciency obtained by updating the high-level layers less
frequently, (b) efﬁciently delivering long-term dependencies with fewer updates at the high-level


key: chroma:document
string_value:
frequently, (b) efﬁciently delivering long-term dependencies with fewer updates at the high-level
layers, which mitigates the vanishing gradient problem, (c) ﬂexible resource allocation (e.g., more
hidden units to the higher layers that focus on modelling long-term dependencies and less hidden
units to the lower layers which are in charge of learning short-term dependencies). In addition, the
learned latent hierarchical structures can provide useful information to other downstream tasks such
Yoshua Bengio is CIFAR Senior Fellow.
1arXiv:1609.01704v7  [cs.LG]  9 Mar 2017
Published as a conference paper at ICLR 2017
as module structures in computer program learning, sub-task structures in hierarchical reinforcement
learning, and story segments in video understanding.
There have been various approaches to implementing the multiscale RNNs. The most popular
approach is to set the timescales as hyperparameters (El Hihi & Bengio, 1995; Koutník et al., 2014;
Bahdanau et al., 2016) instead of treating them as dynamic variables that can be learned from the
data (Schmidhuber, 1991; 1992; Chung et al., 2015; 2016). However, considering the fact that


key: chroma:document
string_value:
data (Schmidhuber, 1991; 1992; Chung et al., 2015; 2016). However, considering the fact that
non-stationarity is prevalent in temporal data, and that many entities of abstraction such as words
and sentences are in variable length, we claim that it is important for an RNN to dynamically adapt
its timescales to the particulars of the input entities of various length. While this is trivial if the
hierarchical boundary structure is provided (Sordoni et al., 2015), it has been a challenge for an RNN
to discover the latent hierarchical structure in temporal data without explicit boundary information.
In this paper, we propose a novel multiscale RNN model, which can learn the hierarchical multiscale
structure from temporal data without explicit boundary information. This model, called a hierarchical
multiscale recurrent neural network (HM-RNN), does not assign ﬁxed update rates, but adaptively
determines proper update times corresponding to different abstraction levels of the layers. We ﬁnd
that this model tends to learn ﬁne timescales for low-level layers and coarse timescales for high-level
layers. To do this, we introduce a binary boundary detector at each layer. The boundary detector is
turned on only at the time steps where a segment of the corresponding abstraction level is completely


key: chroma:document
string_value:
layers. To do this, we introduce a binary boundary detector at each layer. The boundary detector is
turned on only at the time steps where a segment of the corresponding abstraction level is completely
processed. Otherwise, i.e., during the within segment processing, it stays turned off. Using the
hierarchical boundary states, we implement three operations, UPDATE, COPY and FLUSH, and
choose one of them at each time step. The UPDATE operation is similar to the usual update rule of
the long short-term memory (LSTM) (Hochreiter & Schmidhuber, 1997), except that it is executed
sparsely according to the detected boundaries. The COPY operation simply copies the cell and hidden
states of the previous time step. Unlike the leaky integration of the LSTM or the Gated Recurrent
Unit (GRU) (Cho et al., 2014), the COPY operation retains the whole states without any loss of
information. The FLUSH operation is executed when a boundary is detected, where it ﬁrst ejects the
summarized representation of the current segment to the upper layer and then reinitializes the states
to start processing the next segment. Learning to select a proper operation at each time step and to
detect the boundaries, the HM-RNN discovers the latent hierarchical structure of the sequences. We


key: chroma:document
string_value:
to start processing the next segment. Learning to select a proper operation at each time step and to
detect the boundaries, the HM-RNN discovers the latent hierarchical structure of the sequences. We
ﬁnd that the straight-through estimator (Hinton, 2012; Bengio et al., 2013; Courbariaux et al., 2016)
is efﬁcient for training this model containing discrete variables.
We evaluate our model on two tasks: character-level language modelling and handwriting sequence
generation. For the character-level language modelling, the HM-RNN achieves the state-of-the-art
results on the Text8 dataset, and comparable results to the state-of-the-art on the Penn Treebank
and Hutter Prize Wikipedia datasets. The HM-RNN also outperforms the standard RNN on the
handwriting sequence generation using the IAM-OnDB dataset. In addition, we demonstrate that the
hierarchical structure found by the HM-RNN is indeed very similar to the intrinsic structure observed
in the data. The contributions of this paper are:
We propose for the ﬁrst time an RNN model that can learn a latent hierarchical structure of
a sequence without using explicit boundary information.
We show that it is beneﬁcial to utilize the above structure through empirical evaluation.


key: chroma:document
string_value:
a sequence without using explicit boundary information.
We show that it is beneﬁcial to utilize the above structure through empirical evaluation.
We show that the straight-through estimator is an efﬁcient way of training a model containing
discrete variables.
We propose the slope annealing trick to improve the training procedure based on the
straight-through estimator.
2 R ELATED WORK
Two notable early attempts inspiring our model are Schmidhuber (1992) and El Hihi & Bengio (1995).
In these works, it is advocated to stack multiple layers of RNNs in a decreasing order of update
frequency for computational and learning efﬁciency. In Schmidhuber (1992), the author shows a
model that can self-organize a hierarchical multiscale structure. Particularly in El Hihi & Bengio
(1995), the advantages of incorporating a priori knowledge, “ temporal dependencies are structured
hierarchically ", into the RNN architecture is studied. The authors propose an RNN architecture that
updates each layer with a ﬁxed but different rate, called a hierarchical RNN.
2
Published as a conference paper at ICLR 2017
LSTMs (Hochreiter & Schmidhuber, 1997) employ the multiscale update concept, where the hidden


key: chroma:document
string_value:
2
Published as a conference paper at ICLR 2017
LSTMs (Hochreiter & Schmidhuber, 1997) employ the multiscale update concept, where the hidden
units have different forget and update rates and thus can operate with different timescales. However,
unlike our model, these timescales are not organized hierarchically. Although the LSTM has a selfloop for the gradients that helps to capture the long-term dependencies by mitigating the vanishing
gradient problem, in practice, it is still limited to a few hundred time steps due to the leaky integration
by which the contents to memorize for a long-term is gradually diluted at every time step. Also, the
model remains computationally expensive because it has to perform the update at every time step
for each unit. However, our model is less prone to these problems because it learns a hierarchical
structure such that, by design, high-level layers learn to perform less frequent updates than low-level
layers. We hypothesize that this property mitigates the vanishing gradient problem more efﬁciently
while also being computationally more efﬁcient.
A more recent model, the clockwork RNN (CW-RNN) (Koutník et al., 2014) extends the hierarchical


key: chroma:document
string_value:
A more recent model, the clockwork RNN (CW-RNN) (Koutník et al., 2014) extends the hierarchical
RNN (El Hihi & Bengio, 1995) and the NARX RNN (Lin et al., 1996)1. The CW-RNN tries to
solve the issue of using soft timescales in the LSTM, by explicitly assigning hard timescales. In the
CW-RNN, hidden units are partitioned into several modules, and different timescales are assigned to
the modules such that a module iupdates its hidden units at every 2(i1)-th time step. The CW-RNN
is computationally more efﬁcient than the standard RNN including the LSTM since hidden units
are updated only at the assigned clock rates. However, ﬁnding proper timescales in the CW-RNN
remains as a challenge whereas our model learns the intrinsic timescales from the data. In the biscale
RNNs (Chung et al., 2016), the authors proposed to model layer-wise timescales adaptively by having
additional gating units, however this approach still relies on the softgating mechanism like LSTMs.
Other forms of Hierarchical RNN (HRNN) architectures have been proposed in the cases where


key: chroma:document
string_value:
Other forms of Hierarchical RNN (HRNN) architectures have been proposed in the cases where
the explicit hierarchical boundary structure is provided. In Ling et al. (2015), after obtaining the
word boundary via tokenization, the HRNN architecture is used for neural machine translation by
modelling the characters and words using the ﬁrst and second RNN layers, respectively. A similar
HRNN architecture is also adopted in Sordoni et al. (2015) to model dialogue utterances. However,
in many cases, hierarchical boundary information is not explicitly observed or expensive to obtain.
Also, it is unclear how to deploy more layers than the number of boundary levels that is explicitly
observed in the data.
While the above models focus on online prediction problems, where a prediction needs to be made by
using only the past data, in some cases, predictions are made after observing the whole sequence. In
this setting, the input sequence can be regarded as 1-D spatial data, convolutional neural networks
with 1-D kernels are proposed in Kim (2014) and Kim et al. (2015) for language modelling and
sentence classiﬁcation. Also, in Chan et al. (2016) and Bahdanau et al. (2016), the authors proposed
to obtain high-level representation of the sequences of reduced length by repeatedly merging or
pooling the lower-level representation of the sequences.


key: chroma:document
string_value:
to obtain high-level representation of the sequences of reduced length by repeatedly merging or
pooling the lower-level representation of the sequences.
Hierarchical RNN architectures have also been used to discover the segmentation structure in
sequences (Fernández et al., 2007; Kong et al., 2015). It is however different to our model in the
sense that they optimize the objective with explicit labels on the hierarchical segments while our
model discovers the intrinsic structure only from the sequences without segment label information.
The COPY operation used in our model can be related to Zoneout (Krueger et al., 2016) which
is a recurrent generalization of stochastic depth (Huang et al., 2016). In Zoneout, an identity
transformation is randomly applied to each hidden unit at each time step according to a Bernoulli
distribution. This results in occasional copy operations of the previous hidden states. While the focus
of Zoneout is to propose a regularization technique similar to dropout (Srivastava et al., 2014) (where
the regularization strength is controlled by a hyperparameter), our model learns (a) to dynamically
determine when to copy from the context inputs and (b) to discover the hierarchical multiscale
structure and representation. Although the main goal of our proposed model is not regularization, we
found that our model also shows very good generalization performance.


key: chroma:document
string_value:
structure and representation. Although the main goal of our proposed model is not regularization, we
found that our model also shows very good generalization performance.
3
Published as a conference paper at ICLR 2017
h20h22h25h15h14h13h12h11h10x1x2x3x4x5word1word2phrase1
h20h21h22h23h24h25h15h14h13h12h11h10x1x2x3x4x5word1word2phrase1
(a) (b)
Figure 1: (a) The HRNN architecture, which requires the knowledge of the hierarchical boundaries.
(b) The HM-RNN architecture that discovers the hierarchical multiscale structure in the data.
3 H IERARCHICAL MULTISCALE RECURRENT NEURAL NETWORKS
3.1 M OTIVATION
To begin with, we provide an example of how a stacked RNN can model temporal data in an ideal
setting, i.e., when the hierarchy of segments is provided (Sordoni et al., 2015; Ling et al., 2015). In
Figure 1 (a), we depict a hierarchical RNN (HRNN) for language modelling with two layers: the ﬁrst
layer receives characters as inputs and generates word-level representations (C2W-RNN), and the


key: chroma:document
string_value:
layer receives characters as inputs and generates word-level representations (C2W-RNN), and the
second layer takes the word-level representations as inputs and yields phrase-level representations
(W2P-RNN).
As shown, by means of the provided end-of-word labels, the C2W-RNN obtains word-level representation after processing the last character of each word and passes the word-level representation to the
W2P-RNN. Then, the W2P-RNN performs an update of the phrase-level representation. Note that the
hidden states of the W2P-RNN remains unchanged while all the characters of a word are processed by
the C2W-RNN. When the C2W-RNN starts to process the next word, its hidden states are reinitialized
using the latest hidden states of the W2P-RNN, which contain summarized representation of all the
words that have been processed by that time step, in that phrase.
From this simple example, we can see the advantages of having a hierarchical multiscale structure: (1)
as the W2P-RNN is updated at a much slower update rate than the C2W-RNN, a considerable amount
of computation can be saved, (2) gradients are backpropagated through a much smaller number of


key: chroma:document
string_value:
of computation can be saved, (2) gradients are backpropagated through a much smaller number of
time steps, and (3) layer-wise capacity control becomes possible (e.g., use a smaller number of hidden
units in the ﬁrst layer which models short-term dependencies but whose updates are invoked much
more often).
Can an RNN discover such hierarchical multiscale structure without explicit hierarchical boundary
information? Considering the fact that the boundary information is difﬁcult to obtain (for example,
consider languages where words are not always cleanly separated by spaces or punctuation symbols,
and imperfect rules are used to separately perform segmentation) or usually not provided at all, this is
a legitimate problem. It gets worse when we consider higher-level concepts which we would like
the RNN to discover autonomously. In Section 2, we discussed the limitations of the existing RNN
models under this setting, which either have to update all units at every time step or use ﬁxed update
frequencies (El Hihi & Bengio, 1995; Koutník et al., 2014). Unfortunately, this kind of approach is
not well suited to the case where different segments in the hierarchical decomposition have different
lengths: for example, different words have different lengths, so a ﬁxed hierarchy would not update its


key: chroma:document
string_value:
lengths: for example, different words have different lengths, so a ﬁxed hierarchy would not update its
upper-level units in synchrony with the natural boundaries in the data.
3.2 T HEPROPOSED MODEL
A key element of our model is the introduction of a parametrized boundary detector, which outputs
a binary value, in each layer of a stacked RNN, and learns when a segment should end in such
a way to optimize the overall target objective. Whenever the boundary detector is turned on at a
time step of layer `(i.e., when the boundary state is 1), the model considers this to be the end of a
1The acronym NARX stands for Non-linear Auto-Regressive model with eXogenous inputs.
4
Published as a conference paper at ICLR 2017
segment corresponding to the latent abstraction level of that layer (e.g., word or phrase) and feeds the
summarized representation of the detected segment into the upper layer ( `+ 1). Using the boundary
states, at each time step, each layer selects one of the following operations: UPDATE, COPY or
FLUSH. The selection is determined by (1) the boundary state of the current time step in the layer
belowz`1
tand (2) the boundary state of the previous time step in the same layer z`
t1.


key: chroma:document
string_value:
belowz`1
tand (2) the boundary state of the previous time step in the same layer z`
t1.
In the following, we describe an HM-RNN based on the LSTM update rule. We call this model
a hierarchical multiscale LSTM (HM-LSTM). Consider an HM-LSTM model of Llayers (`=
1;:::;L ) which, at each layer `, performs the following update at time step t:
h`
t;c`
t;z`
t=f`
HM-LSTM (c`
t1;h`
t1;h`1
t;h`+1
t1;z`
t1;z`1
t): (1)
Here, handcdenote the hidden and cell states, respectively. The function f`
HM-LSTM is implemented
as follows. First, using the two boundary states z`
t1andz`1
t, the cell state is updated by:
c`
t=8
<
:f`
tc`
t1+i`
tg`
tifz`
t1= 0andz`1
t= 1(UPDATE)
c`


key: chroma:document
string_value:
t1+i`
tg`
tifz`
t1= 0andz`1
t= 1(UPDATE)
c`
t1 ifz`
t1= 0andz`1
t= 0(COPY)
i`
tg`
t ifz`
t1= 1(FLUSH);(2)
and then the hidden state is obtained by:
h`
t=h`
t1 if COPY ,
o`
ttanh (c`
t)otherwise.(3)
Here, (f;i;o)are forget, input, output gates, and gis a cell proposal vector. Note that unlike the
LSTM, it is not necessary to compute these gates and cell proposal values at every time step. For
example, in the case of the COPY operation, we do not need to compute any of these values and thus
can save computations.
The COPY operation, which simply performs (c`
t;h`
t) (c`
t1;h`
t1), implements the observation
that an upper layer should keep its state unchanged until it receives the summarized input from
the lower layer. The UPDATE operation is performed to update the summary representation of the
layer`if the boundary z`1


key: chroma:document
string_value:
the lower layer. The UPDATE operation is performed to update the summary representation of the
layer`if the boundary z`1
tis detected from the layer below but the boundary z`
t1was not found
at the previous time step. Hence, the UPDATE operation is executed sparsely unlike the standard
RNNs where it is executed at every time step, making it computationally inefﬁcient. If a boundary is
detected, the FLUSH operation is executed. The FLUSH operation consists of two sub-operations:
(a) EJECT to pass the current state to the upper layer and then (b) RESET to reinitialize the state
before starting to read a new segment. This operation implicitly forces the upper layer to absorb the
summary information of the lower layer segment, because otherwise it will be lost. Note that the
FLUSH operation is a hard reset in the sense that it completely erases all the previous states of the
same layer, which is different from the softreset or softforget operation in the GRU or LSTM.
Whenever needed (depending on the chosen operation), the gate values ( f`
t;i`
t;o`
t), the cell proposal
g`
t, and the pre-activation of the boundary detector ~z`
t2are then obtained by:
0
BBB@f`
t
i`


key: chroma:document
string_value:
g`
t, and the pre-activation of the boundary detector ~z`
t2are then obtained by:
0
BBB@f`
t
i`
t
o`
t
g`
t
~z`
t1
CCCA=0
BBB@sigm
sigm
sigm
tanh
hard sigm1
CCCAfslice
srecurrent (`)
t +stop-down (`)
t +sbottom-up (`)
t +b(`)
; (4)
where
srecurrent (`)
t =U`
`h`
t1; (5)
stop-down (`)
t =z`
t1U`
`+1h`+1
t1; (6)
sbottom-up (`)
t =z`1
tW`
`1h`1
t: (7)
Here, we use Wj
i2R(4dim (h`)+1)dim (h`1);Uj
i2R(4dim (h`)+1)dim (h`)to denote state transition


key: chroma:document
string_value:
i2R(4dim (h`)+1)dim (h`)to denote state transition
parameters from layer ito layerj, and b2R4dim (h`)+1is a bias term. In the last layer L, the
2~z`
tcan also be implemented as a function of h`
t, e.g., ~z`
t=hard sigm (Uh`
t).
5
Published as a conference paper at ICLR 2017
1z`t1z`t1W``1z`1tz`t1h`+1t1h`t1h`+1th`th`1tU``+1U``W`+1`z`t
ˆxt+1g1tg2tg3th1th2th3thet
Figure 2: Left: The gating mechanism of the HM-RNN. Right: The output module when L= 3.
top-down connection is ignored, and we use h0
t=xt. Since the input should not be omitted, we
setz0
t= 1for allt. Also, we do not use the boundary detector for the last layer. The hard sigm is
deﬁned by hard sigm (x) = max
0;min


key: chroma:document
string_value:
deﬁned by hard sigm (x) = max
0;min
1;ax+1
2withabeing the slope variable.
Unlike the standard LSTM, the HM-LSTM has a top-down connection from (`+ 1) to`, which is
allowed to be activated only if a boundary is detected at the previous time step of the layer `(see
Eq. 6). This makes the layer `to be initialized with more long-term information after the boundary
is detected and execute the FLUSH operation. In addition, the input from the lower layer ( `1)
becomes effective only when a boundary is detected at the current time step in the layer ( `1) due
to the binary gate z`1
t. Figure 2 (left) shows the gating mechanism of the HM-LSTM at time step t.
Finally, the binary boundary state z`
tis obtained by:
z`
t=fbound(~z`
t): (8)
For the binarization function fbound :R!f0;1g, we can either use a deterministic step function:
z`
t=1if~z`
t>0:5
0otherwise;(9)
or sample from a Bernoulli distribution z`
tBernoulli (~z`


key: chroma:document
string_value:
t>0:5
0otherwise;(9)
or sample from a Bernoulli distribution z`
tBernoulli (~z`
t). Although this binary decision is a key to
our model, it is usually difﬁcult to use stochastic gradient descent to train such model with discrete
decisions as it is not differentiable.
3.3 C OMPUTING GRADIENT OF BOUNDARY DETECTOR
Training neural networks with discrete variables requires more efforts since the standard backpropagation is no longer applicable due to the non-differentiability. Among a few methods for training a
neural network with discrete variables such as the REINFORCE (Williams, 1992; Mnih & Gregor,
2014) and the straight-through estimator (Hinton, 2012; Bengio et al., 2013), we use the straightthrough estimator to train our model. The straight-through estimator is a biased estimator because the
non-differentiable function used in the forward pass (i.e., the step function in our case) is replaced by
a differentiable function during the backward pass (i.e., the hard sigmoid function in our case). The
straight-through estimator, however, is much simpler and often works more efﬁciently in practice


key: chroma:document
string_value:
straight-through estimator, however, is much simpler and often works more efﬁciently in practice
than other unbiased but high-variance estimators such as the REINFORCE. The straight-through
estimator has also been used in Courbariaux et al. (2016) and Vezhnevets et al. (2016).
The Slope Annealing Trick . In our experiment, we use the slope annealing trick to reduce the bias
of the straight-through estimator. The idea is to reduce the discrepancy between the two functions
used during the forward pass and the backward pass. That is, by gradually increasing the slope aof
the hard sigmoid function, we make the hard sigmoid be close to the step function. Note that starting
with a high slope value from the beginning can make the training difﬁcult while it is more applicable
later when the model parameters become more stable. In our experiments, starting from slope a= 1,
we slowly increase the slope until it reaches a threshold with an appropriate scheduling.
4 E XPERIMENTS
We evaluate the proposed model on two tasks, character-level language modelling and handwriting
sequence generation. Character-level language modelling is a representative example of discrete
6
Published as a conference paper at ICLR 2017
Penn Treebank
Model BPC


key: chroma:document
string_value:
sequence generation. Character-level language modelling is a representative example of discrete
6
Published as a conference paper at ICLR 2017
Penn Treebank
Model BPC
Norm-stabilized RNN (Krueger & Memisevic, 2015) 1.48
CW-RNN (Koutník et al., 2014) 1.46
HF-MRNN (Mikolov et al., 2012) 1.41
MI-RNN (Wu et al., 2016) 1.39
MEn-gram (Mikolov et al., 2012) 1.37
BatchNorm LSTM (Cooijmans et al., 2016) 1.32
Zoneout RNN (Krueger et al., 2016) 1.27
HyperNetworks (Ha et al., 2016) 1.27
LayerNorm HyperNetworks (Ha et al., 2016) 1.23
LayerNorm CW-RNNy1.40
LayerNorm LSTMy1.29
LayerNorm HM-LSTM Sampling 1.27
LayerNorm HM-LSTM Soft1.27
LayerNorm HM-LSTM Step Fn. 1.25
LayerNorm HM-LSTM Step Fn. & Slope Annealing 1.24Hutter Prize Wikipedia
Model BPC
Stacked LSTM (Graves, 2013) 1.67


key: chroma:document
string_value:
Model BPC
Stacked LSTM (Graves, 2013) 1.67
MRNN (Sutskever et al., 2011) 1.60
GF-LSTM (Chung et al., 2015) 1.58
Grid-LSTM (Kalchbrenner et al., 2015) 1.47
MI-LSTM (Wu et al., 2016) 1.44
Recurrent Memory Array Structures (Rocki, 2016a) 1.40
SF-LSTM (Rocki, 2016b)z1.37
HyperNetworks (Ha et al., 2016) 1.35
LayerNorm HyperNetworks (Ha et al., 2016) 1.34
Recurrent Highway Networks (Zilly et al., 2016) 1.32
LayerNorm LSTMy1.39
HM-LSTM 1.34
LayerNorm HM-LSTM 1.32
PAQ8hp12 (Mahoney, 2005) 1.32
decomp8 (Mahoney, 2009) 1.28
Table 1: BPC on the Penn Treebank test set (left) and Hutter Prize Wikipedia test set (right). ( ) This
model is a variant of the HM-LSTM that does not discretize the boundary detector states. ( y) These
models are implemented by the authors to evaluate the performance using layer normalization (Ba


key: chroma:document
string_value:
models are implemented by the authors to evaluate the performance using layer normalization (Ba
et al., 2016) with the additional output module. ( z) This method uses test error signals for predicting
the next characters, which makes it not comparable to other methods that do not.
sequence modelling, where the discrete symbols form a distinct hierarchical multiscale structure. The
performance on real-valued sequences is tested on the handwriting sequence generation in which a
relatively clear hierarchical multiscale structure exists compared to other data such as speech signals.
4.1 C HARACTER -LEVEL LANGUAGE MODELLING
A sequence modelling task aims at learning the probability distribution over sequences by minimizing
the negative log-likelihood of the training sequences:
min
1
NNX
n=1TnX
t=1logp(xn
tjxn
<t;); (10)
whereis the model parameter, Nis the number of training sequences, and Tnis the length of the
n-th sequence. A symbol at time tof sequence nis denoted by xn
t, andxn
<tdenotes all previous
symbols at time t. We evaluate our model on three benchmark text corpora: (1) Penn Treebank, (2)


key: chroma:document
string_value:
<tdenotes all previous
symbols at time t. We evaluate our model on three benchmark text corpora: (1) Penn Treebank, (2)
Text8 and (3) Hutter Prize Wikipedia. We use the bits-per-character (BPC), E[log2p(xt+1jxt)],
as the evaluation metric.
Model We use a model consisting of an input embedding layer, an RNN module and an output
module. The input embedding layer maps each input symbol into 128-dimensional continuous
vector without using any non-linearity. The RNN module is the HM-LSTM, described in Section 3,
with three layers. The output module is a feedforward neural network with two layers, an output
embedding layer and a softmax layer. Figure 2 (right) shows a diagram of the output module. At each
time step, the output embedding layer receives the hidden states of the three RNN layers as input. In
order to adaptively control the importance of each layer at each time step, we also introduce three
scalar gating units g`
t2Rto each of the layer outputs:
g`
t=sigm (w`[h1
t;;hL
t]); (11)
where w`2RPL


key: chroma:document
string_value:
g`
t=sigm (w`[h1
t;;hL
t]); (11)
where w`2RPL
`=1dim (h`)is the weight parameter. The output embedding he
tis computed by:
he
t=ReLU LX
`=1g`
tWe
`h`
t!
; (12)
whereL= 3andReLU (x) = max(0;x)(Nair & Hinton, 2010). Finally, the probability distribution
for the next target character is computed by the softmax function, softmax (xj) =exjPK
k=1exk, where
each output class is a character.
7
Published as a conference paper at ICLR 2017
Text8
Model BPC
td-LSTM (Zhang et al., 2016) 1.63
HF-MRNN (Mikolov et al., 2012) 1.54
MI-RNN (Wu et al., 2016) 1.52
Skipping-RNN (Pachitariu & Sahani, 2013) 1.48
MI-LSTM (Wu et al., 2016) 1.44
BatchNorm LSTM (Cooijmans et al., 2016) 1.36
HM-LSTM 1.32


key: chroma:document
string_value:
BatchNorm LSTM (Cooijmans et al., 2016) 1.36
HM-LSTM 1.32
LayerNorm HM-LSTM 1.29
Table 2: BPC on the Text8 test set.
Penn Treebank We process the Penn Treebank dataset (Marcus et al., 1993) by following the
procedure introduced in Mikolov et al. (2012). Each update is done by using a mini-batch of 64
examples of length 100to prevent the memory overﬂow problem when unfolding the RNN in time
for backpropagation. The last hidden state of a sequence is used to initialize the hidden state of the
next sequence to approximate the full backpropagation. We train the model using Adam (Kingma &
Ba, 2014) with an initial learning rate of 0:002. We divide the learning rate by a factor of 50when
the validation negative log-likelihood stopped decreasing. The norm of the gradient is clipped with a
threshold of 1(Mikolov et al., 2010; Pascanu et al., 2012). We also apply layer normalization (Ba
et al., 2016) to our models. For all of the character-level language modelling experiments, we apply
the same procedure, but only change the number of hidden units, mini-batch size and the initial
learning rate.


key: chroma:document
string_value:
the same procedure, but only change the number of hidden units, mini-batch size and the initial
learning rate.
For the Penn Treebank dataset, we use 512units in each layer of the HM-LSTM and for the output
embedding layer. In Table 1 (left), we compare the test BPCs of four variants of our model to other
baseline models. Note that the HM-LSTM using the step function for the hard boundary decision
outperforms the others using either sampling orsoftboundary decision (i.e., hard sigmoid). The test
BPC is further improved with the slope annealing trick, which reduces the bias of the straight-through
estimator. We increased the slope awith the following schedule a= min (5 ;1 + 0:04Nepoch ), where
Nepoch is the maximum number of epochs. The HM-LSTM achieves test BPC score of 1:24. For the
remaining tasks, we ﬁxed the hard boundary decision using the step function without slope annealing
due to the difﬁculty of ﬁnding a good annealing schedule on large-scale datasets.
Text8 The Text8 dataset (Mahoney, 2009) consists of 100M characters extracted from the Wikipedia


key: chroma:document
string_value:
Text8 The Text8 dataset (Mahoney, 2009) consists of 100M characters extracted from the Wikipedia
corpus. Text8 contains only alphabets and spaces, and thus we have total 27 symbols. In order to
compare with other previous works, we follow the data splits used in Mikolov et al. (2012). We use
1024 units for each HM-LSTM layer and 2048 units for the output embedding layer. The mini-batch
size and the initial learning rate are set to 128and0:001, respectively. The results are shown in
Table 2. The HM-LSTM obtains the state-of-the-art test BPC 1:29.
Hutter Prize Wikipedia The Hutter Prize Wikipedia ( enwik8 ) dataset (Hutter, 2012) contains
205symbols including XML markups and special characters. We follow the data splits used in Graves
(2013) where the ﬁrst 90M characters are used to train the model, the next 5M characters for validation,
and the remainders for the test set. We use the same model size, mini-batch size and the initial
learning rate as in the Text8. In Table 1 (right), we show the HM-LSTM achieving the test BPC 1:32,


key: chroma:document
string_value:
learning rate as in the Text8. In Table 1 (right), we show the HM-LSTM achieving the test BPC 1:32,
which is a tie with the state-of-the-art result among the neural models. Although the neural models,
show remarkable performances, their compression performance is still behind the best models such
as PAQ8hp12 (Mahoney, 2005) and decomp8 (Mahoney, 2009).
Visualizing Learned Hierarchical Multiscale Structure In Figure 3 and 4, we visualize the
boundaries detected by the boundary detectors of the HM-LSTM while reading a character sequence
of total length 270taken from the validation set of either the Penn Treebank or Hutter Prize Wikipedia
dataset. Due to the page width limit, the ﬁgure contains the sequence partitioned into three segments
of length 90. The white blocks indicate boundaries z`
t= 1 while the black blocks indicate the
non-boundaries z`
t= 0.
Interestingly in both ﬁgures, we can observe that the boundary detector of the ﬁrst layer, z1, tends
to be turned on when it sees a space or after it sees a space, which is a reasonable breakpoint to
separate between words. This is somewhat surprising because the model self-organizes this structure
8


key: chroma:document
string_value:
separate between words. This is somewhat surprising because the model self-organizes this structure
8
Published as a conference paper at ICLR 2017
Figure 3: Hierarchical multiscale structure in the Wikipedia dataset captured by the boundary
detectors of the HM-LSTM.
Penn Treebank Line 1Penn Treebank Line 2Penn Treebank Line 3
Figure 4: The `2-norm of the hidden states shown together with the states of the boundary detectors
of the HM-LSTM.
without any explicit boundary information. In Figure 3, we observe that the z1tends to detect the
boundaries of the words but also ﬁres within the words, where the z2tends to ﬁre when it sees either
an end of a word or 2;3-grams. In Figure 4, we also see ﬂushing in the middle of a word, e.g.,
“tele-FLUSH-phone”. Note that “tele” is a preﬁx after which a various number of postﬁxes can follow.
From these, it seems that the model uses to some extent the concept of surprise to learn the boundary.
Although interpretation of the second layer boundaries is not as apparent as the ﬁrst layer boundaries,


key: chroma:document
string_value:
Although interpretation of the second layer boundaries is not as apparent as the ﬁrst layer boundaries,
it seems to segment at reasonable semantic / syntactic boundaries, e.g., “consumers may” - “want to
move their telephones a” - “little closer to the tv set <unk>”, and so on.
Another remarkable point is the fact that we do not pose any constraint on the number of boundaries
that the model can ﬁre up. The model, however, learns that it is more beneﬁcial to delay the
information ejection to some extent. This is somewhat counterintuitive because it might look more
beneﬁcial to feed the fresh update to the upper layers at every time step without any delay. We
conjecture the reason that the model works in this way is due to the FLUSH operation that poses an
implicit constraint on the frequency of boundary detection, because it contains both a reward (feeding
fresh information to upper layers) and a penalty (erasing accumulated information). The model ﬁnds
an optimal balance between the reward and the penalty.
To understand the update mechanism more intuitively, in Figure 4, we also depict the heatmap of the
`2-norm of the hidden states along with the states of the boundary detectors. As we expect, we can


key: chroma:document
string_value:
`2-norm of the hidden states along with the states of the boundary detectors. As we expect, we can
see that there is no change in the norm value within segments due to the COPY operation. Also, the
color ofkh1kchanges quickly (at every time step) because there is no COPY operation in the ﬁrst
layer. The color of kh2kchanges less frequently based on the states of z1
tandz2
t1. The color of
kh3kchanges even slowly, i.e., only when z2
t= 1.
A notable advantage of the proposed architecture is that the internal process of the RNN becomes
more interpretable. For example, we can substitute the states of z1
tandz2
t1into Eq. 2 and infer
which operation among the UPDATE, COPY and FLUSH was applied to the second layer at time step
t. We can also inspect the update frequencies of the layers simply by counting how many UPDATE
and FLUSH operations were made in each layer. For example in Figure 4, we see that the ﬁrst layer
updates at every time step (which is 270UPDATE operations), the second layer updates 56times,
9
Published as a conference paper at ICLR 2017
IAM-OnDB
Model Average Log-Likelihood
Standard LSTM 1081


key: chroma:document
string_value:
9
Published as a conference paper at ICLR 2017
IAM-OnDB
Model Average Log-Likelihood
Standard LSTM 1081
HM-LSTM 1137
HM-LSTM & Slope Annealing 1167
Table 3: Average log-likelihood per sequence on the IAM-OnDB test set.
the ground truth of pen-tip locationVisualization by segments usingVisualization by segments usingthe states ofz2
Figure 5: The visualization by segments based on either the given pen-tip location or states of the z2.
and only 9updates has made in the third layer. Note that, by design, the ﬁrst layer performs UPDATE
operation at every time step and then the number of UPDATE operations decreases as the layer level
increases. In this example, the total number of updates is 335for the HM-LSTM which is 60% of
reduction from the 810updates of the standard RNN architecture.
4.2 H ANDWRITING SEQUENCE GENERATION
We extend the evaluation of the HM-LSTM to a real-valued sequence modelling task using IAMOnDB (Liwicki & Bunke, 2005) dataset. The IAM-OnDB dataset consists of 12;179handwriting


key: chroma:document
string_value:
examples, each of which is a sequence of (x;y)coordinate and a binary indicator pfor pen-tip
location, giving us (x1:Tn;y1:Tn;p1:Tn), wherenis an index of a sequence. At each time step,
the model receives (xt;yt;pt), and the goal is to predict (xt+1;yt+1;pt+1). The pen-up ( pt= 1)
indicates an end of a stroke, and the pen-down ( pt= 0) indicates that a stroke is in progress. There
is usually a large shift in the (x;y)coordinate to start a new stroke after the pen-up happens. We
remove all sequences whose length is shorter than 300. This leaves us 10;465sequences for training,
581for validation, 582for test. The average length of the sequences is 648. We normalize the range
of the (x;y)coordinates separately with the mean and standard deviation obtained from the training
set. We use the mini-batch size of 32, and the initial learning rate is set to 0:0003 .
We use the same model architecture as used in the character-level language model, except that the
output layer is modiﬁed to predict real-valued outputs. We use the mixture density network as the


key: chroma:document
string_value:
output layer is modiﬁed to predict real-valued outputs. We use the mixture density network as the
output layer following Graves (2013), and use 400units for each HM-LSTM layer and for the output
embedding layer. In Table 3, we compare the log-likelihood averaged over the test sequences of the
IAM-OnDB dataset. We observe that the HM-LSTM outperforms the standard LSTM. The slope
annealing trick further improves the test log-likelihood of the HM-LSTM into 1167 in our setting. In
this experiment, we increased the slope awith the following schedule a= min (3 ;1 + 0:004Nepoch ).
In Figure 5, we let the HM-LSTM to read a randomly picked validation sequence and present the
visualization of handwriting examples by segments based on either the states of z2or the states of
pen-tip location3.
5 C ONCLUSION
In this paper, we proposed the HM-RNN that can capture the latent hierarchical structure of the
sequences. We introduced three types of operations to the RNN, which are the COPY , UPDATE
and FLUSH operations. In order to implement these operations, we introduced a set of binary
variables and a novel update rule that is dependent on the states of these binary variables. Each binary


key: chroma:document
string_value:
and FLUSH operations. In order to implement these operations, we introduced a set of binary
variables and a novel update rule that is dependent on the states of these binary variables. Each binary
variable is learned to ﬁnd segments at its level, therefore, we call this binary variable, a boundary
detector. On the character-level language modelling, the HM-LSTM achieved state-of-the-art result
on the Text8 dataset and comparable results to the state-of-the-art results on the Penn Treebank
and Hutter Prize Wikipedia datasets. Also, the HM-LSTM outperformed the standard LSTM on
the handwriting sequence generation. Our results and analysis suggest that the proposed HM-RNN
can discover the latent hierarchical structure of the sequences and can learn efﬁcient hierarchical
multiscale representation that leads to better generalization performance.
3The plot function could be found at blog.otoro.net/2015/12/12/handwriting-generation-demo-in-tensorflow/ .
10
Published as a conference paper at ICLR 2017
ACKNOWLEDGMENTS
The authors would like to thank Alex Graves, Tom Schaul and Hado van Hasselt for their fruitful
comments and discussion. We acknowledge the support of the following agencies for research


key: chroma:document
string_value:
The authors would like to thank Alex Graves, Tom Schaul and Hado van Hasselt for their fruitful
comments and discussion. We acknowledge the support of the following agencies for research
funding and computing support: Ubisoft, Samsung, IBM, Facebook, Google, Microsoft, NSERC,
Calcul Québec, Compute Canada, the Canada Research Chairs and CIFAR. The authors thank the
developers of Theano (Team et al., 2016). JC would like to thank Arnaud Bergenon and Frédéric
Bastien for their technical support. JC would also like to thank Guillaume Alain, Kyle Kastner and
David Ha for providing us useful pieces of code.
REFERENCES
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450 ,
2016.
Dzmitry Bahdanau, Jan Chorowski, Dmitriy Serdyuk, Yoshua Bengio, et al. End-to-end attention-based large
vocabulary speech recognition. In 2016 IEEE International Conference on Acoustics, Speech and Signal
Processing (ICASSP) , pp. 4945–4949. IEEE, 2016.


key: chroma:document
string_value:
vocabulary speech recognition. In 2016 IEEE International Conference on Acoustics, Speech and Signal
Processing (ICASSP) , pp. 4945–4949. IEEE, 2016.
Yoshua Bengio. Learning deep architectures for ai. Foundations and trends R
in Machine Learning , 2(1):1–127,
2009.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic
neurons for conditional computation. arXiv preprint arXiv:1308.3432 , 2013.
William Chan, Navdeep Jaitly, Quoc Le, and Oriol Vinyals. Listen, attend and spell: A neural network for large
vocabulary conversational speech recognition. In 2016 IEEE International Conference on Acoustics, Speech
and Signal Processing (ICASSP) , pp. 4960–4964. IEEE, 2016.
Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Fethi Bougares, Holger Schwenk, and Yoshua
Bengio. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In
Proceedings of the Empiricial Methods in Natural Language Processing (EMNLP 2014) , October 2014.


key: chroma:document
string_value:
Proceedings of the Empiricial Methods in Natural Language Processing (EMNLP 2014) , October 2014.
Junyoung Chung, Caglar Gulcehre, Kyunghyun Cho, and Yoshua Bengio. Gated feedback recurrent neural
networks. In Proceedings of the 32nd International Conference on Machine Learning (ICML) , 2015.
Junyoung Chung, Kyunghyun Cho, and Yoshua Bengio. A character-level decoder without explicit segmentation
for neural machine translation. Association for Computational Linguistics (ACL) , 2016.
Tim Cooijmans, Nicolas Ballas, César Laurent, and Aaron Courville. Recurrent batch normalization. arXiv
preprint arXiv:1603.09025 , 2016.
Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Binarized neural
networks: Training deep neural networks with weights and activations constrained to+ 1 or-1. arXiv preprint
arXiv:1602.02830 , 2016.
Salah El Hihi and Yoshua Bengio. Hierarchical recurrent neural networks for long-term dependencies. In
Advances in Neural Information Processing Systems , pp. 493–499. Citeseer, 1995.


key: chroma:document
string_value:
Advances in Neural Information Processing Systems , pp. 493–499. Citeseer, 1995.
Santiago Fernández, Alex Graves, and Jürgen Schmidhuber. Sequence labelling in structured domains with
hierarchical recurrent neural networks. In Proceedings of the 20th international joint conference on Artiﬁcal
intelligence , pp. 774–779. Morgan Kaufmann Publishers Inc., 2007.
Alex Graves. Generating sequences with recurrent neural networks. arXiv preprint arXiv:1308.0850 , 2013.
David Ha, Andrew Dai, and Quoc V Le. Hypernetworks. arXiv preprint arXiv:1609.09106 , 2016.
G. Hinton. Neural networks for machine learning. Coursera, video lectures, 2012.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation , 9(8):1735–1780,
1997.
Judy Hoffman, Eric Tzeng, Jeff Donahue, Yangqing Jia, Kate Saenko, and Trevor Darrell. One-shot adaptation
of supervised deep convolutional models. arXiv preprint arXiv:1312.6204 , 2013.


key: chroma:document
string_value:
of supervised deep convolutional models. arXiv preprint arXiv:1312.6204 , 2013.
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Weinberger. Deep networks with stochastic depth.
arXiv preprint arXiv:1603.09382 , 2016.
11
Published as a conference paper at ICLR 2017
Marcus Hutter. The human knowledge compression contest. 2012. URL http://prize.hutter1.net/ .
Nal Kalchbrenner, Ivo Danihelka, and Alex Graves. Grid long short-term memory. arXiv preprint
arXiv:1507.01526 , 2015.
Yoon Kim. Convolutional neural networks for sentence classiﬁcation. arXiv preprint arXiv:1408.5882 , 2014.
Yoon Kim, Yacine Jernite, David Sontag, and Alexander M Rush. Character-aware neural language models.
arXiv preprint arXiv:1508.06615 , 2015.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 ,
2014.


key: chroma:document
string_value:
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 ,
2014.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 , 2013.
Lingpeng Kong, Chris Dyer, and Noah A Smith. Segmental recurrent neural networks. arXiv preprint
arXiv:1511.06018 , 2015.
Jan Koutník, Klaus Greff, Faustino Gomez, and Jürgen Schmidhuber. A clockwork rnn. In Proceedings of the
31st International Conference on Machine Learning (ICML 2014) , 2014.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classiﬁcation with deep convolutional neural
networks. In Advances in Neural Information Processing Systems , pp. 1097–1105, 2012.
David Krueger and Roland Memisevic. Regularizing rnns by stabilizing activations. arXiv preprint
arXiv:1511.08400 , 2015.


key: chroma:document
string_value:
David Krueger and Roland Memisevic. Regularizing rnns by stabilizing activations. arXiv preprint
arXiv:1511.08400 , 2015.
David Krueger, Tegan Maharaj, János Kramár, Mohammad Pezeshki, Nicolas Ballas, Nan Rosemary Ke, Anirudh
Goyal, Yoshua Bengio, Hugo Larochelle, Aaron Courville, et al. Zoneout: Regularizing rnns by randomly
preserving hidden activations. arXiv preprint arXiv:1606.01305 , 2016.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature , 521(7553):436–444, 2015.
Tsungnan Lin, Bill G Horne, Peter Tino, and C Lee Giles. Learning long-term dependencies in narx recurrent
neural networks. IEEE Transactions on Neural Networks , 7(6):1329–1338, 1996.
Wang Ling, Isabel Trancoso, Chris Dyer, and Alan W Black. Character-based neural machine translation. arXiv
preprint arXiv:1511.04586 , 2015.
Marcus Liwicki and Horst Bunke. Iam-ondb-an on-line english sentence database acquired from handwritten


key: chroma:document
string_value:
Marcus Liwicki and Horst Bunke. Iam-ondb-an on-line english sentence database acquired from handwritten
text on a whiteboard. In Eighth International Conference on Document Analysis and Recognition (ICDAR’05) ,
pp. 956–961. IEEE, 2005.
Matthew V Mahoney. Adaptive weighing of context models for lossless data compression. 2005.
Matthew V Mahoney. Large text compression benchmark. URL: http://www. mattmahoney. net/text/text. html ,
2009.
Mitchell P Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. Building a large annotated corpus of
english: The penn treebank. Computational linguistics , 19(2):313–330, 1993.
Tomas Mikolov, Martin Karaﬁát, Lukas Burget, Jan Cernock `y, and Sanjeev Khudanpur. Recurrent neural
network based language model. In INTERSPEECH , volume 2, pp. 3, 2010.
Tomas Mikolov, Ilya Sutskever, Anoop Deoras, Hai-Son Le, Stefan Kombrink, and J Cernocky. Subword language modeling with neural networks. Preprint , 2012. URL http://www.fit.vutbr.cz/


key: chroma:document
string_value:
~imikolov/rnnlm/char.pdf .
Andriy Mnih and Karol Gregor. Neural variational inference and learning in belief networks. In Proceedings of
the 31st International Conference on Machine Learning (ICML-14) , pp. 1791–1799, 2014.
Michael C Mozer. Induction of multiscale temporal structure. Advances in neural information processing
systems , pp. 275–275, 1993.
Vinod Nair and Geoffrey E Hinton. Rectiﬁed linear units improve restricted boltzmann machines. In Proceedings
of the 27th International Conference on Machine Learning (ICML-10) , pp. 807–814, 2010.
Marius Pachitariu and Maneesh Sahani. Regularization and nonlinearities for neural language models: when are
they needed? arXiv preprint arXiv:1301.5650 , 2013.
12
Published as a conference paper at ICLR 2017
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difﬁculty of training recurrent neural networks.
arXiv preprint arXiv:1211.5063 , 2012.
Kamil M Rocki. Recurrent memory array structures. arXiv preprint arXiv:1607.03085 , 2016a.


key: chroma:document
string_value:
Kamil M Rocki. Recurrent memory array structures. arXiv preprint arXiv:1607.03085 , 2016a.
Kamil M Rocki. Surprisal-driven feedback in recurrent networks. arXiv preprint arXiv:1608.06027 , 2016b.
Jürgen Schmidhuber. Neural sequence chunkers. 1991.
Jürgen Schmidhuber. Learning complex, extended sequences using the principle of history compression. Neural
Computation , 4(2):234–242, 1992.
Jürgen Schmidhuber. Deep learning in neural networks: An overview. Neural Networks , 61:85–117, 2015.
Alessandro Sordoni, Yoshua Bengio, Hossein Vahabi, Christina Lioma, Jakob Grue Simonsen, and Jian-Yun Nie.
A hierarchical recurrent encoder-decoder for generative context-aware query suggestion. In Proceedings of
the 24th ACM International on Conference on Information and Knowledge Management , pp. 553–562. ACM,
2015.
Nitish Srivastava, Geoffrey E Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout:


key: chroma:document
string_value:
Nitish Srivastava, Geoffrey E Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout:
a simple way to prevent neural networks from overﬁtting. Journal of Machine Learning Research , 15(1):
1929–1958, 2014.
Ilya Sutskever, James Martens, and Geoffrey E Hinton. Generating text with recurrent neural networks. In
Proceedings of the 28th International Conference on Machine Learning (ICML’11) , pp. 1017–1024, 2011.
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In Advances
in Neural Information Processing Systems , pp. 3104–3112, 2014.
The Theano Development Team, Rami Al-Rfou, Guillaume Alain, Amjad Almahairi, Christof Angermueller,
Dzmitry Bahdanau, Nicolas Ballas, Frédéric Bastien, Justin Bayer, Anatoly Belikov, et al. Theano: A python
framework for fast computation of mathematical expressions. arXiv preprint arXiv:1605.02688 , 2016.


key: chroma:document
string_value:
framework for fast computation of mathematical expressions. arXiv preprint arXiv:1605.02688 , 2016.
Alexander Vezhnevets, V olodymyr Mnih, John Agapiou, Simon Osindero, Alex Graves, Oriol Vinyals, Koray
Kavukcuoglu, et al. Strategic attentive writer for learning macro-actions. arXiv preprint arXiv:1606.04695 ,
2016.
Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: A neural image caption
generator. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pp. 3156–
3164, 2015.
Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning.
Machine learning , 8(3-4):229–256, 1992.
Yuhuai Wu, Saizheng Zhang, Ying Zhang, Yoshua Bengio, and Ruslan Salakhutdinov. On multiplicative
integration with recurrent neural networks. arXiv preprint arXiv:1606.06630 , 2016.
Saizheng Zhang, Yuhuai Wu, Tong Che, Zhouhan Lin, Roland Memisevic, Ruslan Salakhutdinov, and Yoshua


key: chroma:document
string_value:
Saizheng Zhang, Yuhuai Wu, Tong Che, Zhouhan Lin, Roland Memisevic, Ruslan Salakhutdinov, and Yoshua
Bengio. Architectural complexity measures of recurrent neural networks. arXiv preprint arXiv:1602.08210 ,
2016.
Julian Georg Zilly, Rupesh Kumar Srivastava, Jan Koutník, and Jürgen Schmidhuber. Recurrent highway
networks. arXiv preprint arXiv:1607.03474 , 2016.
13


key: chroma:document
string_value:
ARNAB AND TORR: BOTTOM-UP INSTANCE SEGMENTATION 1
Bottom-up Instance Segmentation using
Deep Higher-Order CRFs
Anurag Arnab
anurag.arnab@eng.ox.ac.uk
Philip H.S. Torr
philip.torr@eng.ox.ac.ukDepartment of Engineering Science
University of Oxford
United Kingdom
Abstract
Traditional Scene Understanding problems such as Object Detection and Semantic
Segmentation have made breakthroughs in recent years due to the adoption of deep learning. However, the former task is not able to localise objects at a pixel level, and the latter
task has no notion of different instances of objects of the same class. We focus on the task
of Instance Segmentation which recognises and localises objects down to a pixel level.
Our model is based on a deep neural network trained for semantic segmentation. This
network incorporates a Conditional Random Field with end-to-end trainable higher order potentials based on object detector outputs. This allows us to reason about instances
from an initial, category-level semantic segmentation. Our simple method effectively
leverages the great progress recently made in semantic segmentation and object detection. The accurate instance-level segmentations that our network produces is reﬂected by
the considerable improvements obtained over previous work at high APrIoU thresholds.


key: chroma:document
string_value:
the considerable improvements obtained over previous work at high APrIoU thresholds.
1 Introduction
Object detection and semantic segmentation have been two of the most popular Scene Understanding problems within the Computer Vision community. Great advances have been made
in recent years due to the adoption of deep learning and Convolutional Neural Networks
[19, 22, 32]. In this paper, we focus on the problem of Instance Segmentation . Instance
Segmentation lies at the intersection of Object Detection – which localises different objects
at a bounding box level, but does not segment them – and Semantic Segmentation – which
determines the object-class label of each pixel in the image, but has no notion of different
instances of the same class. As shown in Figure 1, the task of instance segmentation localises
objects to a pixel level.
Many recent instance segmentation works have built on the “Simultaneous Detection
and Segmentation” (SDS) approach of Hariharan et al. [13]. These methods [4, 5, 13, 14]
generate object proposals [1], classify each proposal into an object category, and then reﬁne
the bounding box proposal into a segmentation of the primary object the proposal contains.
However, because these methods localise an object before segmenting it, they are restricted


key: chroma:document
string_value:
the bounding box proposal into a segmentation of the primary object the proposal contains.
However, because these methods localise an object before segmenting it, they are restricted
by the quality of the initial proposal. Moreover, the proposal generator [1] these works used
takes about 40 seconds to process an image.
c/circlecopyrt2016. The copyright of this document resides with its authors.
It may be distributed unchanged freely in print or electronic forms.
arXiv:1609.02583v1  [cs.CV]  8 Sep 2016
2 ARNAB AND TORR: BOTTOM-UP INSTANCE SEGMENTATION
(a) Object Detection (b) Semantic Segmentation (c) Instance Segmentation
Figure 1: Our method uses the results of an object detector and the original image (a) to produce a category-level semantic segmentation (b). The semantic segmentation result, object
detection bounding boxes, and recalibrated detection scores (§3.2) are then used to reﬁne
the semantic segmentation into an instance segmentation (c). The semantic segmentation,
in which a large number of pixels have been correctly labelled as “person”, does not tell us


key: chroma:document
string_value:
the semantic segmentation into an instance segmentation (c). The semantic segmentation,
in which a large number of pixels have been correctly labelled as “person”, does not tell us
how many people there are and where each person is in the image. The instance segmentation answers these questions. Also note how our method is not affected by the false-positive
“bottle” detections.
We present a different approach to instance segmentation, where we initially perform
a category-level, semantic segmentation of the image before reasoning about the different
instances in the scene. We can identify instances because our semantic segmentation network
incorporates a Higher Order Conditional Random Field (CRF) which uses cues from the
output of an object detector. This CRF is fully differentiable and can be inserted as a layer
of a neural network [2, 37]. As a result, it can be used as a simple extension to an existing
semantic segmentation network to perform the related task of instance segmentation.
Our simple, bottom-up method is able to effectively leverage the progress made by stateof-the-art semantic segmentation and object detection networks to perform the related task
of instance segmentation. We show this by evaluating our approach on the PASCAL VOC
2012 [8] dataset, using the standard APrmeasure [13]. The fact that our system is able to


key: chroma:document
string_value:
2012 [8] dataset, using the standard APrmeasure [13]. The fact that our system is able to
produce accurate instance segmentations is reﬂected by the considerable improvements that
we make over previous work at high overlap thresholds.
2 Related Work
Instance segmentation is closely related to the widely-studied problems of semantic segmentation and object detection. We thus review these two tasks ﬁrst.
Semantic Segmentation: This task was traditionally performed by extracting dense features from an image and classifying them on a per-pixel or per-region basis [16, 31]. The
individual predictions made by these classiﬁers were often noisy as they lacked global context. As a result, they were post-processed with a CRF whose priors encourages nearby
pixels, and pixels of similar appearance, to share the same class label. The CRF of [31]
initially contained only unary terms (obtained from the classiﬁer) and 8-connected pairwise
terms. Subsequent improvements included introducing densely-connected pairwise potentials [18], using higher order potentials encouraging consistency over cliques larger than two
pixels [17, 35], modelling the co-occurrence of labels [20, 35] and utilising the results of
object detectors [21, 33, 36].


key: chroma:document
string_value:
pixels [17, 35], modelling the co-occurrence of labels [20, 35] and utilising the results of
object detectors [21, 33, 36].
Using CNNs for semantic segmentation can greatly improve the unary, per-pixel predictions, as shown by [27]. Chen et al. showed further improvements by post-processing the
ARNAB AND TORR: BOTTOM-UP INSTANCE SEGMENTATION 3
CNN output with a CRF [3]. Current state-of-the-art methods [2, 25, 37] incorporate inference of a CRF as layers within a deep network which perform the differentiable mean ﬁeld
inference algorithm. In our work, we extend the end-to-end trainable, higher-order, object
detection potentials proposed by [2] to the task of instance segmentation. Additionally, unlike the aforementioned works which always assume a ﬁxed number of labels in the CRF at
all times, our system includes a CRF where the number of labels varies per input image.
Object Detection: Part-based models [9] were popular before CNNs, which had been
shown to excel in the task of image classiﬁcation [19, 32], were adapted to object detection. Recent detection algorithms are based on using CNNs to classify object proposals


key: chroma:document
string_value:
shown to excel in the task of image classiﬁcation [19, 32], were adapted to object detection. Recent detection algorithms are based on using CNNs to classify object proposals
and then performing bounding box regression to reﬁne the initial proposals’ bounding boxes
[10, 11, 12, 28]. This pipeline can be accelerated by computing convolutional features over
the whole image, and then pooling features for each proposal [11, 15]. Object proposals
were initially hand-crafted [1, 34], but this step has now been integrated into CNNs [28].
Instance Segmentation: This task was popularised by Hariharan et al. [13]. Their approach was similar to object detection pipelines in that region proposals were ﬁrst generated and classiﬁed into different object categories before using bounding box regression as
post-processing. A class-speciﬁc segmentation was then performed to simultaneously detect
and segment an object. However, approaches that segment instances by reﬁning detections
[4, 5, 13, 14] are inherently limited by the quality of the initial proposals, and the systems
cannot recover from errors made by the initial detector. Furthermore, since these methods
generate and classify multiple overlapping region proposals per image, they cannot actually


key: chroma:document
string_value:
cannot recover from errors made by the initial detector. Furthermore, since these methods
generate and classify multiple overlapping region proposals per image, they cannot actually
produce segmentation maps of the form shown in Figures 1(b) and (c) [24]. Liang et al. [23]
address some of these shortcomings by iteratively reﬁning initial object proposals, whilst [7]
generates box-proposals, creates masks from these proposals and then classiﬁes these masks
in a single network trained with three loss functions summed together.
A proposal-free method was recently presented by [24] where a semantic segmentation of the image is ﬁrst performed using the network of [3]. Thereafter the category-level
segmentation output, along with CNN features, are used to predict instance-level bounding
boxes for the objects in the image. The number of instances of each object category are also
predicted in order to facilitate the ﬁnal clustering step. An alternate method was proposed
by [29] where a Recurrent Neural Network outputs an object instance at each time step. This
method, however, has not been evaluated on multiple classes.


key: chroma:document
string_value:
by [29] where a Recurrent Neural Network outputs an object instance at each time step. This
method, however, has not been evaluated on multiple classes.
Our method also ﬁrst performs a bottom-up, semantic segmentation of the image. However, our semantic segmentation network uses the outputs of an object detector as an additional cue in its ﬁnal CRF layer. During inference, each object hypothesis from the detector
is evaluated in light of other energies in the CRF. Thus, the relative score of false positive
detections can be decreased, and correct detections increased. These rescored detections
are then used to identify instances from the initial semantic segmentation. The fact that our
system combines information from semantic segmentation and object detection networks allows us to bypass the complexity of the “instance bounding box” and “instance number”
prediction steps used by [24].
3 Proposed Approach
Our proposed method ﬁrst performs a semantic segmentation of the input image, classifying
each pixel into one of K+1 categories where Kis the number of foreground classes. The
resulting semantic segmentation is then reﬁned into an instance-level segmentation, where
4 ARNAB AND TORR: BOTTOM-UP INSTANCE SEGMENTATION
Pixelwise CNN  Higher Order CRF


key: chroma:document
string_value:
4 ARNAB AND TORR: BOTTOM-UP INSTANCE SEGMENTATION
Pixelwise CNN  Higher Order CRF  
Instance Identification  Instance CRF  
K+1  
Object Detector  
D+1  End-to-end object segmentation subnetwork  
End-to-end Instance segmentation subnetwork  
Input  
Output  Object  
Segmentation  
Identified  
Instances  
Figure 2: Overview of our end-to-end method. Our system ﬁrst computes a category-level
semantic segmentation of the image. A CRF with higher-order detection potentials is used
to obtain this result in a semantic segmentation subnetwork. This results in a W×H×(K+
1)dimensional volume where Kis the number of foreground classes. WandHare the
image’s width and height respectively. The original detections and the detector conﬁdences
recalibrated by the Higher Order CRF are used to identify instances in the image, producing
aW×H×(D+1)dimensional volume where Dis the number of detections (variable per
image). Thereafter, another Instance CRF is used to compute the ﬁnal result.


key: chroma:document
string_value:
image). Thereafter, another Instance CRF is used to compute the ﬁnal result.
the object class of each instance segment is obtained from the previous semantic segmentation. Both of these stages, while conceptually different, are fully differentiable and the entire
system can be implemented as a neural network.
Figure 2 shows our category-level segmentation network, consisting of a pixelwise CNN
[27] followed by a fully differentiable CRF with object detection potentials, as described in
Section 3.2. The object detection results that our CRF takes in, as well as the recalibrated
detection scores that it outputs, allows us to identify object instances, and obtain a ﬁnal
instance-level segmentation in the next stage of our pipeline. This is described in Section
3.3. However, we ﬁrst review Conditional Random Fields and introduce the notation used in
this paper.
3.1 Conditional Random Fields
Assume an image Iwith Npixels, indexed 1 ,2...N, and deﬁne a set of random variables,
X1,X2,...,XN, one for every pixel. In a generic labelling problem, we assign every pixel a


key: chroma:document
string_value:
X1,X2,...,XN, one for every pixel. In a generic labelling problem, we assign every pixel a
label from a predeﬁned set of labels Lsuch that each Xi∈L. Let X= [X1X2...XN]T. In
this case, any particular assignment xtoXis a solution to our labelling problem.
In the case of semantic segmentation, we assign each pixel, Xi, a label corresponding
to object categories such as “person” and “car”. In instance segmentation, the labels are
drawn from the product label space of object categories and instance numbers. Examples are
“person_1” and “person_2” as shown by the different colours in Figure 1.
Given a graph Gwhere the vertices are from {X}and connections between these variables are deﬁned by edges, the pair (I,X)is modelled as a CRF deﬁned by Pr (X=x|I) =
(1/Z(I))exp(−E(x|I)). The term, E(x|I)is the energy of the labelling xandZ(I)is the
ARNAB AND TORR: BOTTOM-UP INSTANCE SEGMENTATION 5


key: chroma:document
string_value:
ARNAB AND TORR: BOTTOM-UP INSTANCE SEGMENTATION 5
data-dependent partition function. From here onwards, we drop the conditioning on Ito
keep the notation uncluttered. The energy of an assignment is deﬁned over the set of cliques
Cin the graph Gas
E(x) =∑
c∈Cψc(xc), (1)
where xcis the vector formed by selecting elements of xthat correspond to random variables
belonging to the clique c, andψc(.)is the cost function for a clique c.
The energy function considered by many semantic segmentation works, E(x) =∑iψU
i(xi)+
∑i<jψP
i j(xi,xj), consists of only unary (cliques of size one) and densely-connected pairwise
potentials (cliques of size two) [3, 18, 37]. When dense pairwise potentials are used in a
CRF to obtain higher accuracy, exact inference is not tractable and approximate inference
methods such as mean ﬁeld are used [18].
3.2 Higher Order Detection Potentials
In addition to the standard unary and densely-connected pairwise terms, we also include a


key: chroma:document
string_value:
3.2 Higher Order Detection Potentials
In addition to the standard unary and densely-connected pairwise terms, we also include a
higher order term based on object detections. Intuitively, a state-of-the-art object detector
[11, 28] can help segmentation performance in cases where the detector correctly predicts an
object at a location in which the pixels have been misclassiﬁed by the unary classiﬁer, and
thus reduce the recognition error [6]. Arnab et al. [2] formulated object detection potentials
in a manner that is amenable to the differentiable mean ﬁeld inference algorithm so that it
could be integrated into a neural network, and showed that it does indeed improve semantic
segmentation performance. We review the detection potentials described in [2] as they also
help us to reason about instances in an input image, a problem not considered by [2].
Assume that we have Dobject detections for an input image (this number varies for every
image), and that the dthdetection is of the form (ld,sd,Fd,Bd)where ld∈Lis the object class
label of the detected object, sdis the detector’s conﬁdence score, Fd⊂{1,2,...N}is the set


key: chroma:document
string_value:
label of the detected object, sdis the detector’s conﬁdence score, Fd⊂{1,2,...N}is the set
of indices of the pixels belonging to the foreground of the detection and Bd⊂{1,2,...N}is
the set of indices falling in the detection’s bounding box. The foreground is obtained from
a foreground/background segmentation method such as GrabCut [30] and indicates a rough
segmentation of the detected object. The detection potentials should encourage the set of
pixels represented by Fdto take the object class label, ld. However, this should be a soft
constraint, since the foreground segmentation could be inaccurate and the entire detection
itself could be a false-positive. The detection dshould also be identiﬁed as invalid if other
energies in the CRF strongly suggest that many pixels in Fddo not take the label ld.
This is formulated by introducing a latent binary random variable, Y1,Y2...YDfor every
detection. If the dthdetection has been found to be valid after inference, Ydwill be set to 1,
and 0 otherwise. The ﬁnal value of Ydis determined probabilistically by mean ﬁeld inference.


key: chroma:document
string_value:
and 0 otherwise. The ﬁnal value of Ydis determined probabilistically by mean ﬁeld inference.
All latent Ydvariables are added to the CRF which previously only contained Xivariables.
Let each (Xd,Yd), where{Xd}={Xi∈{X}|i∈Fd}, form a clique in the CRF. An assignment
(xd,yd)to the clique (Xd,Yd)has the energy:
ψDet
d(Xd=xd,Yd=yd) =

wlsd
|Fd|∑|Fd|
i=1[x(i)
d=ld]ifyd=0,
wlsd
|Fd|∑|Fd|
i=1[x(i)
d/negationslash=ld]ifyd=1,(2)
where x(i)
dis the ithelement in vector xd,[.]is the Iverson bracket and wlis a class-speciﬁc,
learnable weight parameter. This potential encourages consistency among X(i)
dvariables and
6 ARNAB AND TORR: BOTTOM-UP INSTANCE SEGMENTATION


key: chroma:document
string_value:
learnable weight parameter. This potential encourages consistency among X(i)
dvariables and
6 ARNAB AND TORR: BOTTOM-UP INSTANCE SEGMENTATION
Ydsince it encourages X(i)
dvariables to take the label ldwhen Ydis 1, and also encourages Yd
to be 0 when many X(i)
dvariables do not take the label ld.
A unary potential for the latent Yvariables is also included, which is initialised to the
conﬁdence score sdof the object detector (in addition to the joint potential with the Xvariables). This initial conﬁdence then changes throughout CRF inference, depending on the
other potentials. The ﬁnal probability of the latent Yvariable can thus be viewed as a recalibrated detection score, which gets adjusted during inference based on how much it agrees
with the segmentation unaries and pairwise potentials. Note that this “recalibration property”
of the latent Yvariables was not considered by [2]. The ﬁnal form of the energy function
(Eq 1), is thus
E(x) =∑
iψU
i(xi)+∑
i<jψP
i,j(xi,xj)+∑
dψDet


key: chroma:document
string_value:
iψU
i(xi)+∑
i<jψP
i,j(xi,xj)+∑
dψDet
d(xd,yd)+∑
dψU
d(yd). (3)
3.3 Instance Identiﬁcation and Reﬁnement
Once we have a category-level segmentation of the image, each pixel still needs to be assigned to an object instance. We assume that each object detection represents a possible
instance. Since there are Dobject detections (where Dvaries for every image), and some
pixels do not belong to any object instance, but are part of the background, we have a labelling problem involving D+1 labels. Our set of labels, D, is thus{1,2,...D+1}.
A naïve solution to the problem of recovering instances from a category-level segmentation would be as follows: If a pixel falls within the bounding box of a detection, and its
category label agrees with the class predicted by the object detector, then it is assigned to the
instance represented by that detection. However, this method cannot deal with overlapping
bounding boxes which are typical when objects occlude one another (as in Figs. 1 and 2).
Instead, if a pixel falls within the bounding box of a detection, we assign the pixel to


key: chroma:document
string_value:
Instead, if a pixel falls within the bounding box of a detection, we assign the pixel to
that instance with a probability proportional to the rescored detection (obtained from the
probability of the latent Yvariable after inference) and the semantic segmentation conﬁdence
for that class, as shown in Equation 4
Pr(vi=k) =/braceleftBigg1
Z(Y,Q)Qi(lk)Pr(Yk=1)ifi∈Bk
0 otherwise .(4)
Here, viis a multinomial random variable indicating the “identiﬁed instance” at pixel iand
takes on labels from D,Qi(l)is the output of the initial category-level segmentation stage
of our network and denotes the probability of pixel itaking the label l∈L, and Z(Y,Q)
normalises the probability in case there are multiple bounding boxes overlapping the same
pixel. For this formulation to be valid, we also add another detection, d0, denoting background pixels that are not overlapped by any detection. This then acts as the unary potentials
of another CRF with the energy:
E(v) =∑
iψU
i(vi)+∑
i<jψP


key: chroma:document
string_value:
of another CRF with the energy:
E(v) =∑
iψU
i(vi)+∑
i<jψP
i,j(vi,vj) ψU
i(vi) =−lnPr(vi). (5)
This Instance CRF contains unary, ψU
i(vi), and densely-connected pairwise terms, ψP
i,j(vi,vj),
encouraging appearance and spatial consistency [18]. These priors are valid in the case of
instance segmentation as well. We then perform mean ﬁeld inference on this ﬁnal Instance
CRF to get our ﬁnal output. Note that this ﬁnal, pairwise CRF is dynamic – the D+1 labels
ARNAB AND TORR: BOTTOM-UP INSTANCE SEGMENTATION 7
that it deals with are different for every image, where Dis equal to the number of object detections obtained after non-maximal suppression. This differs with the CRFs considered in
semantic segmentation literature which always have a ﬁxed number of labels for all images


key: chroma:document
string_value:
semantic segmentation literature which always have a ﬁxed number of labels for all images
in the dataset [17, 18, 35, 37]. Zheng et al. [37] showed how the iterative mean ﬁeld inference algorithm can be unrolled into a series of differentiable operations and be implemented
as a recurrent neural network. We employ a similar method so that our dynamic CRF is fully
differentiable and can thus be incorporated into a neural network. This is achieved by using
CRF weights that are not class-speciﬁc, and can thus generalise to any number of labels per
image. A label here refers to an instance associated with a detection and has no semantic
meaning. For example, in Fig. 1, label “1” was associated with a dining table, whilst in Fig.
2 it is a sheep. It therefore does not make sense to have class-speciﬁc weights in any case.
This approach to instance segmentation is bottom-up, in the sense that we ﬁrst perform
category-level segmentation of each pixel before predicting instances. However, the fact
that we used object detection cues in performing our semantic segmentation helps us to
subsequently reason about instances. Moreover, the fact that we used results from both
semantic segmentation and object detection makes sense since instance segmentation can be


key: chroma:document
string_value:
subsequently reason about instances. Moreover, the fact that we used results from both
semantic segmentation and object detection makes sense since instance segmentation can be
regarded as being at the intersection of these two more common problems.
4 Experimental Results and Analysis
We ﬁrst describe the dataset and our experimental setup (§4.1), before examining the efﬁcacy
of our detection potentials (§4.2) and comparing our approach with other recent instance
segmentation methods (§4.3).
4.1 Experimental Setup
Following previous work, we evaluate our instance segmentation performance on the PASCAL VOC 2012 [8] validation set which comprises of 1449 images with high-quality annotations. There is no test server for instance segmentation to evaluate on the test set. We
use the standard APrmeasure for evaluation [13], which computes the mean average precision under different Intersection over Union (IoU) scores between the predicted and ground
truth segmentations (rather than IoU between bounding boxes, as in object detection). In object detection, a predicted bounding box is considered correct if the IoU between it and the
ground-truth bounding box is greater than 0.5. Here, we consider different overlap thresholds
between the predicted and ground-truth segments, since higher overlap thresholds require the


key: chroma:document
string_value:
ground-truth bounding box is greater than 0.5. Here, we consider different overlap thresholds
between the predicted and ground-truth segments, since higher overlap thresholds require the
segmentations to be more precise and are therefore better indicators of segmentation performance. We also quote the APr
volmeasure, which is the mean APrscore for overlap thresholds
ranging from 0 .1 to 0 .9 in increments of 0 .1 [13]. The detailed annotations in the VOC 2012
dataset allows us to reliably evaluate the AProf our method at high overlap thresholds.
We ﬁrst trained a network for semantic segmentation, in a similar manner to [27] and
[37]. Although the trained models from these two authors have been made publicly available,
we could not use them since they had been trained on part of the VOC validation set. We
ﬁrst trained a fully convolutional network [27] (ﬁnetuned from VGG-16 trained on ImageNet
[32]) using VOC 2012 training data, augmented with images from the Semantic Boundaries
Dataset [13] which do not overlap with the VOC Validation set, and the Microsoft COCO
dataset [26].


key: chroma:document
string_value:
Dataset [13] which do not overlap with the VOC Validation set, and the Microsoft COCO
dataset [26].
To this pretrained network, we added the Higher Order CRF, and ﬁnetuned with a learning rate of 10−11using only VOC 2012 data which is ﬁnely annotated. The learning rate is
8 ARNAB AND TORR: BOTTOM-UP INSTANCE SEGMENTATION
Table 1: Comparison of instance segmentation performance against baselines on the VOC
2012 Validation Set of 1449 images. We report the APrmeasure at different IoU thresholds.
Method APrat 0.5 APrat 0.6 APrat 0.7 APr
vol
Baseline without detection potentials 54.6 48.5 41.8 50.0
Baseline with detection potentials,
butYvariables ignored57.5 51.6 44.5 52.4
Full system with detection potentials,
and recalibrated detection scores from
Yvariables58.3 52.4 45.4 53.1
low since the loss was not normalised by the number of pixels in the training image. When
training our Higher Order CRF, we use the publicly available Faster-RCNN object detection
framework [28]. The semantic segmentation performance on the VOC validation set is a


key: chroma:document
string_value:
training our Higher Order CRF, we use the publicly available Faster-RCNN object detection
framework [28]. The semantic segmentation performance on the VOC validation set is a
mean IoU of 73.4% when using only pairwise potentials in the CRF. This rises to 75.3%
when using detection potentials as well. The authors of [2] observed a similar increase when
using detection potentials.
4.2 Effect of detection potentials
To analyse the effect of our detection potentials, we evaluate our system by disabling the
higher order potentials in our Semantic Segmentation CRF. Because our detection potentials
are disabled, we can no longer use the output of the latent Yvariables as our recalibrated
detection score, and use the original detector’s conﬁdence score instead. As shown in Table
1, the result is that the APrat 0.5 is 3.7% lower and the APr
volis 3.1% lower.
The detection potentials improve instance segmentation performance as they improve
both our initial semantic segmentation, and also recalibrate the detector’s initial scores. To
decouple these two effects, we then include detection potentials in our Segmentation CRF,
but ignore the latent Yvariables output by the CRF and use the detector’s original conﬁdence


key: chroma:document
string_value:
but ignore the latent Yvariables output by the CRF and use the detector’s original conﬁdence
score instead as the input to the instance segmentation network. The second row of Table
1 shows that this second baseline has an APrat 0.5 and an APr
volthat are respectively 0.8%
and 0.7% lower than our ﬁnal method. The difference between the ﬁnal method and this
second baseline is due to the score recalibration performed by the detection potentials. The
two baselines differ in performance since the second baseline uses a semantic segmentation
(mean IoU of 75.3%) which is better than the other (73.4%) due to its use of detection
potentials. The large gap in their overall instance segmentation performance emphasises
the importance of a good initial semantic segmentation for our method, and underlines the
beneﬁts of a bottom-up approach to instance segmentation.
From the difference in performance between the baseline that does not fully make use of
detection potentials (Row 1 of Table 1), and our ﬁnal method (Row 3 of Table 1), we can
conclude that our higher order detection potentials are an integral part of our system.
4.3 Comparison to other current methods


key: chroma:document
string_value:
conclude that our higher order detection potentials are an integral part of our system.
4.3 Comparison to other current methods
In Table 2, we compare our methods to other approaches that have been evaluated on the
same dataset and reported APrresults at multiple IoU thresholds. The SDS method of [13]
was evaluated on the VOC Validation set by [4]. Our approach signiﬁcantly outperforms
ARNAB AND TORR: BOTTOM-UP INSTANCE SEGMENTATION 9
Table 2: Comparison of instance segmentation performance to other methods on the VOC
2012 Validation Set. We report the APrmeasure at ﬁve different IoU thresholds.
Method APrat 0.5 APrat 0.6 APrat 0.7 APrat 0.8 APrat 0.9 APr
vol
SDS [13] 43.8 34.5 21.3 8.7 0.9 Chen et al. [4] 46.3 38.2 27.0 13.5 2.6 PFN [24] 58.7 51.3 42.5 31.2 15.7 52.3
Ours 58.3 52.4 45.4 34.9 20.1 53.1
both [13] and [4] which initially perform object detection, and reﬁne these detections into an


key: chroma:document
string_value:
both [13] and [4] which initially perform object detection, and reﬁne these detections into an
instance segmentation. In fact, even our baselines in Table 1 outperform them. The model
of [24] differs from the other two works in that the ﬁrst stage of their pipeline is to also
perform a category-level segmentation of the image. However, their method does not make
use of any information from object detectors, and our approach performs better, particularly
at high IoU thresholds where the difference in APrat an IoU threshold of 0.9 is 4.4%. This
indicates that our instance segmentations are more precise. The fact that our APr
volis only
0.8% greater, despite our APrbeing signiﬁcantly larger at high thresholds suggests that the
method of [24] has a higher APrat lower thresholds (below 0.5, which were not reported by
[24]). This indicates that their method identiﬁes more instances than ours, but the fact that
their APrat higher thresholds is lower than ours also suggests that their segmentations are
not as accurate.
Some success and failure cases of our method are shown in Figure 3. Our method has
trouble differentiating instances which are occluded and also very visually similar to each


key: chroma:document
string_value:
Some success and failure cases of our method are shown in Figure 3. Our method has
trouble differentiating instances which are occluded and also very visually similar to each
other (Fig. 3, last two rows). The supplementary material includes a more detailed results
table, and more examples of success and failure cases.
5 Conclusion
We have presented a simple method that effectively leverages state-of-the-art semantic segmentation and object detection networks to perform the less widely studied task of instance
segmentation. Our approach begins with a bottom-up semantic segmentation of the input
image, but is able to reason about instances since the ﬁnal CRF layer in our semantic segmentation network incorporates information from an object detector in the form of a higherorder, detection potential. Our method produces state-of-the-art instance segmentation results, achieving considerable improvements over existing work at high APrthresholds. Our
ﬁnal Instance CRF – which is dynamic in its number of labels – is fully differentiable. This
means that our neural network can be dynamically instantiated for every input image and
trained end-to-end.
Acknowledgements
We thank Bernardino Romera-Paredes, Stuart Golodetz and Shuai Zheng for insightful discussions and feedback. This work was supported by the EPSRC, Clarendon Fund, ERC


key: chroma:document
string_value:
We thank Bernardino Romera-Paredes, Stuart Golodetz and Shuai Zheng for insightful discussions and feedback. This work was supported by the EPSRC, Clarendon Fund, ERC
grant ERC-2012-AdG 321162-HELIOS, EPRSRC grant Seebibyte EP/M013774/1 and EPSRC/MURI grant EP/N019474/1.
10 ARNAB AND TORR: BOTTOM-UP INSTANCE SEGMENTATION
Input image Object Segmentation Instance Segmentation Ground truth
Figure 3: Some results of our system. Detector outputs are overlayed on the input image. Top
row: An example where our method performs well, handling occlusions and false-positive
detections. Second row: Our method performs well on this easy example which has no
occluding objects of the same class. The naïve method described in §3.3 would have produced the same result. Third row: A failure case where the system is not able to differentiate
visually similar instances. Bottom row: Heavily occluded objects are difﬁcult to identify.
References
[1] Pablo Arbelaez, Jordi Pont-Tuset, Jon Barron, Ferran Marques, and Jitendra Malik.


key: chroma:document
string_value:
References
[1] Pablo Arbelaez, Jordi Pont-Tuset, Jon Barron, Ferran Marques, and Jitendra Malik.
Multiscale combinatorial grouping. In CVPR , pages 328–335. IEEE, 2014.
[2] Anurag Arnab, Sadeep Jayasumana, Shuai Zheng, and Philip H. S. Torr. Higher order
conditional random ﬁelds in deep neural networks. In ECCV , 2016.
[3] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L
Yuille. Semantic image segmentation with deep convolutional nets and fully connected
crfs. ICLR , 2015.
[4] Yi-Ting Chen, Xiaokai Liu, and Ming-Hsuan Yang. Multi-instance object segmentation
with occlusion handling. In CVPR , pages 3470–3478, 2015.
[5] Jifeng Dai, Kaiming He, and Jian Sun. Convolutional feature masking for joint object
and stuff segmentation. In CVPR , 2015.
[6] Jifeng Dai, Kaiming He, and Jian Sun. Boxsup: Exploiting bounding boxes to supervise
convolutional networks for semantic segmentation. In ICCV , 2015.


key: chroma:document
string_value:
convolutional networks for semantic segmentation. In ICCV , 2015.
ARNAB AND TORR: BOTTOM-UP INSTANCE SEGMENTATION 11
[7] Jifeng Dai, Kaiming He, and Jian Sun. Instance-aware semantic segmentation via
multi-task network cascades. In CVPR , 2016.
[8] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew
Zisserman. The pascal visual object classes (voc) challenge. IJCV , 2010.
[9] Pedro Felzenszwalb, David McAllester, and Deva Ramanan. A discriminatively
trained, multiscale, deformable part model. In CVPR , pages 1–8. IEEE, 2008.
[10] Spyros Gidaris and Nikos Komodakis. Object detection via a multi-region and semantic
segmentation-aware cnn model. In ICCV , pages 1134–1142, 2015.
[11] Ross Girshick. Fast r-cnn. In ICCV , 2015.
[12] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In CVPR , 2014.


key: chroma:document
string_value:
[13] Bharath Hariharan, Pablo Arbeláez, Ross Girshick, and Jitendra Malik. Simultaneous
detection and segmentation. In ECCV , pages 297–312. Springer, 2014.
[14] Bharath Hariharan, Pablo Arbeláez, Ross Girshick, and Jitendra Malik. Hypercolumns
for object segmentation and ﬁne-grained localization. CVPR , 2015.
[15] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Spatial pyramid pooling in
deep convolutional networks for visual recognition. In ECCV , 2014.
[16] Xuming He, Richard S Zemel, and Miguel Á Carreira-Perpiñán. Multiscale conditional
random ﬁelds for image labeling. In CVPR , volume 2, pages II–695. IEEE, 2004.
[17] Pushmeet Kohli, Lubor Ladicky, and Philip H.S. Torr. Robust higher order potentials
for enforcing label consistency. IJCV , 82(3):302–324, 2009.
[18] P. Krähenbühl and V . Koltun. Efﬁcient inference in fully connected CRFs with Gaussian edge potentials. In NIPS , 2011.


key: chroma:document
string_value:
[19] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classiﬁcation with
deep convolutional neural networks. In NIPS , pages 1097–1105. 2012.
[20] Lubor Ladicky, Chris Russell, Pushmeet Kohli, and Philip HS Torr. Graph cut based
inference with co-occurrence statistics. In ECCV , pages 239–253. 2010.
[21] L’ubor Ladický, Paul Sturgess, Karteek Alahari, Chris Russell, and Philip H. S. Torr.
What, where and how many? combining object detectors and crfs. In ECCV , pages
424–437, 2010.
[22] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278–2324,
1998.
[23] Xiaodan Liang, Yunchao Wei, Xiaohui Shen, Zequn Jie, Jiashi Feng, Liang Lin, and
Shuicheng Yan. Reversible recursive instance-level object segmentation. arXiv preprint
arXiv:1511.04517 , 2015.


key: chroma:document
string_value:
Shuicheng Yan. Reversible recursive instance-level object segmentation. arXiv preprint
arXiv:1511.04517 , 2015.
12 ARNAB AND TORR: BOTTOM-UP INSTANCE SEGMENTATION
[24] Xiaodan Liang, Yunchao Wei, Xiaohui Shen, Jianchao Yang, Liang Lin, and Shuicheng
Yan. Proposal-free network for instance-level object segmentation. arXiv preprint
arXiv:1509.02636 , 2015.
[25] Guosheng Lin, Chunhua Shen, Ian Reid, et al. Efﬁcient piecewise training of deep
structured models for semantic segmentation. In CVPR , 2016.
[26] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in
context. In ECCV , pages 740–755. Springer, 2014.
[27] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for
semantic segmentation. In CVPR , 2015.
[28] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards


key: chroma:document
string_value:
[28] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards
real-time object detection with region proposal networks. In NIPS , 2015.
[29] Bernardino Romera-Paredes and Philip HS Torr. Recurrent instance segmentation. In
ECCV , 2016.
[30] Carsten Rother, Vladimir Kolmogorov, and Andrew Blake. Grabcut: Interactive foreground extraction using iterated graph cuts. ACM TOG , 2004.
[31] Jamie Shotton, John Winn, Carsten Rother, and Antonio Criminisi. Textonboost for
image understanding: Multi-class object recognition and segmentation by jointly modeling texture, layout, and context. IJCV , 2009.
[32] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for largescale image recognition. In ICLR , 2015.
[33] Min Sun, Byung-soo Kim, Pushmeet Kohli, and Silvio Savarese. Relating things and
stuff via object property interactions. PAMI , 36(7):1370–1383, 2014.
[34] Jasper RR Uijlings, Koen EA van de Sande, Theo Gevers, and Arnold WM Smeulders.


key: chroma:document
string_value:
[34] Jasper RR Uijlings, Koen EA van de Sande, Theo Gevers, and Arnold WM Smeulders.
Selective search for object recognition. IJCV , 104(2):154–171, 2013.
[35] Vibhav Vineet, Jonathan Warrell, and Philip HS Torr. Filter-based mean-ﬁeld inference
for random ﬁelds with higher-order terms and product label-spaces. IJCV , 2014.
[36] Jian Yao, Sanja Fidler, and Raquel Urtasun. Describing the Scene as a Whole: Joint
Object Detection, Scene Classiﬁcation and Semantic Segmentation. In CVPR , pages
702–709, 2012.
[37] Shuai Zheng, Sadeep Jayasumana, Bernardino Romera-Paredes, Vibhav Vineet,
Zhizhong Su, Dalong Du, Chang Huang, and Philip Torr. Conditional random ﬁelds as
recurrent neural networks. In ICCV , 2015.


key: chroma:document
string_value:
Aggregated Residual Transformations for Deep Neural Networks
Saining Xie1Ross Girshick2Piotr Doll ´ar2Zhuowen Tu1Kaiming He2
1UC San Diego2Facebook AI Research
fs9xie,ztug@ucsd.edu frbg,pdollar,kaiminghe g@fb.com
Abstract
We present a simple, highly modularized network architecture for image classiﬁcation. Our network is constructed
by repeating a building block that aggregates a set of transformations with the same topology. Our simple design results in a homogeneous, multi-branch architecture that has
only a few hyper-parameters to set. This strategy exposes a
new dimension, which we call “cardinality” (the size of the
set of transformations), as an essential factor in addition to
the dimensions of depth and width. On the ImageNet-1K
dataset, we empirically show that even under the restricted
condition of maintaining complexity, increasing cardinality
is able to improve classiﬁcation accuracy. Moreover, increasing cardinality is more effective than going deeper or
wider when we increase the capacity. Our models, named
ResNeXt, are the foundations of our entry to the ILSVRC
2016 classiﬁcation task in which we secured 2nd place.


key: chroma:document
string_value:
ResNeXt, are the foundations of our entry to the ILSVRC
2016 classiﬁcation task in which we secured 2nd place.
We further investigate ResNeXt on an ImageNet-5K set and
the COCO detection set, also showing better results than
its ResNet counterpart. The code and models are publicly
available online1.
1. Introduction
Research on visual recognition is undergoing a transition from “feature engineering” to “network engineering”
[25, 24, 44, 34, 36, 38, 14]. In contrast to traditional handdesigned features ( e.g., SIFT [29] and HOG [5]), features
learned by neural networks from large-scale data [33] require minimal human involvement during training, and can
be transferred to a variety of recognition tasks [7, 10, 28].
Nevertheless, human effort has been shifted to designing
better network architectures for learning representations.
Designing architectures becomes increasingly difﬁcult
with the growing number of hyper-parameters (width2, ﬁlter sizes, strides, etc.), especially when there are many layers. The VGG-nets [36] exhibit a simple yet effective strategy of constructing very deep networks: stacking build1https://github.com/facebookresearch/ResNeXt
2Width refers to the number of channels in a layer.


key: chroma:document
string_value:
2Width refers to the number of channels in a layer.
256, 1x1, 4
4, 3x3, 4
4, 1x1, 256
+256, 1x1, 4
4, 3x3, 4
4, 1x1, 256256, 1x1, 4
4, 3x3, 4
4, 1x1, 256....total 32
paths256-d in
+256, 1x1, 64
64, 3x3, 64
64, 1x1, 256
+256-d in
256-d out
256-d outFigure 1. Left: A block of ResNet [14]. Right : A block of
ResNeXt with cardinality = 32 , with roughly the same complexity. A layer is shown as (# in channels, ﬁlter size, # out channels).
ing blocks of the same shape. This strategy is inherited
by ResNets [14] which stack modules of the same topology. This simple rule reduces the free choices of hyperparameters, and depth is exposed as an essential dimension
in neural networks. Moreover, we argue that the simplicity


key: chroma:document
string_value:
in neural networks. Moreover, we argue that the simplicity
of this rule may reduce the risk of over-adapting the hyperparameters to a speciﬁc dataset. The robustness of VGGnets and ResNets has been proven by various visual recognition tasks [7, 10, 9, 28, 31, 14] and by non-visual tasks
involving speech [42, 30] and language [4, 41, 20].
Unlike VGG-nets, the family of Inception models [38,
17, 39, 37] have demonstrated that carefully designed
topologies are able to achieve compelling accuracy with low
theoretical complexity. The Inception models have evolved
over time [38, 39], but an important common property is
asplit-transform-merge strategy. In an Inception module,
the input is split into a few lower-dimensional embeddings
(by 11 convolutions), transformed by a set of specialized
ﬁlters (33, 55,etc.), and merged by concatenation. It
can be shown that the solution space of this architecture is a
strict subspace of the solution space of a single large layer
(e.g., 55) operating on a high-dimensional embedding.
The split-transform-merge behavior of Inception modules


key: chroma:document
string_value:
(e.g., 55) operating on a high-dimensional embedding.
The split-transform-merge behavior of Inception modules
is expected to approach the representational power of large
and dense layers, but at a considerably lower computational
complexity.
Despite good accuracy, the realization of Inception models has been accompanied with a series of complicating fac1arXiv:1611.05431v2  [cs.CV]  11 Apr 2017
tors — the ﬁlter numbers and sizes are tailored for each
individual transformation, and the modules are customized
stage-by-stage. Although careful combinations of these
components yield excellent neural network recipes, it is in
general unclear how to adapt the Inception architectures to
new datasets/tasks, especially when there are many factors
and hyper-parameters to be designed.
In this paper, we present a simple architecture which
adopts VGG/ResNets’ strategy of repeating layers, while
exploiting the split-transform-merge strategy in an easy, extensible way. A module in our network performs a set
of transformations, each on a low-dimensional embedding,
whose outputs are aggregated by summation. We pursuit a
simple realization of this idea — the transformations to be
aggregated are all of the same topology ( e.g., Fig. 1 (right)).


key: chroma:document
string_value:
simple realization of this idea — the transformations to be
aggregated are all of the same topology ( e.g., Fig. 1 (right)).
This design allows us to extend to any large number of
transformations without specialized designs.
Interestingly, under this simpliﬁed situation we show that
our model has two other equivalent forms (Fig. 3). The reformulation in Fig. 3(b) appears similar to the InceptionResNet module [37] in that it concatenates multiple paths;
but our module differs from all existing Inception modules
in that all our paths share the same topology and thus the
number of paths can be easily isolated as a factor to be investigated. In a more succinct reformulation, our module
can be reshaped by Krizhevsky et al .’s grouped convolutions [24] (Fig. 3(c)), which, however, had been developed
as an engineering compromise.
We empirically demonstrate that our aggregated transformations outperform the original ResNet module, even
under the restricted condition of maintaining computational
complexity and model size — e.g., Fig. 1(right) is designed
to keep the FLOPs complexity and number of parameters of
Fig. 1(left). We emphasize that while it is relatively easy to
increase accuracy by increasing capacity (going deeper or
wider), methods that increase accuracy while maintaining


key: chroma:document
string_value:
Fig. 1(left). We emphasize that while it is relatively easy to
increase accuracy by increasing capacity (going deeper or
wider), methods that increase accuracy while maintaining
(or reducing) complexity are rare in the literature.
Our method indicates that cardinality (the size of the
set of transformations) is a concrete, measurable dimension that is of central importance, in addition to the dimensions of width and depth. Experiments demonstrate that increasing cardinality is a more effective way of gaining accuracy than going deeper or wider , especially when depth and
width starts to give diminishing returns for existing models.
Our neural networks, named ResNeXt (suggesting the
nextdimension), outperform ResNet-101/152 [14], ResNet200 [15], Inception-v3 [39], and Inception-ResNet-v2 [37]
on the ImageNet classiﬁcation dataset. In particular, a
101-layer ResNeXt is able to achieve better accuracy than
ResNet-200 [15] but has only 50% complexity. Moreover,
ResNeXt exhibits considerably simpler designs than all Inception models. ResNeXt was the foundation of our submission to the ILSVRC 2016 classiﬁcation task, in whichwe secured second place. This paper further evaluates
ResNeXt on a larger ImageNet-5K set and the COCO object


key: chroma:document
string_value:
ResNeXt on a larger ImageNet-5K set and the COCO object
detection dataset [27], showing consistently better accuracy
than its ResNet counterparts. We expect that ResNeXt will
also generalize well to other visual (and non-visual) recognition tasks.
2. Related Work
Multi-branch convolutional networks . The Inception
models [38, 17, 39, 37] are successful multi-branch architectures where each branch is carefully customized.
ResNets [14] can be thought of as two-branch networks
where one branch is the identity mapping. Deep neural decision forests [22] are tree-patterned multi-branch networks
with learned splitting functions.
Grouped convolutions . The use of grouped convolutions
dates back to the AlexNet paper [24], if not earlier. The
motivation given by Krizhevsky et al. [24] is for distributing
the model over two GPUs. Grouped convolutions are supported by Caffe [19], Torch [3], and other libraries, mainly
for compatibility of AlexNet. To the best of our knowledge,
there has been little evidence on exploiting grouped convolutions to improve accuracy. A special case of grouped convolutions is channel-wise convolutions in which the number
of groups is equal to the number of channels. Channel-wise


key: chroma:document
string_value:
of groups is equal to the number of channels. Channel-wise
convolutions are part of the separable convolutions in [35].
Compressing convolutional networks . Decomposition (at
spatial [6, 18] and/or channel [6, 21, 16] level) is a widely
adopted technique to reduce redundancy of deep convolutional networks and accelerate/compress them. Ioannouet al . [16] present a “root”-patterned network for reducing computation, and branches in the root are realized
by grouped convolutions. These methods [6, 18, 21, 16]
have shown elegant compromise of accuracy with lower
complexity and smaller model sizes. Instead of compression, our method is an architecture that empirically shows
stronger representational power.
Ensembling . Averaging a set of independently trained networks is an effective solution to improving accuracy [24],
widely adopted in recognition competitions [33]. Veit et al.
[40] interpret a single ResNet as an ensemble of shallower
networks, which results from ResNet’s additive behaviors
[15]. Our method harnesses additions to aggregate a set of
transformations. But we argue that it is imprecise to view
our method as ensembling, because the members to be aggregated are trained jointly, not independently.
3. Method
3.1. Template


key: chroma:document
string_value:
our method as ensembling, because the members to be aggregated are trained jointly, not independently.
3. Method
3.1. Template
We adopt a highly modularized design following
VGG/ResNets. Our network consists of a stack of residstage output ResNet-50 ResNeXt-50 (32 4d)
conv1 112112 77, 64, stride 2 77, 64, stride 2
conv2 565633 max pool, stride 2 33 max pool, stride 22
6411, 64
33, 64
11, 2563
7532
6411, 128
33, 128, C=32
11, 2563
753
conv3 28282
6411, 128
33, 128
11, 5123
7542
6411, 256
33, 256, C=32
11, 5123
754
conv4 14142
6411, 256
33, 256
11, 10243
7562
6411, 512
33, 512, C=32
11, 10243
756
conv5 772
6411, 512
33, 512


key: chroma:document
string_value:
33, 512, C=32
11, 10243
756
conv5 772
6411, 512
33, 512
11, 20483
7532
6411, 1024
33, 1024, C=32
11, 20483
753
11global average pool global average pool
1000-d fc, softmax 1000-d fc, softmax
# params. 25.510625.0106
FLOPs 4.11094.2109
Table 1. ( Left) ResNet-50. ( Right ) ResNeXt-50 with a 32 4d
template (using the reformulation in Fig. 3(c)). Inside the brackets
are the shape of a residual block, and outside the brackets is the
number of stacked blocks on a stage. “ C=32” suggests grouped
convolutions [24] with 32 groups. The numbers of parameters and
FLOPs are similar between these two models.
ual blocks. These blocks have the same topology, and are
subject to two simple rules inspired by VGG/ResNets: (i)
if producing spatial maps of the same size, the blocks share


key: chroma:document
string_value:
subject to two simple rules inspired by VGG/ResNets: (i)
if producing spatial maps of the same size, the blocks share
the same hyper-parameters (width and ﬁlter sizes), and (ii)
each time when the spatial map is downsampled by a factor of 2, the width of the blocks is multiplied by a factor
of 2. The second rule ensures that the computational complexity, in terms of FLOPs (ﬂoating-point operations, in #
of multiply-adds), is roughly the same for all blocks.
With these two rules, we only need to design a template
module, and all modules in a network can be determined
accordingly. So these two rules greatly narrow down the
design space and allow us to focus on a few key factors.
The networks constructed by these rules are in Table 1.
3.2. Revisiting Simple Neurons
The simplest neurons in artiﬁcial neural networks perform inner product (weighted sum), which is the elementary transformation done by fully-connected and convolutional layers. Inner product can be thought of as a form of
aggregating transformation:
DX
i=1wixi; (1)
where x= [x1; x2; :::; x D]is aD-channel input vector to


key: chroma:document
string_value:
DX
i=1wixi; (1)
where x= [x1; x2; :::; x D]is aD-channel input vector to
the neuron and wiis a ﬁlter’s weight for the i-th chan.......
+x1 x2 xD x3
w1w2w3wDxFigure 2. A simple neuron that performs inner product.
nel. This operation (usually including some output nonlinearity) is referred to as a “neuron”. See Fig. 2.
The above operation can be recast as a combination of
splitting, transforming, and aggregating . (i) Splitting : the
vector xis sliced as a low-dimensional embedding, and
in the above, it is a single-dimension subspace xi. (ii)
Transforming : the low-dimensional representation is transformed, and in the above, it is simply scaled: wixi. (iii)
Aggregating : the transformations in all embeddings are aggregated byPD
i=1.
3.3. Aggregated Transformations
Given the above analysis of a simple neuron, we consider replacing the elementary transformation ( wixi) with
a more generic function, which in itself can also be a network. In contrast to “Network-in-Network” [26] that turns


key: chroma:document
string_value:
a more generic function, which in itself can also be a network. In contrast to “Network-in-Network” [26] that turns
out to increase the dimension of depth, we show that our
“Network-in- Neuron ” expands along a new dimension.
Formally, we present aggregated transformations as:
F(x) =CX
i=1Ti(x); (2)
whereTi(x)can be an arbitrary function. Analogous to a
simple neuron,Tishould project xinto an (optionally lowdimensional) embedding and then transform it.
In Eqn.(2), Cis the size of the set of transformations
to be aggregated. We refer to Cascardinality [2]. In
Eqn.(2) Cis in a position similar to Din Eqn.(1), but C
need not equal Dand can be an arbitrary number. While
the dimension of width is related to the number of simple
transformations (inner product), we argue that the dimension of cardinality controls the number of more complex
transformations. We show by experiments that cardinality
is an essential dimension and can be more effective than the
dimensions of width and depth.
In this paper, we consider a simple way of designing the
transformation functions: all Ti’s have the same topology.


key: chroma:document
string_value:
dimensions of width and depth.
In this paper, we consider a simple way of designing the
transformation functions: all Ti’s have the same topology.
This extends the VGG-style strategy of repeating layers of
the same shape, which is helpful for isolating a few factors
and extending to any large number of transformations. We
set the individual transformation Tito be the bottleneckshaped architecture [14], as illustrated in Fig. 1 (right). In
this case, the ﬁrst 1 1 layer in eachTiproduces the lowdimensional embedding.
equivalent
256, 1x1, 4
4, 3x3, 4
4, 1x1, 256
+256, 1x1, 4
4, 3x3, 4
4, 1x1, 256256, 1x1, 4
4, 3x3, 4
4, 1x1, 256
(a)....total 32
paths256-d in
+256, 1x1, 4
4, 3x3, 4256, 1x1, 4
4, 3x3, 4256, 1x1, 4
4, 3x3, 4
+concatenate
128, 1x1, 256256-d in
....total 32
paths
(b)256, 1x1, 128


key: chroma:document
string_value:
+concatenate
128, 1x1, 256256-d in
....total 32
paths
(b)256, 1x1, 128
128, 3x3, 128
group = 32
128, 1x1, 256
+256-d in
(c)256-d out 256-d out 256-d outFigure 3. Equivalent building blocks of ResNeXt. (a): Aggregated residual transformations, the same as Fig. 1 right. (b): A block equivalent
to (a), implemented as early concatenation. (c): A block equivalent to (a,b), implemented as grouped convolutions [24]. Notations in bold
text highlight the reformulation changes. A layer is denoted as (# input channels, ﬁlter size, # output channels).
The aggregated transformation in Eqn.(2) serves as the
residual function [14] (Fig. 1 right):
y=x+CX
i=1Ti(x); (3)
where yis the output.
Relation to Inception-ResNet . Some tensor manipulations show that the module in Fig. 1(right) (also shown in


key: chroma:document
string_value:
where yis the output.
Relation to Inception-ResNet . Some tensor manipulations show that the module in Fig. 1(right) (also shown in
Fig. 3(a)) is equivalent to Fig. 3(b).3Fig. 3(b) appears similar to the Inception-ResNet [37] block in that it involves
branching and concatenating in the residual function. But
unlike all Inception or Inception-ResNet modules, we share
the same topology among the multiple paths. Our module
requires minimal extra effort designing each path.
Relation to Grouped Convolutions . The above module becomes more succinct using the notation of grouped convolutions [24].4This reformulation is illustrated in Fig. 3(c).
All the low-dimensional embeddings (the ﬁrst 1 1 layers)
can be replaced by a single, wider layer ( e.g., 11, 128-d
in Fig 3(c)). Splitting is essentially done by the grouped
convolutional layer when it divides its input channels into
groups. The grouped convolutional layer in Fig. 3(c) performs 32 groups of convolutions whose input and output
channels are 4-dimensional. The grouped convolutional
layer concatenates them as the outputs of the layer. The


key: chroma:document
string_value:
channels are 4-dimensional. The grouped convolutional
layer concatenates them as the outputs of the layer. The
block in Fig. 3(c) looks like the original bottleneck residual block in Fig. 1(left), except that Fig. 3(c) is a wider but
sparsely connected module.
3An informal but descriptive proof is as follows. Note the equality:
A1B1+A2B2= [A1; A2][B1;B2]where [;]is horizontal concatenation and [ ; ]is vertical concatenation. Let Aibe the weight of the last layer
andBibe the output response of the second-last layer in the block. In the
case of C= 2, the element-wise addition in Fig. 3(a) is A1B1+A2B2,
the weight of the last layer in Fig. 3(b) is [A1; A2], and the concatenation
of outputs of second-last layers in Fig. 3(b) is [B1;B2].
4In a group conv layer [24], input and output channels are divided into
Cgroups, and convolutions are separately performed within each group.
equivalent
64, 3x3, 4 64, 3x3, 4 64, 3x3, 4
....total 32
paths


key: chroma:document
string_value:
equivalent
64, 3x3, 4 64, 3x3, 4 64, 3x3, 4
....total 32
paths
+4, 3x3, 64
+4, 3x3, 64 4, 3x3, 6464, 3x3, 128
128, 3x3, 64
+Figure 4. ( Left): Aggregating transformations of depth = 2 .
(Right ): An equivalent block, which is trivially wider.
We note that the reformulations produce nontrivial
topologies only when the block has depth 3. If the block
has depth =2 (e.g., the basic block in [14]), the reformulations lead to trivially a wide, dense module. See the illustration in Fig. 4.
Discussion . We note that although we present reformulations that exhibit concatenation (Fig. 3(b)) or grouped convolutions (Fig. 3(c)), such reformulations are not always applicable for the general form of Eqn.(3), e.g., if the transformationTitakes arbitrary forms and are heterogenous.
We choose to use homogenous forms in this paper because
they are simpler and extensible. Under this simpliﬁed case,
grouped convolutions in the form of Fig. 3(c) are helpful for
easing implementation.
3.4. Model Capacity


key: chroma:document
string_value:
grouped convolutions in the form of Fig. 3(c) are helpful for
easing implementation.
3.4. Model Capacity
Our experiments in the next section will show that
our models improve accuracy when maintaining the model
complexity and number of parameters. This is not only interesting in practice, but more importantly, the complexity
and number of parameters represent inherent capacity of
models and thus are often investigated as fundamental properties of deep networks [8].
When we evaluate different cardinalities Cwhile preserving complexity, we want to minimize the modiﬁcation
of other hyper-parameters. We choose to adjust the width of
cardinality C 1 2 4 8 32
width of bottleneck d 64 40 24 14 4
width of group conv. 64 80 96 112 128
Table 2. Relations between cardinality and width (for the template
of conv2), with roughly preserved complexity on a residual block.
The number of parameters is 70k for the template of conv2. The
number of FLOPs is 0.22 billion (# params 5656 for conv2).
the bottleneck ( e.g., 4-d in Fig 1(right)), because it can be
isolated from the input and output of the block. This strategy introduces no change to other hyper-parameters (depth
or input/output width of blocks), so is helpful for us to focus


key: chroma:document
string_value:
isolated from the input and output of the block. This strategy introduces no change to other hyper-parameters (depth
or input/output width of blocks), so is helpful for us to focus
on the impact of cardinality.
In Fig. 1(left), the original ResNet bottleneck block [14]
has25664 + 336464 + 6425670k parameters and
proportional FLOPs (on the same feature map size). With
bottleneck width d, our template in Fig. 1(right) has:
C(256d+ 33dd+d256) (4)
parameters and proportional FLOPs. When C= 32 and
d= 4, Eqn.(4)70k. Table 2 shows the relationship between cardinality Cand bottleneck width d.
Because we adopt the two rules in Sec. 3.1, the above
approximate equality is valid between a ResNet bottleneck
block and our ResNeXt on all stages (except for the subsampling layers where the feature maps size changes). Table 1 compares the original ResNet-50 and our ResNeXt-50
that is of similar capacity.5We note that the complexity can
only be preserved approximately, but the difference of the
complexity is minor and does not bias our results.
4. Implementation details


key: chroma:document
string_value:
only be preserved approximately, but the difference of the
complexity is minor and does not bias our results.
4. Implementation details
Our implementation follows [14] and the publicly available code of fb.resnet.torch [11]. On the ImageNet
dataset, the input image is 224 224 randomly cropped
from a resized image using the scale and aspect ratio augmentation of [38] implemented by [11]. The shortcuts are
identity connections except for those increasing dimensions
which are projections (type B in [14]). Downsampling of
conv3, 4, and 5 is done by stride-2 convolutions in the 3 3
layer of the ﬁrst block in each stage, as suggested in [11].
We use SGD with a mini-batch size of 256 on 8 GPUs (32
per GPU). The weight decay is 0.0001 and the momentum
is 0.9. We start from a learning rate of 0.1, and divide it by
10 for three times using the schedule in [11]. We adopt the
weight initialization of [13]. In all ablation comparisons, we
evaluate the error on the single 224 224 center crop from
an image whose shorter side is 256.
Our models are realized by the form of Fig. 3(c). We


key: chroma:document
string_value:
evaluate the error on the single 224 224 center crop from
an image whose shorter side is 256.
Our models are realized by the form of Fig. 3(c). We
perform batch normalization (BN) [17] right after the con5The marginally smaller number of parameters and marginally higher
FLOPs are mainly caused by the blocks where the map sizes change.volutions in Fig. 3(c).6ReLU is performed right after each
BN, expect for the output of the block where ReLU is performed after the adding to the shortcut, following [14].
We note that the three forms in Fig. 3 are strictly equivalent, when BN and ReLU are appropriately addressed as
mentioned above. We have trained all three forms and
obtained the same results. We choose to implement by
Fig. 3(c) because it is more succinct and faster than the other
two forms.
5. Experiments
5.1. Experiments on ImageNet-1K
We conduct ablation experiments on the 1000-class ImageNet classiﬁcation task [33]. We follow [14] to construct
50-layer and 101-layer residual networks. We simply replace all blocks in ResNet-50/101 with our blocks.
Notations . Because we adopt the two rules in Sec. 3.1, it is


key: chroma:document
string_value:
Notations . Because we adopt the two rules in Sec. 3.1, it is
sufﬁcient for us to refer to an architecture by the template.
For example, Table 1 shows a ResNeXt-50 constructed by a
template with cardinality = 32 and bottleneck width =4d
(Fig. 3). This network is denoted as ResNeXt-50 ( 324d)
for simplicity. We note that the input/output width of the
template is ﬁxed as 256-d (Fig. 3), and all widths are doubled each time when the feature map is subsampled (see
Table 1).
Cardinality vs. Width . We ﬁrst evaluate the trade-off between cardinality Cand bottleneck width, under preserved
complexity as listed in Table 2. Table 3 shows the results
and Fig. 5 shows the curves of error vs. epochs. Comparing with ResNet-50 (Table 3 top and Fig. 5 left), the 32 4d
ResNeXt-50 has a validation error of 22.2%, which is 1.7%
lower than the ResNet baseline’s 23.9%. With cardinality C
increasing from 1 to 32 while keeping complexity, the error
rate keeps reducing. Furthermore, the 32 4d ResNeXt also


key: chroma:document
string_value:
increasing from 1 to 32 while keeping complexity, the error
rate keeps reducing. Furthermore, the 32 4d ResNeXt also
has a much lower training error than the ResNet counterpart, suggesting that the gains are notfrom regularization
but from stronger representations .
Similar trends are observed in the case of ResNet-101
(Fig. 5 right, Table 3 bottom), where the 32 4d ResNeXt101 outperforms the ResNet-101 counterpart by 0.8%. Although this improvement of validation error is smaller than
that of the 50-layer case, the improvement of training error is still big (20% for ResNet-101 and 16% for 32 4d
ResNeXt-101, Fig. 5 right). In fact, more training data
will enlarge the gap of validation error, as we show on an
ImageNet-5K set in the next subsection.
Table 3 also suggests that with complexity preserved , increasing cardinality at the price of reducing width starts
to show saturating accuracy when the bottleneck width is
6With BN, for the equivalent form in Fig. 3(a), BN is employed after
aggregating the transformations and before adding to the shortcut.
0 1 02 03 04 05 06 07 08 09 0
epochs1520253035404550top-1 error (%)ResNet-50 (1 x 64d) train


key: chroma:document
string_value:
0 1 02 03 04 05 06 07 08 09 0
epochs1520253035404550top-1 error (%)ResNet-50 (1 x 64d) train
ResNet-50 (1 x 64d) val
ResNeXt-50 (32 x 4d) train
ResNeXt-50 (32 x 4d) val
0 1 02 03 04 05 06 07 08 09 0
epochs1520253035404550top-1 error (%)ResNet-101 (1 x 64d) train
ResNet-101 (1 x 64d) val
ResNeXt-101 (32 x 4d) train
ResNeXt-101 (32 x 4d) valFigure 5. Training curves on ImageNet-1K. ( Left): ResNet/ResNeXt-50 with preserved complexity ( 4.1 billion FLOPs, 25 million
parameters); ( Right ): ResNet/ResNeXt-101 with preserved complexity ( 7.8 billion FLOPs,44 million parameters).
setting top-1 error (%)
ResNet-50 164d 23.9
ResNeXt-50 240d 23.0
ResNeXt-50 424d 22.6
ResNeXt-50 814d 22.3


key: chroma:document
string_value:
ResNeXt-50 424d 22.6
ResNeXt-50 814d 22.3
ResNeXt-50 324d 22.2
ResNet-101 164d 22.0
ResNeXt-101 240d 21.7
ResNeXt-101 424d 21.4
ResNeXt-101 814d 21.3
ResNeXt-101 324d 21.2
Table 3. Ablation experiments on ImageNet-1K. ( Top): ResNet50 with preserved complexity ( 4.1 billion FLOPs); ( Bottom ):
ResNet-101 with preserved complexity ( 7.8 billion FLOPs). The
error rate is evaluated on the single crop of 224 224 pixels.
small. We argue that it is not worthwhile to keep reducing
width in such a trade-off. So we adopt a bottleneck width
no smaller than 4d in the following.
Increasing Cardinality vs. Deeper/Wider . Next we investigate increasing complexity by increasing cardinality C
or increasing depth or width. The following comparison
can also be viewed as with reference to 2 FLOPs of the
ResNet-101 baseline. We compare the following variants


key: chroma:document
string_value:
or increasing depth or width. The following comparison
can also be viewed as with reference to 2 FLOPs of the
ResNet-101 baseline. We compare the following variants
that have15 billion FLOPs. (i) Going deeper to 200 layers. We adopt the ResNet-200 [15] implemented in [11].
(ii)Going wider by increasing the bottleneck width. (iii)
Increasing cardinality by doubling C.
Table 4 shows that increasing complexity by 2 consistently reduces error vs. the ResNet-101 baseline (22.0%).
But the improvement is small when going deeper (ResNet200, by 0.3%) or wider (wider ResNet-101, by 0.7%).
On the contrary, increasing cardinality Cshows muchsetting top-1 err (%) top-5 err (%)
1complexity references:
ResNet-101 164d 22.0 6.0
ResNeXt-101 324d 21.2 5.6
2complexity models follow:
ResNet- 200[15] 164d 21.7 5.8
ResNet-101, wider 1100d 21.3 5.7
ResNeXt-101 264d 20.7 5.5
ResNeXt-101 644d 20.4 5.3


key: chroma:document
string_value:
ResNeXt-101 264d 20.7 5.5
ResNeXt-101 644d 20.4 5.3
Table 4. Comparisons on ImageNet-1K when the number of
FLOPs is increased to 2 of ResNet-101’s. The error rate is evaluated on the single crop of 224 224 pixels. The highlighted factors
are the factors that increase complexity.
better results than going deeper or wider . The 264d
ResNeXt-101 ( i.e., doubling Con 164d ResNet-101 baseline and keeping the width) reduces the top-1 error by 1.3%
to 20.7%. The 644d ResNeXt-101 ( i.e., doubling Con
324d ResNeXt-101 and keeping the width) reduces the
top-1 error to 20.4%.
We also note that 32 4d ResNet-101 (21.2%) performs
better than the deeper ResNet-200 and the wider ResNet101, even though it has only 50% complexity. This again
shows that cardinality is a more effective dimension than
the dimensions of depth and width.
Residual connections. The following table shows the effects of the residual (shortcut) connections:
setting w/ residual w/o residual


key: chroma:document
string_value:
the dimensions of depth and width.
Residual connections. The following table shows the effects of the residual (shortcut) connections:
setting w/ residual w/o residual
ResNet-50 164d 23.9 31.2
ResNeXt-50 324d 22.2 26.1
Removing shortcuts from the ResNeXt-50 increases the error by 3.9 points to 26.1%. Removing shortcuts from its
ResNet-50 counterpart is much worse (31.2%). These comparisons suggest that the residual connections are helpful
foroptimization , whereas aggregated transformations are
stronger representations , as shown by the fact that they
perform consistently better than their counterparts with or
without residual connections.
Performance. For simplicity we use Torch’s built-in
grouped convolution implementation, without special optimization. We note that this implementation was brute-force
and not parallelization-friendly. On 8 GPUs of NVIDIA
M40, training 324d ResNeXt-101 in Table 3 takes 0.95s
per mini-batch, vs. 0.70s of ResNet-101 baseline that has
similar FLOPs. We argue that this is a reasonable overhead.
We expect carefully engineered lower-level implementation
(e.g., in CUDA) will reduce this overhead. We also expect


key: chroma:document
string_value:
We expect carefully engineered lower-level implementation
(e.g., in CUDA) will reduce this overhead. We also expect
that the inference time on CPUs will present less overhead.
Training the 2complexity model (64 4d ResNeXt-101)
takes 1.7s per mini-batch and 10 days total on 8 GPUs.
Comparisons with state-of-the-art results. Table 5 shows
more results of single-crop testing on the ImageNet validation set. In addition to testing a 224 224 crop, we
also evaluate a 320 320 crop following [15]. Our results compare favorably with ResNet, Inception-v3/v4, and
Inception-ResNet-v2, achieving a single-crop top-5 error
rate of 4.4%. In addition, our architecture design is much
simpler than all Inception models, and requires considerably fewer hyper-parameters to be set by hand.
ResNeXt is the foundation of our entries to the ILSVRC
2016 classiﬁcation task, in which we achieved 2ndplace.
We note that many models (including ours) start to get saturated on this dataset after using multi-scale and/or multicrop testing. We had a single-model top-1/top-5 error rates


key: chroma:document
string_value:
of 17.7%/3.7% using the multi-scale dense testing in [14],
on par with Inception-ResNet-v2’s single-model results of
17.8%/3.7% that adopts multi-scale, multi-crop testing. We
had an ensemble result of 3.03% top-5 error on the test set,
on par with the winner’s 2.99% and Inception-v4/InceptionResNet-v2’s 3.08% [37].
224224 320320 / 299 299
top-1 err top-5 err top-1 err top-5 err
ResNet-101 [14] 22.0 6.0 - ResNet-200 [15] 21.7 5.8 20.1 4.8
Inception-v3 [39] - - 21.2 5.6
Inception-v4 [37] - - 20.0 5.0
Inception-ResNet-v2 [37] - - 19.9 4.9
ResNeXt-101 ( 644d) 20.4 5.3 19.1 4.4
Table 5. State-of-the-art models on the ImageNet-1K validation


key: chroma:document
string_value:
Table 5. State-of-the-art models on the ImageNet-1K validation
set (single-crop testing). The test size of ResNet/ResNeXt is
224224 and 320320 as in [15] and of the Inception models
is 299299.
12345
mini-batches ×1052025303540455055top-1 error (%)ResNet-101 (1 x 64d) val
ResNeXt-101 (32 x 4d) valFigure 6. ImageNet-5K experiments. Models are trained on the
5K set and evaluated on the original 1K validation set, plotted as
a 1K-way classiﬁcation task. ResNeXt and its ResNet counterpart
have similar complexity.
5K-way classiﬁcation 1K-way classiﬁcation
setting top-1 top-5 top-1 top-5
ResNet-50 164d 45.5 19.4 27.1 8.2
ResNeXt-50 324d 42.3 16.8 24.4 6.6
ResNet-101 164d 42.4 16.9 24.2 6.8
ResNeXt-101 324d 40.1 15.1 22.2 5.7


key: chroma:document
string_value:
ResNeXt-101 324d 40.1 15.1 22.2 5.7
Table 6. Error (%) on ImageNet-5K . The models are trained on
ImageNet-5K and tested on the ImageNet-1K val set , treated as a
5K-way classiﬁcation task or a 1K-way classiﬁcation task at test
time. ResNeXt and its ResNet counterpart have similar complexity. The error is evaluated on the single crop of 224 224 pixels.
5.2. Experiments on ImageNet-5K
The performance on ImageNet-1K appears to saturate.
But we argue that this is not because of the capability of the
models but because of the complexity of the dataset. Next
we evaluate our models on a larger ImageNet subset that
has 5000 categories.
Our 5K dataset is a subset of the full ImageNet-22K set
[33]. The 5000 categories consist of the original ImageNet1K categories and additional 4000 categories that have the
largest number of images in the full ImageNet set. The 5K
set has 6.8 million images, about 5 of the 1K set. There is
no ofﬁcial train/val split available, so we opt to evaluate on


key: chroma:document
string_value:
set has 6.8 million images, about 5 of the 1K set. There is
no ofﬁcial train/val split available, so we opt to evaluate on
the original ImageNet-1K validation set. On this 1K-class
val set, the models can be evaluated as a 5K-way classiﬁcation task (all labels predicted to be the other 4K classes are
automatically erroneous) or as a 1K-way classiﬁcation task
(softmax is applied only on the 1K classes) at test time.
The implementation details are the same as in Sec. 4.
The 5K-training models are all trained from scratch, and
# params CIFAR-10 CIFAR-100
Wide ResNet [43] 36.5M 4.17 20.50
ResNeXt-29, 864d 34.4M 3.65 17.77
ResNeXt-29, 1664d 68.1M 3.58 17.31
Table 7. Test error (%) and model size on CIFAR. Our results are
the average of 10 runs.
are trained for the same number of mini-batches as the 1Ktraining models (so 1/5 epochs). Table 6 and Fig. 6 show


key: chroma:document
string_value:
are trained for the same number of mini-batches as the 1Ktraining models (so 1/5 epochs). Table 6 and Fig. 6 show
the comparisons under preserved complexity. ResNeXt-50
reduces the 5K-way top-1 error by 3.2% comparing with
ResNet-50, and ResNetXt-101 reduces the 5K-way top-1
error by 2.3% comparing with ResNet-101. Similar gaps
are observed on the 1K-way error. These demonstrate the
stronger representational power of ResNeXt.
Moreover, we ﬁnd that the models trained on the 5K
set (with 1K-way error 22.2%/5.7% in Table 6) perform
competitively comparing with those trained on the 1K set
(21.2%/5.6% in Table 3), evaluated on the same 1K-way
classiﬁcation task on the validation set. This result is
achieved without increasing the training time (due to the
same number of mini-batches) and without ﬁne-tuning. We
argue that this is a promising result, given that the training
task of classifying 5K categories is a more challenging one.
5.3. Experiments on CIFAR


key: chroma:document
string_value:
argue that this is a promising result, given that the training
task of classifying 5K categories is a more challenging one.
5.3. Experiments on CIFAR
We conduct more experiments on CIFAR-10 and 100
datasets [23]. We use the architectures as in [14] and replace the basic residual block by the bottleneck template
of2
6411, 64
33, 64
11, 2563
75. Our networks start with a single 3 3 conv
layer, followed by 3 stages each having 3 residual blocks,
and end with average pooling and a fully-connected classiﬁer (total 29-layer deep), following [14]. We adopt the same
translation and ﬂipping data augmentation as [14]. Implementation details are in the appendix.
We compare two cases of increasing complexity based
on the above baseline: (i) increase cardinality and ﬁx all
widths, or (ii) increase width of the bottleneck and ﬁx cardinality = 1. We train and evaluate a series of networks
under these changes. Fig. 7 shows the comparisons of test
error rates vs. model sizes. We ﬁnd that increasing cardinality is more effective than increasing width, consistent to
what we have observed on ImageNet-1K. Table 7 shows the


key: chroma:document
string_value:
what we have observed on ImageNet-1K. Table 7 shows the
results and model sizes, comparing with the Wide ResNet
[43] which is the best published record. Our model with a
similar model size (34.4M) shows results better than Wide
ResNet. Our larger method achieves 3.58% test error (average of 10 runs) on CIFAR-10 and 17.31% on CIFAR-100.
To the best of our knowledge, these are the state-of-the-art
results (with similar data augmentation) in the literature including unpublished technical reports.
4 8 16 32 64 128
# of parameters (M)3.53.63.73.83.94.04.14.24.34.44.5test error (%)ResNet-29 (increase width)
ResNeXt-29 (increase cardinality)1x64d
16x64d8x64d4x64d2x64d
1x384d1x128d
1x256d1x192dFigure 7. Test error vs. model size on CIFAR-10. The results are
computed with 10 runs, shown with standard error bars. The labels
show the settings of the templates.
setting AP@0.5 AP
ResNet-50 164d 47.6 26.5


key: chroma:document
string_value:
show the settings of the templates.
setting AP@0.5 AP
ResNet-50 164d 47.6 26.5
ResNeXt-50 324d 49.7 27.5
ResNet-101 164d 51.1 29.8
ResNeXt-101 324d 51.9 30.0
Table 8. Object detection results on the COCO minival set.
ResNeXt and its ResNet counterpart have similar complexity.
5.4. Experiments on COCO object detection
Next we evaluate the generalizability on the COCO object detection set [27]. We train the models on the 80k training set plus a 35k val subset and evaluate on a 5k val subset
(called minival ), following [1]. We evaluate the COCOstyle Average Precision (AP) as well as AP@IoU=0.5 [27].
We adopt the basic Faster R-CNN [32] and follow [14] to
plug ResNet/ResNeXt into it. The models are pre-trained
on ImageNet-1K and ﬁne-tuned on the detection set. Implementation details are in the appendix.


key: chroma:document
string_value:
on ImageNet-1K and ﬁne-tuned on the detection set. Implementation details are in the appendix.
Table 8 shows the comparisons. On the 50-layer baseline, ResNeXt improves AP@0.5 by 2.1% and AP by 1.0%,
without increasing complexity. ResNeXt shows smaller improvements on the 101-layer baseline. We conjecture that
more training data will lead to a larger gap, as observed on
the ImageNet-5K set.
It is also worth noting that recently ResNeXt has been
adopted in Mask R-CNN [12] that achieves state-of-the-art
results on COCO instance segmentation and object detection tasks.
Acknowledgment
S.X. and Z.T.’s research was partly supported by NSF
IIS-1618477. The authors would like to thank Tsung-Yi
Lin and Priya Goyal for valuable discussions.
A. Implementation Details: CIFAR
We train the models on the 50k training set and evaluate
on the 10k test set. The input image is 32 32 randomly
cropped from a zero-padded 40 40 image or its ﬂipping,
following [14]. No other data augmentation is used. The


key: chroma:document
string_value:
cropped from a zero-padded 40 40 image or its ﬂipping,
following [14]. No other data augmentation is used. The
ﬁrst layer is 33 conv with 64 ﬁlters. There are 3 stages
each having 3 residual blocks, and the output map size is
32, 16, and 8 for each stage [14]. The network ends with a
global average pooling and a fully-connected layer. Width
is increased by 2when the stage changes (downsampling),
as in Sec. 3.1. The models are trained on 8 GPUs with a
mini-batch size of 128, with a weight decay of 0.0005 and
a momentum of 0.9. We start with a learning rate of 0.1
and train the models for 300 epochs, reducing the learning
rate at the 150-th and 225-th epoch. Other implementation
details are as in [11].
B. Implementation Details: Object Detection
We adopt the Faster R-CNN system [32]. For simplicity
we do not share the features between RPN and Fast R-CNN.
In the RPN step, we train on 8 GPUs with each GPU holding
2 images per mini-batch and 256 anchors per image. We
train the RPN step for 120k mini-batches at a learning rate


key: chroma:document
string_value:
2 images per mini-batch and 256 anchors per image. We
train the RPN step for 120k mini-batches at a learning rate
of 0.02 and next 60k at 0.002. In the Fast R-CNN step, we
train on 8 GPUs with each GPU holding 1 image and 64
regions per mini-batch. We train the Fast R-CNN step for
120k mini-batches at a learning rate of 0.005 and next 60k at
0.0005, We use a weight decay of 0.0001 and a momentum
of 0.9. Other implementation details are as in https://
github.com/rbgirshick/py-faster-rcnn .
References
[1] S. Bell, C. L. Zitnick, K. Bala, and R. Girshick. Insideoutside net: Detecting objects in context with skip
pooling and recurrent neural networks. In CVPR ,
2016.
[2] G. Cantor. ¨Uber unendliche, lineare punktmannichfaltigkeiten, arbeiten zur mengenlehre aus den jahren
1872-1884. 1884.
[3] R. Collobert, S. Bengio, and J. Mari ´ethoz. Torch: a


key: chroma:document
string_value:
1872-1884. 1884.
[3] R. Collobert, S. Bengio, and J. Mari ´ethoz. Torch: a
modular machine learning software library. Technical
report, Idiap, 2002.
[4] A. Conneau, H. Schwenk, L. Barrault, and Y . Lecun. Very deep convolutional networks for natural language processing. arXiv:1606.01781 , 2016.
[5] N. Dalal and B. Triggs. Histograms of oriented gradients for human detection. In CVPR , 2005.
[6] E. Denton, W. Zaremba, J. Bruna, Y . LeCun, and
R. Fergus. Exploiting linear structure within convolutional networks for efﬁcient evaluation. In NIPS ,
2014.[7] J. Donahue, Y . Jia, O. Vinyals, J. Hoffman, N. Zhang,
E. Tzeng, and T. Darrell. Decaf: A deep convolutional
activation feature for generic visual recognition. In
ICML , 2014.


key: chroma:document
string_value:
E. Tzeng, and T. Darrell. Decaf: A deep convolutional
activation feature for generic visual recognition. In
ICML , 2014.
[8] D. Eigen, J. Rolfe, R. Fergus, and Y . LeCun. Understanding deep architectures using a recursive convolutional network. arXiv:1312.1847 , 2013.
[9] R. Girshick. Fast R-CNN. In ICCV , 2015.
[10] R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich
feature hierarchies for accurate object detection and
semantic segmentation. In CVPR , 2014.
[11] S. Gross and M. Wilber. Training and investigating Residual Nets. https://github.com/
facebook/fb.resnet.torch , 2016.
[12] K. He, G. Gkioxari, P. Doll ´ar, and R. Girshick. Mask
R-CNN. arXiv:1703.06870 , 2017.
[13] K. He, X. Zhang, S. Ren, and J. Sun. Delving deep
into rectiﬁers: Surpassing human-level performance
on imagenet classiﬁcation. In ICCV , 2015.


key: chroma:document
string_value:
into rectiﬁers: Surpassing human-level performance
on imagenet classiﬁcation. In ICCV , 2015.
[14] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual
learning for image recognition. In CVPR , 2016.
[15] K. He, X. Zhang, S. Ren, and J. Sun. Identity mappings in deep residual networks. In ECCV , 2016.
[16] Y . Ioannou, D. Robertson, R. Cipolla, and A. Criminisi. Deep roots: Improving cnn efﬁciency with hierarchical ﬁlter groups. arXiv:1605.06489 , 2016.
[17] S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal
covariate shift. In ICML , 2015.
[18] M. Jaderberg, A. Vedaldi, and A. Zisserman. Speeding up convolutional neural networks with low rank
expansions. In BMVC , 2014.
[19] Y . Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long,


key: chroma:document
string_value:
[19] Y . Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long,
R. Girshick, S. Guadarrama, and T. Darrell. Caffe:
Convolutional architecture for fast feature embedding.
arXiv:1408.5093 , 2014.
[20] N. Kalchbrenner, L. Espeholt, K. Simonyan, A. v. d.
Oord, A. Graves, and K. Kavukcuoglu. Neural machine translation in linear time. arXiv:1610.10099 ,
2016.
[21] Y .-D. Kim, E. Park, S. Yoo, T. Choi, L. Yang, and
D. Shin. Compression of deep convolutional neural
networks for fast and low power mobile applications.
InICLR , 2016.
[22] P. Kontschieder, M. Fiterau, A. Criminisi, and S. R.
Bul`o. Deep convolutional neural decision forests. In
ICCV , 2015.
[23] A. Krizhevsky. Learning multiple layers of features
from tiny images. Tech Report , 2009.


key: chroma:document
string_value:
ICCV , 2015.
[23] A. Krizhevsky. Learning multiple layers of features
from tiny images. Tech Report , 2009.
[24] A. Krizhevsky, I. Sutskever, and G. Hinton. Imagenet classiﬁcation with deep convolutional neural
networks. In NIPS , 2012.
[25] Y . LeCun, B. Boser, J. S. Denker, D. Henderson, R. E.
Howard, W. Hubbard, and L. D. Jackel. Backpropagation applied to handwritten zip code recognition.
Neural computation , 1989.
[26] M. Lin, Q. Chen, and S. Yan. Network in network. In
ICLR , 2014.
[27] T.-Y . Lin, M. Maire, S. Belongie, J. Hays, P. Perona,
D. Ramanan, P. Doll ´ar, and C. L. Zitnick. Microsoft
COCO: Common objects in context. In ECCV . 2014.
[28] J. Long, E. Shelhamer, and T. Darrell. Fully convolutional networks for semantic segmentation. In CVPR ,
2015.


key: chroma:document
string_value:
[28] J. Long, E. Shelhamer, and T. Darrell. Fully convolutional networks for semantic segmentation. In CVPR ,
2015.
[29] D. G. Lowe. Distinctive image features from scaleinvariant keypoints. IJCV , 2004.
[30] A. Oord, S. Dieleman, H. Zen, K. Simonyan,
O. Vinyals, A. Graves, N. Kalchbrenner, A. Senior,
and K. Kavukcuoglu. Wavenet: A generative model
for raw audio. arXiv:1609.03499 , 2016.
[31] P. O. Pinheiro, R. Collobert, and P. Dollar. Learning
to segment object candidates. In NIPS , 2015.
[32] S. Ren, K. He, R. Girshick, and J. Sun. Faster RCNN: Towards real-time object detection with region
proposal networks. In NIPS , 2015.
[33] O. Russakovsky, J. Deng, H. Su, J. Krause,
S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla,
M. Bernstein, A. C. Berg, and L. Fei-Fei. ImageNet


key: chroma:document
string_value:
M. Bernstein, A. C. Berg, and L. Fei-Fei. ImageNet
Large Scale Visual Recognition Challenge. IJCV ,
2015.
[34] P. Sermanet, D. Eigen, X. Zhang, M. Mathieu, R. Fergus, and Y . LeCun. Overfeat: Integrated recognition, localization and detection using convolutional
networks. In ICLR , 2014.
[35] L. Sifre and S. Mallat. Rigid-motion scattering for
texture classiﬁcation. arXiv:1403.1687 , 2014.
[36] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. In
ICLR , 2015.
[37] C. Szegedy, S. Ioffe, and V . Vanhoucke. Inceptionv4, inception-resnet and the impact of residual connections on learning. In ICLR Workshop , 2016.
[38] C. Szegedy, W. Liu, Y . Jia, P. Sermanet, S. Reed,
D. Anguelov, D. Erhan, V . Vanhoucke, and A. Rabinovich. Going deeper with convolutions. In CVPR ,
2015.


key: chroma:document
string_value:
D. Anguelov, D. Erhan, V . Vanhoucke, and A. Rabinovich. Going deeper with convolutions. In CVPR ,
2015.
[39] C. Szegedy, V . Vanhoucke, S. Ioffe, J. Shlens, and
Z. Wojna. Rethinking the inception architecture for
computer vision. In CVPR , 2016.[40] A. Veit, M. Wilber, and S. Belongie. Residual networks behave like ensembles of relatively shallow network. In NIPS , 2016.
[41] Y . Wu, M. Schuster, Z. Chen, Q. V . Le, M. Norouzi,
W. Macherey, M. Krikun, Y . Cao, Q. Gao,
K. Macherey, et al. Google’s neural machine translation system: Bridging the gap between human and
machine translation. arXiv:1609.08144 , 2016.
[42] W. Xiong, J. Droppo, X. Huang, F. Seide, M. Seltzer,
A. Stolcke, D. Yu, and G. Zweig. The Microsoft
2016 Conversational Speech Recognition System.
arXiv:1609.03528 , 2016.


key: chroma:document
string_value:
2016 Conversational Speech Recognition System.
arXiv:1609.03528 , 2016.
[43] S. Zagoruyko and N. Komodakis. Wide residual networks. In BMVC , 2016.
[44] M. D. Zeiler and R. Fergus. Visualizing and understanding convolutional neural networks. In ECCV ,
2014.


key: chroma:document
string_value:
Deep Watershed Transform for Instance Segmentation
Min Bai Raquel Urtasun
Department of Computer Science, University of Toronto
fmbai, urtasun g@cs.toronto.edu
Abstract
Most contemporary approaches to instance segmentation use complex pipelines involving conditional random
ﬁelds, recurrent neural networks, object proposals, or template matching schemes. In this paper, we present a simple yet powerful end-to-end convolutional neural network to
tackle this task. Our approach combines intuitions from the
classical watershed transform and modern deep learning to
produce an energy map of the image where object instances
are unambiguously represented as energy basins. We then
perform a cut at a single energy level to directly yield connected components corresponding to object instances. Our
model achieves more than double the performance over
the state-of-the-art on the challenging Cityscapes Instance
Level Segmentation task.
1. Introduction
Instance segmentation seeks to identify the semantic
class of each pixel as well as associate each pixel with a
physical instance of an object. This is in contrast with
semantic segmentation, which is only concerned with the
ﬁrst task. Instance segmentation is particularly challenging in street scenes, where the scale of the objects can vary
tremendously. Furthermore, the appearance of objects are


key: chroma:document
string_value:
ﬁrst task. Instance segmentation is particularly challenging in street scenes, where the scale of the objects can vary
tremendously. Furthermore, the appearance of objects are
affected by partial occlusions, specularities, intensity saturation, and motion blur. Solving this task will, however,
tremendously beneﬁt applications such as object manipulation in robotics, or scene understanding and tracking for
self-driving cars.
Current approaches generally use complex pipelines to
handle instance extraction involving object proposals [20,
22, 7], conditional random ﬁelds (CRF) [32, 33], large recurrent neural networks (RNN) [24, 23, 2], or template
matching [28]. In contrast, we present an exceptionally
simple and intuitive method that signiﬁcantly outperforms
the state-of-the-art. In particular, we derive a novel approach which brings together classical grouping techniques
and modern deep neural networks.
The watershed transform is a well studied method inmathematical morphology. Its application to image segmentation can be traced back to the 70’s [4, 3]. The idea
behind this transform is fairly intuitive. Any greyscale image can be considered as a topographic surface. If we ﬂood
this surface from its minima and prevent the merging of the


key: chroma:document
string_value:
behind this transform is fairly intuitive. Any greyscale image can be considered as a topographic surface. If we ﬂood
this surface from its minima and prevent the merging of the
waters coming from different sources, we effectively partition the image into different components (i.e., regions).
This transformation is typically applied to the image gradient, thus the basins correspond to homogeneous regions in
the image. A signiﬁcant limitation of the watershed transform is its propensity to over-segment the image. One of
the possible solutions is to estimate the locations of object
instance markers, which guide the selection of a subset of
these basins [11, 21]. Heuristics on the relative depth of the
basins can be exploited in order to merge basins. However,
extracting appropriate markers and creating good heuristics
is difﬁcult in practice. As a consequence, modern techniques for instance segmentation do not exploit the watershed transform.
In this paper, we propose a novel approach which combines the strengths of modern deep neural networks with the
power of this classical bottom-up grouping technique. We
propose to directly learn the energy of the watershed transform such that each basin corresponds to a single instance,
while all dividing ridges are at the same height in the energy domain. As a consequence, the components can be


key: chroma:document
string_value:
while all dividing ridges are at the same height in the energy domain. As a consequence, the components can be
extracted by a cut at a single energy level without leading
to over-segmentation. Our approach has several key advantages: it can be easily trained end-to-end, and produces very
fast and accurate estimates. Our method does not rely on iterative strategies such as RNNs, thus has a constant runtime
regardless of the number of object instances.
We demonstrate the effectiveness of our approach in the
challenging Cityscapes Instance Segmentation benchmark
[6], and show that we more than double the performance
of the current state-of-the-art. In the following sections, we
ﬁrst review related work. We then present the details behind
our intuition and model design, followed by an analysis of
our model’s performance. Finally, we explore the impact of
various parts of our model in ablation studies.
1arXiv:1611.08303v2  [cs.CV]  4 May 2017
(a) Input Image
 (b) Semantic Segmentation [34]
 (c) Our Instance Segmentation
 (d) GT Instance Segmentation
Figure 1: Sample prediction: the input image is gated by sem. segmentation from [34] and passed through our model.


key: chroma:document
string_value:
(d) GT Instance Segmentation
Figure 1: Sample prediction: the input image is gated by sem. segmentation from [34] and passed through our model.
2. Related
Several instance level segmentation approaches have
been proposed in recent years. We now brieﬂy review them.
Proposal based: Many approaches are based on the reﬁnement of object proposals. For example, [1] generates
object segments proposals, and reasons about combining
them into object instances. In a similar spirit, [12] selects
proposals using CNN features and non-maximum suppression. Based on this, [5] further reasons about multiple object proposals to handle occlusion scenarios where single
objects are split into multiple disconnected patches. [7] uses
a deep cascaded neural network to propose object instance
bounding boxes, reﬁne instance masks, and semantically
label the masks in sequence. [20, 22] generate segmentation proposals using deep CNNs, which are then further reﬁned to achieve better segmentation boundaries. Additionally, [31] uses a modiﬁed R-CNN model to propose instance
bounding boxes, which are then further reﬁned to obtain instance level segmentation.


key: chroma:document
string_value:
bounding boxes, which are then further reﬁned to obtain instance level segmentation.
Deep structured models: [32, 33] ﬁrst use CNNs to perform local instance disambiguation and labelling, followed
by a global conditional random ﬁeld (CRF) to achieve instance label consistency. Recent work by [2] uses object
detection proposals in conjunction with a deep high order
CRF to reason about pixel assignment in overlapping object
proposal boxes.
Template matching: [28] extracts image features using
CNNs to assign a sector label to each pixel in an object instance, which corresponds to one of eight discretized radial
bins around the object’s visible center. A template matching
scheme is then used to associate instance center proposals
and pixels with an object instance.
Recurrent Networks: [24] uses CNNs for feature extraction, followed by a recurrent neural network (RNN) that
generates instance labels for one object at a time. The recurrent structures (based on ConvLSTM [25]) keep track of
instances that have already been generated, and inhibit these
regions from further instance generation. Additionally, [23]
extracts image features similar to [28] and employs a fairly
complex pipeline including a ConvLSTM structure to direct


key: chroma:document
string_value:
regions from further instance generation. Additionally, [23]
extracts image features similar to [28] and employs a fairly
complex pipeline including a ConvLSTM structure to direct
a bounding box generation network followed by a segmentation network that extracts individual instances.CNN: [15] leverages only a CNN trained to provide multiple outputs to simultaneously predict instance numbers,
bounding box coordinates, and category conﬁdence scores
for each pixel. This is followed by generic clustering algorithms to group the resulting output into instance-wise labels. Additionally, [13] proposed deep convolutional neural
network that learns the underlying shapes of objects, and
performs multiple unconstrained inference steps to reﬁne
regions corresponding to an object instance while ignoring
neighboring pixels that do not belong to the primary object.
Proposal + recursion: [14] proposed a novel method
that recursively reﬁnes proposals.
In contrast, in this paper we propose a novel approach
which combines the strengths of modern deep neural networks with the power of the watershed transform. Our
model is simple, fast, accurate, and inherently handles an
arbitrary number of instances per image with ease.
3. A Review on the Watershed Transform
We start our discussion with a review of the watershed


key: chroma:document
string_value:
arbitrary number of instances per image with ease.
3. A Review on the Watershed Transform
We start our discussion with a review of the watershed
transform, a well studied method in mathematical morphology. This technique is built on the fact that any greyscale
image can be considered as a topographic surface. If we
ﬂood this surface from its minima while building barriers
to prevent the merging of the waters coming from different sources, we effectively partition the image into different
components or regions. These components are called catchment basins. The barriers or watershed lines then represent
the boundaries between the different basins (i.e., boundaries
between regions).
This process is illustrated in the ﬁrst row of Fig. 2 for
a one dimensional energy function. In this case, the watershed transform results in seven components, which are
illustrated in different colors. Note that the traditional watershed transform tends to produce an over-segmentation of
the image due to spurious small ridges which produce separate components. In this example, although there are 3
main components, the watershed transform over-segments
the image because of small perturbations in the energy.
Several algorithms have been developed to estimate the
components. [3] proposed an algorithm that iteratively ﬁlls
the watershed landscape from each local minimum, adding
dams wherever two neighbouring bodies of water meet.


key: chroma:document
string_value:
components. [3] proposed an algorithm that iteratively ﬁlls
the watershed landscape from each local minimum, adding
dams wherever two neighbouring bodies of water meet.
(a) Traditional Watershed Energy
(b) Our learned energy
Figure 2: Conceptual 1-D comparison between the traditional watershed transform and our deep watershed transform. The resulting instances are represented as colors
across the top.
These dams deﬁne the segmentation boundaries. Additionally, [18] details a number of alternative watershed transform algorithms, including topological distance, shortest
path algorithms, spanning trees, and marker based methods.
The watershed transform is typically applied to the image gradient, while the catchment basins correspond to homogeneous grey level regions in the image. However, estimating sharp gradients that represent the boundaries between different instances is a very challenging process.
In the next section, we will show an alternative approach
which directly learns to predict the energy landscape.
4. Deep Watershed Tranform
In this section, we present our novel approach to instance
level segmentation. In particular, we learn the energy of the
watershed transform with a feed-forward neural network.
The idea behind our approach is very intuitive. It consists
of learning to predict an energy landscape such that each
basin corresponds to a single instance, while all ridges are
at the same height in the energy domain. As a consequence,


key: chroma:document
string_value:
of learning to predict an energy landscape such that each
basin corresponds to a single instance, while all ridges are
at the same height in the energy domain. As a consequence,
the watershed cut corresponds to a single threshold in the
energy, which does not lead to over segmentation. We refer
the reader to the lower half of Fig. 2 for an illustration of
the desired energy.
Unfortunately, learning the energy landscape from
scratch is a complex task. Therefore, we aid the network
by deﬁning an intermediate task, where we learn the direction of descent of the watershed energy. This is then passedthrough another set of network layers to learn the ﬁnal energy. In principle, one can interpret this network as learning
to perform the distance transform of each point within an
object instance to the instance’s boundary. Fig. 4 shows an
example of the input, intermediate results, and ﬁnal output
of our method. We refer the reader to Fig. 3 for an illustration of our network architecture.
4.1. Direction Network (DN)
Our approach leverages semantic segmentation as input
to focus only on relevant areas of the image. Note that our
network is agnostic to the choice of segmentation algorithm.
In our paper, we use the semantic segmentation results from
PSPNet [34].


key: chroma:document
string_value:
network is agnostic to the choice of segmentation algorithm.
In our paper, we use the semantic segmentation results from
PSPNet [34].
The network takes as input the original RGB image gated
by a binarized semantic segmentation, where all pixels that
are not part of one of the semantic classes of interest are set
to zero. The input image is augmented by adding the semantic segmentation as a fourth channel. Because the RGB
values range from 0 to 255 for each channel before mean
subtraction, we likewise scale the encoding of the semantic
segmentation image such that the class labels are equally
spaced numerically, and the variance is approximately equal
to that of values in the RGB channels.
To aid the model in producing an accurate energy landscape, we pre-train the overall network’s ﬁrst part (referenced here as the Direction Network, DN) to estimate the
direction of descent of the energy at each pixel. We parameterize it with a unit vector pointing away from the nearest
point on the object’s instance boundary. This supervision
gives a very strong training signal: the direction of the nearest boundary is directly encoded in the unit vector. Furthermore, a pair of pixels straddling an occlusion boundary
between two objects will have target unit vectors pointing
in opposite directions. Here, associating a pixel with the


key: chroma:document
string_value:
between two objects will have target unit vectors pointing
in opposite directions. Here, associating a pixel with the
wrong object instance incurs the maximum possible angular
error. This is true regardless of the shape of the objects, even
when they are highly concave or elongated. This forces the
network to learn very accurate boundary localization at the
pixel level.
LetDgt(p)be the ground truth distance transform from
a point to the boundary of the instance it belongs to. We
deﬁne our ground truth targets as the normalized gradient
~ upof this distance transform. More formally,
~ up;gt=rDgt(p)
jrDgt(p)j
Thus, the DN produces a two channel output at the input
image’s resolution representing the directional unit vector.
It is important to note that the normalization layer at the output of the DN restricts the sum of each channel’s squared
output to be 1. This greatly reduces the difﬁculty of using the output of non-linear activation functions to model
Figure 3: Complete network architecture. The network takes the original RGB image gated by semantic segmentation and
concatenated with the semantic segmentation as input, and produces the deep watershed transform energy map as the output.
(a) Input Image
 (b) GT angle of ~ up
 (c) GT Watershed Energy


key: chroma:document
string_value:
(a) Input Image
 (b) GT angle of ~ up
 (c) GT Watershed Energy
 (d) GT Instances
(e) Sem. Segmentation of [34]
 (f) Pred. angle of ~ up
 (g) Pred. Watershed Transform
 (h) Pred. Instances
Figure 4: Our network takes the RGB image (a) and the semantic segmentation (e) as input, and predicts a unit vector at each
foreground pixel pointing directly away from the nearest boundary (f). Based on this, we then predict a modiﬁed watershed
transform energy (g), upon which we perform cut at a ﬁxed threshold to yield the ﬁnal predictions (h).
a scalar target. Additionally, phrasing the training target as
a unit vector instead of an angle bypasses the problem of
the equivalent angles 0and 360having a large numerical
difference.
The feature extraction portion of the DN’s architecture
is inspired by VGG16 [27]. However, there are several important modiﬁcations. Direction prediction is a very precise task, as the output can vary greatly between neighboring pixels. Thus, it is critical to avoid losing spatial
resolution. We utilize a modiﬁed version of the ﬁrst 13


key: chroma:document
string_value:
resolution. We utilize a modiﬁed version of the ﬁrst 13
layers of VGG, where the third and fourth pooling layers are changed to average pooling while the ﬁfth pooling
layer is removed. To preserve spatial resolution, we exploit a high-capacity, multi-scale information aggregation
scheme inspired by popular methods in semantic segmenta-tion [9, 30, 16]. In particular, the outputs of conv3 ,conv4 ,
andconv5 individually undergo a 55convolution followed
by two 11convolutions. After this, the outputs of the
second and third paths are upsampled to the resolution of
the ﬁrst path. The resulting feature volumes from the three
paths are concatenated. This undergoes an additional set
of three 11convolutions, before being upsampled to the
input resolution.
4.2. Watershed Transform Network (WTN)
The second half of the overall network takes as input the
2-channel unit vector map, and produces a discretized modiﬁed watershed transform map with K= 16 possible energy
values. In this case, bin 0 corresponds to background or regions within 2 pixels of an instance boundary and is referred
to as having an energy level of 0. Meanwhile, higher numbered bins with higher energy levels correspond to regions


key: chroma:document
string_value:
to as having an energy level of 0. Meanwhile, higher numbered bins with higher energy levels correspond to regions
in the interior of object instances. The bins are chosen to
maximize the binning resolution of energy levels near zero
(to facilitate accurate cutting), while achieving an approximate balance between the total numbers of pixels within
each class.
The WTN is a fairly generic CNN, with an emphasis on
reasoning with high spatial accuracy and resolution. The architecture is shown in the right half of Fig. 3. In particular,
the network consists of two 55convolutional ﬁlter blocks
each followed by a 22average pooling, before undergoing a set of 11convolutions and upsampling to the input
image resolution.
4.3. Network Training
We ﬁrst pre-train our DN and WTN networks. This process is followed by end-to-end training of the whole network.
Direction Network pre-training: We pre-train the network using the mean squared error in the angular domain as
loss:
ldirection =X
p2P objwpkcos1< ~ up;GT; ~ up;pred>k2


key: chroma:document
string_value:
loss:
ldirection =X
p2P objwpkcos1< ~ up;GT; ~ up;pred>k2
wherePobjis the set of all pixels belonging to a semantic class with object instances, and wpis a weighting factor proportional to the inverse of the square root of the object instance’s area. We use the mean squared angular error as opposed to the commonly used cosine distance, as
large angular errors (occuring when the network associates
a pixel with the wrong object instance) should incur significant larger penalties than small angular errors (which have
little impact).
This network is pre-trained using the original input images gated by the ground truth semantic segmentation and
the ground truth unit vectors ~ uGT. The lower layers (conv1
to conv5) are initialized with VGG16 [27]. The weights of
the upper layers are initialized randomly according to the
Xavier initialization scheme [10]. However, the intialization variance of the weights of the ﬁnal layer of each prediction branch is set manually such that the variance of each
branch’s output is of the same order of magnitude before
concatenation. This encourages the network to consider the
output of each branch. We use the ADAM optimizer to train


key: chroma:document
string_value:
concatenation. This encourages the network to consider the
output of each branch. We use the ADAM optimizer to train
the network for 20 epochs with a batch size of 4, with a constant learning rate of 1e-5 and L2 weight penalty of 1e-5.
Watershed Network pre-training: The network is
trained using a modiﬁed cross-entropy loss
lwatershed =X
p2P objKX
k=1wpck(tp;klog yp;k+tp;klogyp;k)where tp;kis the k-th element of pixel p’s one-hot target
vector, yp;kis the k-th channel of the network output at p,
wpis a coefﬁcient to adjust the importance of smaller objects (as deﬁned in the DN pre-training section), and ckis
a scaling constant speciﬁc to each discretization class. Because the single energy level cut occurs at just above zero
energy (i.e., near object instance boundaries), an accurate
estimation of pixels at low energy levels is of crucial importance. Thus, the set fckgis selected to be in increasing order. In this case, any errors of low energy levels are assigned
a greater level of penalty, which encourages the network to


key: chroma:document
string_value:
a greater level of penalty, which encourages the network to
focus on boundary regions. We pre-train the network using
the ground truth semantic segmentation and ground truth
direction predictions as input, and the discretized ground
truth modiﬁed watershed transform as target. All weights
are initialized using Xavier initialization [10]. We train the
network for 25 epochs using the ADAM optimizer. A batch
size of 6, constant learning rate of 5e-4, and a L2 weight
penalty of 1e-6 are used.
End-to-end ﬁne-tuning: We cascaded the pre-trained
models for the DN and WTN and ﬁne-tuned the complete
model for 20 epochs using the RGB image and semantic
segmentation output of PSPNet as input, and the ground
truth distance transforms as the training target. We use a
batch size of 3, constant learning rate of 5e-6, and a L2
weight penalty of 1e-6.
4.4. Energy Cut and Instance Extraction
We cut our watershed transform output at energy level 1
(0 to 2 pixels from boundary) for classes with many smaller
objects ( person ,rider ,bicycle ,motorcycle ), and level 2 (3 to
4 pixels from boundary) for classes with larger objects ( car,


key: chroma:document
string_value:
objects ( person ,rider ,bicycle ,motorcycle ), and level 2 (3 to
4 pixels from boundary) for classes with larger objects ( car,
bus,truck ,train ). Following this, the instances are dilated
using a circular structuring element whose radius equals the
maximum erosion at the energy level of the cut. This offsets the boundary erosion from the non-zero threshold. The
connected components are identiﬁed in the resulting image,
directly yielding the proposed instances. The proposals are
further reﬁned by basic hole-ﬁlling. Finally, we remove
small, spurious instances.
5. Experimental Evaluation
In this section, we evaluate our approach on the challenging Cityscapes Instance-Level Semantic Labelling Task [6].
The ofﬁcial benchmark test and validation set results are
found in Tables 1 and 2. We then perform ablation studies with the validation set to examine the performance of
various aspects of our model.
Dataset: The Cityscapes Instance Labelling Task contains 5000 ﬁnely annotated street scene images taken by a
vehicle-mounted camera. Each image has a resolution of
MethodCityscapes Test Set Cityscapes Val Set
AP AP 50% AP 100m AP 50m AP muCov


key: chroma:document
string_value:
vehicle-mounted camera. Each image has a resolution of
MethodCityscapes Test Set Cityscapes Val Set
AP AP 50% AP 100m AP 50m AP muCov
van den Brand et al. [29] 2.3% 3.7% 3.9% 4.9% - Cordts et al. [6] 4.6% 12.9% 7.7% 10.3% - Uhrig et al. [28] 8.9% 21.1% 15.3% 16.7% 9.9% Ours 19.4% 35.3% 31.4% 36.8% 21.2% 68.0%
Table 1: Cityscapes instance segmentation results using metrics deﬁned in [6] for AP and [26] for muCov.
Method Person Rider Car Truck Bus Train Motorcycle Bicycle
van den Brand et al. [29] - - 18.2% - - - - Cordts et al. [6] 1.3% 0.6% 10.5% 6.1% 9.7% 5.9% 1.7% 0.5%
Uhrig et al. [28] 12.5% 11.7% 22.5% 3.3% 5.9% 3.2% 6.9% 5.1%


key: chroma:document
string_value:
Uhrig et al. [28] 12.5% 11.7% 22.5% 3.3% 5.9% 3.2% 6.9% 5.1%
Ours 15.5% 14.1% 31.5% 22.5% 27.0% 22.9% 13.9% 8.0%
Table 2: Cityscapes instance segmentation class speciﬁc test set AP scores using metrics deﬁned in [6].
2048x1024 pixels. Unlike other commonly used datasets
for instance segmentation (e.g., Pascal VOC [8], BSDS500
[17], and CVPPP [19]) Cityscapes has a large number of
scenes involving dozens of instances with large degrees of
occlusions at vastly different scales. Eight of the semantically labelled categories have instance-level labelling. We
refer the reader to Table 3 for a summary of the statistics of
the object instances in this dataset. Note that while the car
andpeople classes have signiﬁcant numbers of instances,
the other six classes are rather uncommon. As a result, the
rare classes have far less training data, and are thus much
more challenging. We use the ofﬁcial training, validation,
and testing set splits, with 2975, 500, and 1525 images, respectively.


key: chroma:document
string_value:
more challenging. We use the ofﬁcial training, validation,
and testing set splits, with 2975, 500, and 1525 images, respectively.
Metric: We report the metrics employed by the
Cityscapes leaderboard. Several variants of the instancelevel average precision score are used. This is calculated
by ﬁnding the precision at various levels of intersectionover-union (IoU) between predicted instances and ground
truth instances, ranging from 50% to 95%. The main score
is the Average Precision (AP). Additionally, there are three
minor scores: AP at 50% overlap, AP of objects closer
than 50m, and AP of objects closer than 100m. Methods
are ranked by the AP score over all classes (mean AP).
Note that AP is a detection score, which does not penalize
overlapping instances or a large number of predictions,
as long as they are ranked in the proper order. Thus,
it places approaches (like ours) which predict a single
instance label (or background) per pixel at a disadvantage,
while favoring detection-based methods. Despite this, we
use these metrics to be consistent with the the evaluation
of the state-of-the-art. For the fbus,truck ,traingclasses,
we order the instances by simply averaging the semantic


key: chroma:document
string_value:
of the state-of-the-art. For the fbus,truck ,traingclasses,
we order the instances by simply averaging the semantic
segmentation’s output conﬁdence within each instance to
somewhat counteract the errors by semantic segmentation.Category Total
PixelsTotal
InstancesAverage
SizeInstances
/ Image
Person 6.6e+7 17900 3691 6.0
Rider 7.4e+6 1754 4244 0.6
Car 3.8e+8 26929 14233 9.1
Truck 1.5e+7 482 30648 0.2
Bus 1.3e+7 379 34275 0.1
Train 3.2e+6 168 19143 0.1
Motorcycle 5.4e+6 735 7298 0.2
Bicycle 1.6e+7 3655 4332 1.2
Table 3: Statistics of the training set of the cityscapes instance level segmentation. Average size is in pixels.
Sem. Seg. Src. Sem. Seg. IoU Inst. Seg. AP
LRR [9] 71.8% 20.2%
PSPNet [34] 80.2% 21.2%
Table 4: Comparison of instance segmentation performance
on the validation set with various semantic segmentation


key: chroma:document
string_value:
PSPNet [34] 80.2% 21.2%
Table 4: Comparison of instance segmentation performance
on the validation set with various semantic segmentation
sources. Semantic segmentation IoU Class scores [6] are
also provided.
For all other classes, we use a random ordering. Note that
sophisticated ranking techniques can be used to further
improve the score. However, the segmentation quality
remains this same. This again highlights a shortcoming of
using AP as the metric.
In addition to AP, we report the mean weighted coverage
score introduced in [26] on the validation set. This metric
enforces a single instance label per pixel, and is therefore
more suitable for evaluating our approach. We hope that
future work will likewise report this score when applicable.
Method Average Person Rider Car Truck Bus Train Motorcycle Bicycle
Ours 20.50% 15.50% 13.80% 33.10% 23.60% 39.90% 17.40% 11.90% 8.80%
Ours + Ordering 21.24% 15.50% 13.80% 33.10% 27.10% 45.20% 14.50% 11.90% 8.80%


key: chroma:document
string_value:
Ours + Oracle Ordering 27.58% 20.60% 18.70% 40.10% 31.50% 50.60% 28.30% 17.40% 13.40%
Table 5: Comparison of AP scores with various instance ordering techniques using the validation set.
Comparison to the state-of-the-art: We show the instance segmentation test set scores in Table 1. Additionally, we show the class-speciﬁc AP scores in Table 2. It is
evident that we achieve a large improvement over the stateof-the-art in all semantic classes. Moreover, we do not use
depth information to train our model, unlike [28].
Analysis of the intermediate training target: Our ﬁnal
network is the result of the end-to-end ﬁne-tuning of two
pre-trained sub-networks (DN and WTN). Fig. 4 (f) shows
the output of the DN after ﬁnetuning. It is evident that the
ﬁne-tuned model retained the direction prediction as an intermediate task. This suggests that the intermediate training
target is effective.
Inﬂuence of semantic segmentation: While we elected
to use PSPNet [34] as our semantic segmentation source, we
additionally demonstrate that our method is able to use other


key: chroma:document
string_value:
to use PSPNet [34] as our semantic segmentation source, we
additionally demonstrate that our method is able to use other
sources. Table 4 shows the use of LRR [9] for semantic segmentation. Using the same pre-trained DN and DTN models, we perform end-to-end ﬁne-tuning using LRR as semantic segmentation. We note that the performance of our
model improves with better semantic segmentation. Thus,
future advances in segmentic segmentation methods can
further improve the performance of our approach.
Conﬁdence score estimate: As mentioned, the AP score
calculated by the Cityscapes benchmark requires a conﬁdence score for each instance. For the fbus,truck ,traing
set, we produce a weak ranking based on semantic segmentation softmax conﬁdence. Instances of all other classes are
randomly ranked. Table 5 explores the impact of various ordering schemes. We compare our ordering with random for
all semantic classes, as well as optimal ordering using oracle IoU. We see that ordering using oracle IoU can increase
our model’s performance by 6.34%. Note, however, that
this has no impact on the actual quality of proposed object
instances, which remain the same. This shows the necessity


key: chroma:document
string_value:
this has no impact on the actual quality of proposed object
instances, which remain the same. This shows the necessity
of a different metric such as muCov [26] that can evaluate
segmentation-based approaches fairly.
Qualitative Results: Fig. 5 depicts visualizations of sample results on the validation set, which is not used as part of
training. It is evident that our model produces very high
quality instance segmentation results. In these results, predicted and ground truth object instances only share the same
color if they have greater than 50% IoU.Failure Modes: Our model has several weaknesses.
Some images in Fig. 5 demonstrate these cases. The ﬁrst
issue is that the current formulation of our method does not
handle objects that have been separated into multiple pieces
by occlusion. This is most obvious in the 3rd image from
the bottom in Fig. 5 as the far-right vehicle is bisected by the
street sign pole, and the bicycle in the right part of the image just above. The resulting pieces are not merged into one
component. This is a drawback of most bottom-up grouping approaches. The second issue are cases where two objects sharing an occlusion boundary are mostly but not fully
separated by a line of low energy. This is seen in the rightmost vehicle in the 11th image. We anticipate that a combination of our method with top-down reasoning approaches


key: chroma:document
string_value:
separated by a line of low energy. This is seen in the rightmost vehicle in the 11th image. We anticipate that a combination of our method with top-down reasoning approaches
will greatly alleviate these two issues.
Because we rely upon correct semantic segmentation, errors from this (such as identifying a train as a bus) cannot be
ﬁxed by our method. This is clearly shown by the truck in
the last example. A possible solution could be to use semantic segmentation as soft gating, or to reason about semantic
and instance segmentation jointly. Finally, some very complex scenes such as some subgroups of people on the left in
the second to fourth example are incorrectly separated by
our model, and are fused together.
6. Conclusion
In this paper, we have proposed a simple instance segmentation technique inspired by the intuitive and classical watershed transform. Using a novel deep convolutional neural network and innovative loss functions for pretraining and ﬁne-tuning, we proposed a model that generates a modiﬁed watershed energy landscape. From this energy landscape, we directly extract high quality object instances. Our experiments show that we can more than double the performance of the state-of-the-art in the challenging Cityscapes Instance Segmentation task. We will release
the network weights and code for training and testing our


key: chroma:document
string_value:
the network weights and code for training and testing our
model. In the future, we plan to augment the method to handle object instances bisected by occlusions. Additionally,
we wish to explore the possibility of extending our approach
to perform joint semantic and instance level segmentation,
in hopes of further reﬁning both outputs simultaneously.
Acknowledgements: This work was partially supported
by ONR-N00014-14-1-0232, Samsung, NVIDIA, Google
and NSERC.
(a) Input Image
 (b) Sem. Segmentation [34]
 (c) Our Instance Segmentation
 (d) GT Instance Segmentation
Figure 5: Sample output of our model on the validation set. Note that predicted object instances and ground truth object
instances are only given the same color if they have over 50% IoU.
References
[1] P. Arbelaez, J. Pont-Tuset, J. Barron, F. Marques, and J. Malik. Multiscale Combinatorial Grouping. In CVPR , 2014.
2
[2] A. Arnab and P. Torr. Bottom-up instance segmentation using deep higher-order crfs. In BMVC , 2016. 1, 2
[3] S. Beucher. The watershed transformation applied to image segmentation. Scanning Microscopy International ,


key: chroma:document
string_value:
[3] S. Beucher. The watershed transformation applied to image segmentation. Scanning Microscopy International ,
Suppl:6(1):299–314, 1991. 1, 2
[4] S. Beucher and C. Lantuejoul. Use of watersheds in contour
detection. Proc. Int. Workshop Image Processing, Real-Time
Edge and Motion Detection/Estimation , 1976. 1
[5] Y . Chen, X. Liu, and M. Yang. Bottom-up instance segmentation using deep higher-order crfs. In CVPR , 2015. 2
[6] M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler,
R. Benenson, U. Franke, S. Roth, and B. Schiele. The
Cityscapes Dataset for Semantic Urban Scene Understanding. In CVPR , 2016. 1, 5, 6
[7] J. Dai, K. He, and J. Sun. Instance-aware semantic segmentation via multi-task network cascades. In CVPR , 2016. 1,
2
[8] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn,
and A. Zisserman. The PASCAL Visual Object Classes


key: chroma:document
string_value:
and A. Zisserman. The PASCAL Visual Object Classes
Challenge 2012 (VOC2012) Results. http://www.pascalnetwork.org/challenges/VOC/voc2012/workshop/index.html.
6
[9] C. F. G. Ghiasi. Laplacian Pyramid Reconstruction and Reﬁnement for Semantic Segmentation. In ECCV , 2016. 4, 6,
7
[10] X. Glorot and Y . Bengio. Understanding the difﬁculty of
training deep feedforward neural networks. In AISTATS ,
2010. 5
[11] V . Grau, A. Mewes, M. Alcaiz, R. Kikinis, and S. Warﬁeld.
Improved watershed transform for medical image segmentation using prior information. IEEE Transactions on Medical
Imaging , 23(4):447–458, 2004. 1
[12] B. Hariharan, P. Arbelaez, R. Girshick, and J. Malik. Simultaneous detection and segmentation. In ECCV , 2014. 2
[13] K. Li, B. Hariharan, and J. Malik. Iterative instance segmentation. In CVPR , 2016. 2


key: chroma:document
string_value:
[13] K. Li, B. Hariharan, and J. Malik. Iterative instance segmentation. In CVPR , 2016. 2
[14] X. Liang, Y . Wei, X. Shen, Z. Jie, J. Feng, L. Lin, and S. Yan.
Reversible recursive instance-level object segmentation. In
CVPR , 2016. 2
[15] X. Liang, Y . Wei, X. Shen, J. Yang, L. Lin, and S. Yan. Endto-end instance segmentation and counting with recurrent attention. arXiv preprint arXiv:1509.02636 , 2015. 2
[16] J. Long, E. Shelhamer, and T. Darrell. Fully Convolutional
Networks for Semantic Segmentation. In CVPR , 2015. 4
[17] D. Martin, C. Fowlkes, D. Tal, and J. Malik. A database
of human segmented natural images and its application to
evaluating segmentation algorithms and measuring ecological statistics. In ICCV , 2001. 6
[18] F. Meyer. The watershed concept and its use in segmentation:
a brief history. arXiv preprint arXiv:1202.0216 , 2012. 3


key: chroma:document
string_value:
[18] F. Meyer. The watershed concept and its use in segmentation:
a brief history. arXiv preprint arXiv:1202.0216 , 2012. 3
[19] M. Minervini, A. Fischbach, H.Scharr, , and S. Tsaftaris.
Finely-grained annotated datasets for image-based plant phenotyping. Pattern Recognition Letters , pages 1–10, 2015. 6[20] P. O. Pinheiro, R. Collobert, and P. Dollr. Learning to segment object candidates. In NIPS , 2015. 1, 2
[21] P. O. Pinheiro, T.-Y . Lin, R. Collobert, and P. Dollr. Segmentation of 3d head mr images using morphological reconstruction under constraints and automatic selection of markers. In
ICIP , pages 1075–1078, 2001. 1
[22] P. O. Pinheiro, T.-Y . Lin, R. Collobert, and P. Dollr. Learning
to reﬁne object segments. In ECCV , 2016. 1, 2
[23] M. Ren and R. Zemel. End-to-end instance segmentation and counting with recurrent attention. arXiv preprint


key: chroma:document
string_value:
[23] M. Ren and R. Zemel. End-to-end instance segmentation and counting with recurrent attention. arXiv preprint
arXiv:1605.09410 , 2016. 1, 2
[24] B. Romera-Paredes and P. H. S. Torr. Recurrent Instance
Segmentation. In ECCV , 2016. 1, 2
[25] X. Shi, Z. Chen, H. Wang, D. Yeung, W. Wong, and W. Woo.
Convolutional LSTM network: A machine learning approach for precipitation nowcasting. In NIPS , 2015. 2
[26] N. Silberman, D. Sontag, and R. Fergus. Instance segmentation of indoor scenes using a coverage loss. In ECCV , 2014.
6, 7
[27] K. Simonyan and A. Zisserman. Very Deep Convolutional
Networks for Large-Scale Image Recognition. In ICLR ,
2015. 4, 5
[28] J. Uhrig, M. Cordts, U. Franke, and T. Brox. Pixel-level
Encoding and Depth Layering for Instance-level Semantic
Labeling. In GCPR , 2016. 1, 2, 6, 7


key: chroma:document
string_value:
Encoding and Depth Layering for Instance-level Semantic
Labeling. In GCPR , 2016. 1, 2, 6, 7
[29] J. van den Brand, M. Ochs, and R. Mester. Instance-level
Segmentation of Vehicles by Deep Contours. In ACCV Workshop , 2016. 6
[30] F. Yu and V . Koltun. Multi-Scale Context Aggregation by
Dilated Convolutions. In ICLR , 2016. 4
[31] S. Zagoruyko, A. Lerer, T.-Y . Lin, P. O. Pinheiro, S. Gross,
S. Chintala, and P. Doll ´ar. A multipath network for object
detection. In BMVC , 2016. 2
[32] Z. Zhang, S. Fidler, and R. Urtasun. Instance-Level Segmentation with Deep Densely Connected MRFs. In CVPR , 2016.
1, 2
[33] Z. Zhang, A. Schwing, S. Fidler, and R. Urtasun. Monocular
object instance segmentation and depth ordering with cnns.
InICCV , 2015. 1, 2


key: chroma:document
string_value:
object instance segmentation and depth ordering with cnns.
InICCV , 2015. 1, 2
[34] H. Zhao, J. Shi, X. Qi, X. Wang, and J. Jia. Pyramid Scene
Parsing Network. arXiv preprint arXiv:1612.01105 , 2016. 2,
3, 4, 6, 7, 8


key: chroma:document
string_value:
Interaction Networks for Learning about Objects,
Relations and Physics
Peter W. Battaglia
Google DeepMind
London, UK N1C 4AG
peterbattaglia@google.comRazvan Pascanu
Google DeepMind
London, UK N1C 4AG
razp@google.comMatthew Lai
Google DeepMind
London, UK N1C 4AG
matthewlai@google.com
Danilo Rezende
Google DeepMind
London, UK N1C 4AG
danilor@google.comKoray Kavukcuoglu
Google DeepMind
London, UK N1C 4AG
korayk@google.com
Abstract
Reasoning about objects, relations, and physics is central to human intelligence, and
a key goal of artiﬁcial intelligence. Here we introduce the interaction network , a
model which can reason about how objects in complex systems interact, supporting
dynamical predictions, as well as inferences about the abstract properties of the
system. Our model takes graphs as input, performs object- and relation-centric
reasoning in a way that is analogous to a simulation, and is implemented using
deep neural networks. We evaluate its ability to reason about several challenging
physical domains: n-body problems, rigid-body collision, and non-rigid dynamics.


key: chroma:document
string_value:
deep neural networks. We evaluate its ability to reason about several challenging
physical domains: n-body problems, rigid-body collision, and non-rigid dynamics.
Our results show it can be trained to accurately simulate the physical trajectories of
dozens of objects over thousands of time steps, estimate abstract quantities such
as energy, and generalize automatically to systems with different numbers and
conﬁgurations of objects and relations. Our interaction network implementation
is the ﬁrst general-purpose, learnable physics engine, and a powerful general
framework for reasoning about object and relations in a wide variety of complex
real-world domains.
1 Introduction
Representing and reasoning about objects, relations and physics is a “core” domain of human common
sense knowledge [ 26], and among the most basic and important aspects of intelligence [ 28,15]. Many
everyday problems, such as predicting what will happen next in physical environments or inferring
underlying properties of complex scenes, are challenging because their elements can be composed
in combinatorially many possible arrangements. People can nevertheless solve such problems by
decomposing the scenario into distinct objects and relations, and reasoning about the consequences
of their interactions and dynamics. Here we introduce the interaction network – a model that can
perform an analogous form of reasoning about objects and relations in complex systems.


key: chroma:document
string_value:
decomposing the scenario into distinct objects and relations, and reasoning about the consequences
of their interactions and dynamics. Here we introduce the interaction network – a model that can
perform an analogous form of reasoning about objects and relations in complex systems.
Interaction networks combine three powerful approaches: structured models, simulation, and deep


key: chroma:document
string_value:
perform an analogous form of reasoning about objects and relations in complex systems.
Interaction networks combine three powerful approaches: structured models, simulation, and deep
learning. Structured models [ 7] can exploit rich, explicit knowledge of relations among objects,
independent of the objects themselves, which supports general-purpose reasoning across diverse
contexts. Simulation is an effective method for approximating dynamical systems, predicting how the
elements in a complex system are inﬂuenced by interactions with one another, and by the dynamics
of the system. Deep learning [ 24,16] couples generic architectures with efﬁcient optimization
algorithms to provide highly scalable learning and inference in challenging real-world settings.arXiv:1612.00222v1  [cs.AI]  1 Dec 2016
Object reasoning Relational reasoning
Compute interaction Apply object dynamicsEffectsObjects,
relationsPredictions, 
inferences
a.
b.Figure 1: Schematic of an interaction network. a.For physical reasoning, the model takes objects and relations
as input, reasons about their interactions, and applies the effects and physical dynamics to predict new states. b.
For more complex systems, the model takes as input a graph that represents a system of objects, oj, and relations,


key: chroma:document
string_value:
For more complex systems, the model takes as input a graph that represents a system of objects, oj, and relations,
hi; j; r kik, instantiates the pairwise interaction terms, bk, and computes their effects, ek, via a relational model,
fR(). The ekare then aggregated and combined with the ojand external effects, xj, to generate input (as cj),
for an object model, fO(), which predicts how the interactions and dynamics inﬂuence the objects, p.
Interaction networks explicitly separate how they reason about relations from how they reason about
objects, assigning each task to distinct models which are: fundamentally object- and relation-centric;
and independent of the observation modality and task speciﬁcation (see Model section 2 below
and Fig. 1a). This lets interaction networks automatically generalize their learning across variable
numbers of arbitrarily ordered objects and relations, and also recompose their knowledge of entities
and interactions in novel and combinatorially many ways. They take relations as explicit input,
allowing them to selectively process different potential interactions for different input data, rather
than being forced to consider every possible interaction or those imposed by a ﬁxed architecture.


key: chroma:document
string_value:
allowing them to selectively process different potential interactions for different input data, rather
than being forced to consider every possible interaction or those imposed by a ﬁxed architecture.
We evaluate interaction networks by testing their ability to make predictions and inferences about various physical systems, including n-body problems, and rigid-body collision, and non-rigid dynamics.
Our interaction networks learn to capture the complex interactions that can be used to predict future
states and abstract physical properties, such as energy. We show that they can roll out thousands of
realistic future state predictions, even when trained only on single-step predictions. We also explore
how they generalize to novel systems with different numbers and conﬁgurations of elements. Though
they are not restricted to physical reasoning, the interaction networks used here represent the ﬁrst
general-purpose learnable physics engine, and even have the potential to learn novel physical systems
for which no physics engines currently exist.
Related work Our model draws inspiration from previous work that reasons about graphs and
relations using neural networks. The “graph neural network” [ 23,19] is a framework that shares
learning across nodes and edges, the “recursive autoencoder” [ 25] adapts its processing architecture
to exploit an input parse tree, the “neural programmer-interpreter” [ 22] is a composable network that


key: chroma:document
string_value:
to exploit an input parse tree, the “neural programmer-interpreter” [ 22] is a composable network that
mimics the execution trace of a program, and the “spatial transformer” [ 11] learns to dynamically
modify network connectivity to capture certain types of interactions. Others have explored deep
learning of logical and arithmetic relations [ 27], and relations suitable for visual question-answering
[1].
The behavior of our model is similar in spirit to a physical simulation engine [ 2], which generates
sequences of states by repeatedly applying rules that approximate the effects of physical interactions
and dynamics on objects over time. The interaction rules are relation-centric, operating on two or
more objects that are interacting, and the dynamics rules are object-centric, operating on individual
objects and the aggregated effects of the interactions they participate in. A similar approach has been
developed independently by Chang et al. [4].
Previous AI work on physical reasoning explored commonsense knowledge, qualitative representations, and simulation techniques for approximating physical prediction and inference [ 29,9] . The
“NeuroAnimator” [ 8] was perhaps the ﬁrst quantitative approach to learning physical dynamics, by
training neural networks to predict and control the state of articulated bodies. Ladický et al. [ 14]


