# embedding_metadata

key: chroma:document
string_value:
training neural networks to predict and control the state of articulated bodies. Ladický et al. [ 14]
recently used regression forests to learn ﬂuid dynamics. Recent advances in convolutional neural
networks (CNNs) have led to efforts that learn to predict coarse-grained physical dynamics from
2
images [ 20,17,18]. Notably, Fragkiadaki et al. [ 6] used CNNs to predict and control a moving
ball from an image centered at its coordinates. Mottaghi et al. [ 21] trained CNNs to predict the 3D
trajectory of an object after an external impulse is applied. Wu et al. [ 30] used CNNs to parse objects
from images, which were then input to a physics engine that supported prediction and inference.
2 Model
Deﬁnition To describe our model, we use physical reasoning as an example (Fig. 1a), and build
from a simple model to the full interaction network (abbreviated IN). To predict the dynamics of a
single object, one might use an object-centric function, fO, which inputs the object’s state, ot, at
timet, and outputs a future state, ot+1. If two or more objects are governed by the same dynamics,
fOcould be applied to each, independently, to predict their respective future states. But if the


key: chroma:document
string_value:
fOcould be applied to each, independently, to predict their respective future states. But if the
objects interact with one another, then fOis insufﬁcient because it does not capture their relationship.
Assuming two objects and one directed relationship, e.g., a ﬁxed object attached by a spring to a freely
moving mass, the ﬁrst (the sender ,o1) inﬂuences the second (the receiver ,o2) via their interaction.
The effect of this interaction, et+1, can be predicted by a relation-centric function, fR. ThefRtakes
as inputo1,o2, as well as attributes of their relationship, r, e.g., the spring constant. The fOis
modiﬁed so it can input both et+1and the receiver’s current state, o2;t, enabling the interaction to
inﬂuence its future state, o2;t+1,
et+1=fR(o1;t;o2;t;r) o2;t+1=fO(o2;t;et+1)
The above formulation can be expanded to larger and more complex systems by representing them


key: chroma:document
string_value:
The above formulation can be expanded to larger and more complex systems by representing them
as a graph,G=hO;Ri, where the nodes, O, correspond to the objects, and the edges, R, to the
relations (see Fig. 1b). We assume an attributed, directed multigraph because the relations have
attributes, and there can be multiple distinct relations between two objects (e.g., rigid and magnetic
interactions). For a system with NOobjects andNRrelations, the inputs to the IN are,
O=fojgj=1:::NO; R=fhi;j;r kikgk=1:::NRwherei6=j;1i;jNO; X=fxjgj=1:::NO
TheOrepresents the states of each object. The triplet, hi;j;r kik, represents the k-th relation in the
system, from sender, oi, to receiver, oj, with relation attribute, rk. TheXrepresents external effects,
such as active control inputs or gravitational acceleration, which we deﬁne as not being part of the
system, and which are applied to each object separately.
The basic IN is deﬁned as,


key: chroma:document
string_value:
system, and which are applied to each object separately.
The basic IN is deﬁned as,
IN(G) =O(a(G; X;  R(m(G) ) )) (1)
m(G) =B=fbkgk=1:::NR
fR(bk) =ek
R(B) =E=fekgk=1:::NRa(G;X;E ) =C=fcjgj=1:::NO
fO(cj) =pj
O(C) =P=fpjgj=1:::NO(2)
The marshalling function, m, rearranges the objects and relations into interaction terms, bk=
hoi;oj;rki2B, one per relation, which correspond to each interaction’s receiver, sender, and
relation attributes. The relational model, R, predicts the effect of each interaction, ek2E, by
applyingfRto eachbk. The aggregation function, a, collects all effects, ek2E, that apply to each
receiver object, merges them, and combines them with OandXto form a set of object model inputs,


key: chroma:document
string_value:
receiver object, merges them, and combines them with OandXto form a set of object model inputs,
cj2C, one per object. The object model, O, predicts how the interactions and dynamics inﬂuence
the objects by applying fOto eachcj, and returning the results, pj2P. This basic IN can predict
the evolution of states in a dynamical system – for physical simulation, Pmay equal the future states
of the objects, Ot+1.
The IN can also be augmented with an additional component to make abstract inferences about the
system. The pj2P, rather than serving as output, can be combined by another aggregation function,
g, and input to an abstraction model, A, which returns a single output, q, for the whole system. We
explore this variant in our ﬁnal experiments that use the IN to predict potential energy.
An IN applies the same fRandfOto everybkandcj, respectively, which makes their relational and
object reasoning able to handle variable numbers of arbitrarily ordered objects and relations. But
one additional constraint must be satisﬁed to maintain this: the afunction must be commutative and
3
associative over the objects and relations. Using summation within ato merge the elements of Einto


key: chroma:document
string_value:
3
associative over the objects and relations. Using summation within ato merge the elements of Einto
Csatisﬁes this, but division would not.
Here we focus on binary relations, which means there is one interaction term per relation, but another
option is to have the interactions correspond to n-th order relations by combining nsenders in each bk.
The interactions could even have variable order, where each bkincludes all sender objects that interact
with a receiver, but would require a fRthan can handle variable-length inputs. These possibilities are
beyond the scope of this work, but are interesting future directions.
Implementation The general deﬁnition of the IN in the previous section is agnostic to the choice
of functions and algorithms, but we now outline a learnable implementation capable of reasoning
about complex systems with nonlinear relations and dynamics. We use standard deep neural network
building blocks, multilayer perceptrons (MLP), matrix operations, etc., which can be trained efﬁciently
from data using gradient-based optimization, such as stochastic gradient descent.
We deﬁneOas aDSNOmatrix, whose columns correspond to the objects’ DS-length state vectors.


key: chroma:document
string_value:
We deﬁneOas aDSNOmatrix, whose columns correspond to the objects’ DS-length state vectors.
The relations are a triplet, R=hRr;Rs;Rai, whereRrandRsareNONRbinary matrices which
index the receiver and sender objects, respectively, and Rais aDRNRmatrix whose DR-length
columns represent the NRrelations’ attributes. The j-th column of Rris a one-hot vector which
indicates the receiver object’s index; Rsindicates the sender similarly. For the graph in Fig. 1b,
Rr=h0 0
1 1
0 0i
andRs=h1 0
0 0
0 1i
. TheXis aDXNOmatrix, whose columns are DX-length vectors
that represent the external effect applied each of the NOobjects.
The marshalling function, m, computes the matrix products, ORrandORs, and concatenates them
withRa:m(G) = [ORr;ORs;Ra] =B.
The resulting Bis a(2DS+DR)NRmatrix, whose columns represent the interaction terms, bk,
for theNRrelations (we denote vertical and horizontal matrix concatenation with a semicolon and


key: chroma:document
string_value:
for theNRrelations (we denote vertical and horizontal matrix concatenation with a semicolon and
comma, respectively). The way mconstructs interaction terms can be modiﬁed, as described in our
Experiments section (3).
TheBis input toR, which applies fR, an MLP, to each column. The output of fRis aDE-length
vector,ek, a distributed representation of the effects. The Rconcatenates the NReffects to form the
DENReffect matrix, E.
TheG,X, andEare input toa, which computes the DENOmatrix product, E=ERT
r, whose
j-th column is equivalent to the elementwise sum across all ekwhose corresponding relation has
receiver object, j. The Eis concatenated with OandX:a(G;X;E ) = [O;X;E] =C.
The resulting Cis a(DS+DX+DE)NOmatrix, whose NOcolumns represent the object states,
external effects, and per-object aggregate interaction effects.
TheCis input toO, which applies fO, another MLP, to each of the NOcolumns. The output of fO


key: chroma:document
string_value:
TheCis input toO, which applies fO, another MLP, to each of the NOcolumns. The output of fO
is aDP-length vector, pj, andOconcatenates them to form the output matrix, P.
To infer abstract properties of a system, an additional Ais appended and takes Pas input. The g
aggregation function performs an elementwise sum across the columns of Pto return aDP-length
vector, P. The Pis input toA, another MLP, which returns a DA-length vector, q, that represents
an abstract, global property of the system.
Training an IN requires optimizing an objective function over the learnable parameters of RandO.
Note,mandainvolve matrix operations that do not contain learnable parameters.
BecauseRandOare shared across all relations and objects, respectively, training them is statistically efﬁcient. This is similar to CNNs, which are very efﬁcient due to their weight-sharing scheme.
A CNN treats a local neighborhood of pixels as related, interacting entities: each pixel is effectively
a receiver object and its neighboring pixels are senders. The convolution operator is analogous to
R, wherefRis the local linear/nonlinear kernel applied to each neighborhood. Skip connections,


key: chroma:document
string_value:
R, wherefRis the local linear/nonlinear kernel applied to each neighborhood. Skip connections,
recently popularized by residual networks, are loosely analogous to how the IN inputs Oto both
RandO, though in CNNs relation- and object-centric reasoning are not delineated. But because
CNNs exploit local interactions in a ﬁxed way which is well-suited to the speciﬁc topology of images,
capturing longer-range dependencies requires either broad, insensitive convolution kernels, or deep
stacks of layers, in order to implement sufﬁciently large receptive ﬁelds. The IN avoids this restriction
by being able to process arbitrary neighborhoods that are explicitly speciﬁed by the Rinput.
4
3 Experiments
Physical reasoning tasks Our experiments explored two types of physical reasoning tasks: predicting future states of a system, and estimating their abstract properties, speciﬁcally potential energy.
We evaluated the IN’s ability to learn to make these judgments in three complex physical domains:
n-body systems; balls bouncing in a box; and strings composed of springs that collide with rigid
objects. We simulated the 2D trajectories of the elements of these systems with a physics engine, and
recorded their sequences of states. See the Supplementary Material for full details.


key: chroma:document
string_value:
objects. We simulated the 2D trajectories of the elements of these systems with a physics engine, and
recorded their sequences of states. See the Supplementary Material for full details.
In the n-body domain, such as solar systems, all nbodies exert distance- and mass-dependent
gravitational forces on each other, so there were n(n1)relations input to our model. Across
simulations, the objects’ masses varied, while all other ﬁxed attributes were held constant. The
training scenes always included 6 bodies, and for testing we used 3, 6, and 12 bodies. In half of
the systems, bodies were initialized with velocities that would cause stable orbits, if not for the
interactions with other objects; the other half had random velocities.
In the bouncing balls domain, moving balls could collide with each other and with static walls.
The walls were represented as objects whose shape attribute represented a rectangle, and whose
inverse-mass was 0. The relations input to the model were between the nobjects (which included the
walls), for (n(n1)relations). Collisions are more difﬁcult to simulate than gravitational forces, and
the data distribution was much more challenging: each ball participated in a collision on less than 1%


key: chroma:document
string_value:
the data distribution was much more challenging: each ball participated in a collision on less than 1%
of the steps, following straight-line motion at all other times. The model thus had to learn that despite
there being a rigid relation between two objects, they only had meaningful collision interactions when
they were in contact. We also varied more of the object attributes – shape, scale and mass (as before)
– as well as the coefﬁcient of restitution, which was a relation attribute. Training scenes contained 6
balls inside a box with 4 variably sized walls, and test scenes contained either 3, 6, or 9 balls.
The string domain used two types of relations (indicated in rk), relation structures that were more
sparse and speciﬁc than all-to-all, as well as variable external effects. Each scene contained a string,
comprised of masses connected by springs, and a static, rigid circle positioned below the string. The
nmasses had spring relations with their immediate neighbors ( 2(n1)), and all masses had rigid
relations with the rigid object ( 2n). Gravitational acceleration, with a magnitude that was varied
across simulation runs, was applied so that the string always fell, usually colliding with the static
object. The gravitational acceleration was an external input (not to be confused with the gravitational


key: chroma:document
string_value:
across simulation runs, was applied so that the string always fell, usually colliding with the static
object. The gravitational acceleration was an external input (not to be confused with the gravitational
attraction relations in the n-body experiments). Each training scene contained a string with 15 point
masses, and test scenes contained either 5, 15, or 30 mass strings. In training, one of the point masses
at the end of the string, chosen at random, was always held static, as if pinned to the wall, while the
other masses were free to move. In the test conditions, we also included strings that had both ends
pinned, and no ends pinned, to evaluate generalization.
Our model takes as input the state of each system, G, decomposed into the objects, O(e.g., n-body
objects, balls, walls, points masses that represented string elements), and their physical relations, R
(e.g., gravitational attraction, collisions, springs), as well as the external effects, X(e.g., gravitational
acceleration). Each object state, oj, could be further divided into a dynamic state component
(e.g., position and velocity) and a static attribute component (e.g., mass, size, shape). The relation
attributes,Ra, represented quantities such as the coefﬁcient of restitution, and spring constant. The


key: chroma:document
string_value:
attributes,Ra, represented quantities such as the coefﬁcient of restitution, and spring constant. The
input represented the system at the current time. The prediction experiment’s target outputs were the
velocities of the objects on the subsequent time step, and the energy estimation experiment’s targets
were the potential energies of the system on the current time step. We also generated multi-step
rollouts for the prediction experiments (Fig. 2), to assess the model’s effectiveness at creating visually
realistic simulations. The output velocity, vt, on time step tbecame the input velocity on t+ 1, and
the position at t+ 1was updated by the predicted velocity at t.
Data Each of the training, validation, test data sets were generated by simulating 2000 scenes
over 1000 time steps, and randomly sampling 1 million, 200k, and 200k one-step input/target pairs,
respectively. The model was trained for 2000 epochs, randomly shufﬂing the data indices between
each. We used mini-batches of 100, and balanced their data distributions so the targets had similar
per-element statistics. The performance reported in the Results was measured on held-out test data.
We explored adding a small amount of Gaussian noise to 20% of the data’s input positions and


key: chroma:document
string_value:
We explored adding a small amount of Gaussian noise to 20% of the data’s input positions and
velocities during the initial phase of training, which was reduced to 0% from epochs 50 to 250. The
5
noise std. dev. was 0:05the std. dev. of each element’s values across the dataset. It allowed the
model to experience physically impossible states which could not have been generated by the physics
engine, and learn to project them back to nearby, possible states. Our error measure did not reﬂect
clear differences with or without noise, but rollouts from models trained with noise were slightly
more visually realistic, and static objects were less subject to drift over many steps.
Model architecture ThefRandfOMLPs contained multiple hidden layers of linear transforms
plus biases, followed by rectiﬁed linear units (ReLUs), and an output layer that was a linear transform
plus bias. The best model architecture was selected by a grid search over layer sizes and depths. All
inputs (except RrandRs) were normalized by centering at the median and rescaling the 5th and 95th
percentiles to -1 and 1. All training objectives and test measures used mean squared error (MSE)
between the model’s prediction and the ground truth target.


key: chroma:document
string_value:
percentiles to -1 and 1. All training objectives and test measures used mean squared error (MSE)
between the model’s prediction and the ground truth target.
All prediction experiments used the same architecture, with parameters selected by a hyperparameter
search. The fRMLP had four, 150-length hidden layers, and output length DE= 50 . ThefOMLP
had one, 100-length hidden layer, and output length DP= 2, which targeted the x;y-velocity. The
mandawere customized so that the model was invariant to the absolute positions of objects in the
scene. Themconcatenated three terms for each bk: the difference vector between the dynamic states
of the receiver and sender, the concatenated receiver and sender attribute vectors, and the relation
attribute vector. The aonly outputs the velocities, not the positions, for input to O.
The energy estimation experiments used the IN from the prediction experiments with an additional
AMLP which had one, 25-length hidden layer. Its Pinputs’ columns were length DP= 10 , and
its output length was DA= 1.
We optimized the parameters using Adam [ 13], with a waterfall schedule that began with a learning
rate of 0:001and down-scaled the learning rate by 0:8each time the validation error, estimated over


key: chroma:document
string_value:
rate of 0:001and down-scaled the learning rate by 0:8each time the validation error, estimated over
a window of 40epochs, stopped decreasing.
Two forms of L2 regularization were explored: one applied to the effects, E, and another to the model
parameters. Regularizing Eimproved generalization to different numbers of objects and reduced
drift over many rollout steps. It likely incentivizes sparser communication between the RandO,
prompting them to operate more independently. Regularizing the parameters generally improved
performance and reduced overﬁtting. Both penalty factors were selected by a grid search.
Few competing models are available in the literature to compare our model against, but we considered
several alternatives: a constant velocity baseline which output the input velocity; an MLP baseline,
with two 300-length hidden layers, which took as input a ﬂattened vector of all of the input data; and
a variant of the IN with the Rcomponent removed (the interaction effects, E, was set to a 0-matrix).
4 Results
Prediction experiments Our results show that the IN can predict the next-step dynamics of our task
domains very accurately after training, with orders of magnitude lower test error than the alternative


key: chroma:document
string_value:
4 Results
Prediction experiments Our results show that the IN can predict the next-step dynamics of our task
domains very accurately after training, with orders of magnitude lower test error than the alternative
models (Fig. 3a, d and g, and Table 2). Because the dynamics of each domain depended crucially on
interactions among objects, the IN was able to learn to exploit these relationships for its predictions.
The dynamics-only IN had no mechanism for processing interactions, and performed similarly to the
constant velocity model. The baseline MLP’s connectivity makes it possible, in principle, for it to
learn the interactions, but that would require learning how to use the relation indices to selectively
process the interactions. It would also not beneﬁt from sharing its learning across relations and
objects, instead being forced to approximate the interactive dynamics in parallel for each objects.
The IN also generalized well to systems with fewer and greater numbers of objects (Figs. 3b-c, e-f
and h-k, and Table SM1 in Supp. Mat.). For each domain, we selected the best IN model from the
system size on which it was trained, and evaluated its MSE on a different system size. When tested
on smaller n-body and spring systems from those on which it was trained, its performance actually


key: chroma:document
string_value:
on smaller n-body and spring systems from those on which it was trained, its performance actually
exceeded a model trained on the smaller system. This may be due to the model’s ability to exploit its
greater experience with how objects and relations behave, available in the more complex system.
We also found that the IN trained on single-step predictions can be used to simulate trajectories over
thousands of steps very effectively, often tracking the ground truth closely, especially in the n-body
and string domains. When rendered into images and videos, the model-generated trajectories are
6
True Model True Model True ModelTime Time TimeFigure 2: Prediction rollouts. Each column contains three panels of three video frames (with motion blur),
each spanning 1000 rollout steps. Columns 1-2 are ground truth and model predictions for n-body systems, 3-4
are bouncing balls, and 5-6 are strings. Each model column was generated by a single model, trained on the
underlying states of a system of the size in the top panel. The middle and bottom panels show its generalization
to systems of different sizes and structure. For n-body, the training was on 6 bodies, and generalization was to 3
and 12 bodies. For balls, the training was on 6 balls, and generalization was to 3 and 9 balls. For strings, the


key: chroma:document
string_value:
and 12 bodies. For balls, the training was on 6 balls, and generalization was to 3 and 9 balls. For strings, the
training was on 15 masses with 1 end pinned, and generalization was to 30 masses with 0 and 2 ends pinned.
The URLs to the full videos of each rollout are in Table 1.
7
Table 1: Rollout video URLs
System True Model
n-body 6 https://youtu.be/otIGNTFJwpU https://youtu.be/CXzubiwu4GM
n-body 3 https://youtu.be/zOTVzziJz24 https://youtu.be/3bFrbjdTc6s
n-body 12 https://youtu.be/JgSRaYmQyNY https://youtu.be/Rl44jdaIIiw
Balls 6 https://youtu.be/9c-UPXSjOHI https://youtu.be/SV741g_qp8M
Balls 3 https://youtu.be/op2d9IHh3Ak https://youtu.be/i0lOn_v8xVg
Balls 9 https://youtu.be/oQoquoVJ8V0 https://youtu.be/41y_saKvi3s


key: chroma:document
string_value:
Balls 9 https://youtu.be/oQoquoVJ8V0 https://youtu.be/41y_saKvi3s
String 15, 1 pinned https://youtu.be/xiz49xOPX9c https://youtu.be/pbk6VM_XQb4
String 30, 0 pinned https://youtu.be/TpixFr_ekAE https://youtu.be/tsE7cev0w08
String 30, 2 pinned https://youtu.be/x6g1ogRG31o https://youtu.be/ZDCZyqyrv5o
10-2
10-3g. 15, 1h. 5, 1i. 30, 1j. 15, 0k. 15, 2String
1
10-110102MSE (log-scale)a. 6b. 3c. 12n-body
10-210-1
10-3d. 6e. 3f. 9Balls
IN (15 obj, 1 pin) IN (5 obj, 1 pin)
IN (15 obj, 0 pin)IN (30 obj, 1 pin)
IN (15 obj, 2 pin)IN (3 obj) IN (12 obj) IN (6 obj)


key: chroma:document
string_value:
IN (15 obj, 0 pin)IN (30 obj, 1 pin)
IN (15 obj, 2 pin)IN (3 obj) IN (12 obj) IN (6 obj)
Constant velocity Baseline MLP Dynamics-only ININ (3 obj) IN (9 obj) IN (6 obj)10-2
Figure 3: Prediction experiment accuracy and generalization. Each colored bar represents the MSE between a
model’s predicted velocity and the ground truth physics engine’s (the y-axes are log-scaled). Sublots ( a-c) show
n-body performance, ( d-f) show balls, and ( g-k) show string. The leftmost subplots in each ( a, d, g ) for each
domain compare the constant velocity model (black), baseline MLP (grey), dynamics-only IN (red), and full IN
(blue). The other panels show the IN’s generalization performance to different numbers and conﬁgurations of
objects, as indicated by the subplot titles. For the string systems, the numbers correspond to: (the number of
masses, how many ends were pinned).
usually visually indistinguishable from those of the ground truth physics engine (Fig. 2; see Table 1
for URLs to the full videos of each rollout). This is not to say that given the same initial conditions,


key: chroma:document
string_value:
for URLs to the full videos of each rollout). This is not to say that given the same initial conditions,
they cohere perfectly: the dynamics are highly nonlinear and imperceptible prediction errors by the
model can rapidly lead to large differences in the systems’ states. But the incoherent rollouts do
not violate people’s expectations, and might be roughly on par with people’s understanding of these
domains.
Estimating abstract properties We trained an abstract-estimation variant of our model to predict
potential energies in the n-body and string domains (the ball domain’s potential energies were always
0), and found it was much more accurate (n-body MSE 1:4, string MSE 1:1) than the MLP baseline
(n-body MSE 19, string MSE 425). The IN presumably learns the gravitational and spring potential
energy functions, applies them to the relations in their respective domains, and combines the results.
5 Discussion
We introduced interaction networks as a ﬂexible and efﬁcient model for explicit reasoning about
objects and relations in complex systems. Our results provide surprisingly strong evidence of their
ability to learn accurate physical simulations and generalize their training to novel systems with
different numbers and conﬁgurations of objects and relations. They could also learn to infer abstract


key: chroma:document
string_value:
ability to learn accurate physical simulations and generalize their training to novel systems with
different numbers and conﬁgurations of objects and relations. They could also learn to infer abstract
properties of physical systems, such as potential energy. The alternative models we tested performed
much more poorly, with orders of magnitude greater error. Simulation over rich mental models is
thought to be a crucial mechanism of how humans reason about physics and other complex domains
[5,12,10], and Battaglia et al. [ 3] recently posited a simulation-based “intuitive physics engine”
model to explain human physical scene understanding. Our interaction network implementation is the
ﬁrst learnable physics engine that can scale up to real-world problems, and is a promising template for
new AI approaches to reasoning about other physical and mechanical systems, scene understanding,
social perception, hierarchical planning, and analogical reasoning.
In the future, it will be important to develop techniques that allow interaction networks to handle
very large systems with many interactions, such as by culling interaction computations that will have
8
Table 2: Prediction experiment MSEs
Domain Constant velocity Baseline Dynamics-only IN IN
n-body 82 79 76 0.25
Balls 0.074 0.072 0.074 0.0020
String 0.018 0.016 0.017 0.0011


key: chroma:document
string_value:
Balls 0.074 0.072 0.074 0.0020
String 0.018 0.016 0.017 0.0011
negligible effects. The interaction network may also serve as a powerful model for model-predictive
control inputting active control signals as external effects – because it is differentiable, it naturally
supports gradient-based planning. It will also be important to prepend a perceptual front-end that
can infer from objects and relations raw observations, which can then be provided as input to an
interaction network that can reason about the underlying structure of a scene. By adapting the
interaction network into a recurrent neural network, even more accurate long-term predictions might
be possible, though preliminary tests found little beneﬁt beyond its already-strong performance.
By modifying the interaction network to be a probabilistic generative model, it may also support
probabilistic inference over unknown object properties and relations.
By combining three powerful tools from the modern machine learning toolkit – relational reasoning
over structured knowledge, simulation, and deep learning – interaction networks offer ﬂexible,
accurate, and efﬁcient learning and inference in challenging domains. Decomposing complex
systems into objects and relations, and reasoning about them explicitly, provides for combinatorial


key: chroma:document
string_value:
accurate, and efﬁcient learning and inference in challenging domains. Decomposing complex
systems into objects and relations, and reasoning about them explicitly, provides for combinatorial
generalization to novel contexts, one of the most important future challenges for AI, and a crucial
step toward closing the gap between how humans and machines think.
References
[1]J Andreas, M Rohrbach, T Darrell, and D Klein. Learning to compose neural networks for question
answering. NAACL , 2016.
[2]D Baraff. Physically based modeling: Rigid body simulation. SIGGRAPH Course Notes, ACM SIGGRAPH ,
2(1), 2001.
[3]PW Battaglia, JB Hamrick, and JB Tenenbaum. Simulation as an engine of physical scene understanding.
Proceedings of the National Academy of Sciences , 110(45):18327–18332, 2013.
[4]Michael B. Chang, Tomer Ullman, Antonio Torralba, and Joshua B. Tenenbaum. A compositional
object-based approach to learning physical dynamics. Under Review , 2016.
[5] K.J.W. Craik. The nature of explanation . Cambridge University Press, 1943.


key: chroma:document
string_value:
object-based approach to learning physical dynamics. Under Review , 2016.
[5] K.J.W. Craik. The nature of explanation . Cambridge University Press, 1943.
[6]K Fragkiadaki, P Agrawal, S Levine, and J Malik. Learning visual predictive models of physics for playing
billiards. ICLR , 2016.
[7]Z. Ghahramani. Probabilistic machine learning and artiﬁcial intelligence. Nature , 521(7553):452–459,
2015.
[8]R Grzeszczuk, D Terzopoulos, and G Hinton. Neuroanimator: Fast neural network emulation and control of
physics-based models. In Proceedings of the 25th annual conference on Computer graphics and interactive
techniques , pages 9–20. ACM, 1998.
[9]P.J Hayes. The naive physics manifesto . Université de Genève, Institut pour les études s é mantiques et
cognitives, 1978.
[10] M. Hegarty. Mechanical reasoning by mental simulation. TICS , 8(6):280–285, 2004.
[11] M Jaderberg, K Simonyan, and A Zisserman. Spatial transformer networks. In NIPS , pages 2008–2016,
2015.


key: chroma:document
string_value:
[11] M Jaderberg, K Simonyan, and A Zisserman. Spatial transformer networks. In NIPS , pages 2008–2016,
2015.
[12] P.N. Johnson-Laird. Mental models: towards a cognitive science of language, inference, and consciousness ,
volume 6. Cambridge University Press, 1983.
[13] D. Kingma and J. Ba. Adam: A method for stochastic optimization. ICLR , 2015.
[14] L Ladický, S Jeong, B Solenthaler, M Pollefeys, and M Gross. Data-driven ﬂuid simulations using
regression forests. ACM Transactions on Graphics (TOG) , 34(6):199, 2015.
[15] B Lake, T Ullman, J Tenenbaum, and S Gershman. Building machines that learn and think like people.
arXiv:1604.00289 , 2016.
[16] Y LeCun, Y Bengio, and G Hinton. Deep learning. Nature , 521(7553):436–444, 2015.
9
[17] A Lerer, S Gross, and R Fergus. Learning physical intuition of block towers by example. arXiv:1603.01312 ,
2016.


key: chroma:document
string_value:
9
[17] A Lerer, S Gross, and R Fergus. Learning physical intuition of block towers by example. arXiv:1603.01312 ,
2016.
[18] W Li, S Azimi, A Leonardis, and M Fritz. To fall or not to fall: A visual approach to physical stability
prediction. arXiv:1604.00066 , 2016.
[19] Yujia Li, Daniel Tarlow, Marc Brockschmidt, and Richard Zemel. Gated graph sequence neural networks.
arXiv preprint arXiv:1511.05493 , 2015.
[20] R Mottaghi, H Bagherinezhad, M Rastegari, and A Farhadi. Newtonian image understanding: Unfolding
the dynamics of objects in static images. arXiv:1511.04048 , 2015.
[21] R Mottaghi, M Rastegari, A Gupta, and A Farhadi. " what happens if..." learning to predict the effect of
forces in images. arXiv:1603.05600 , 2016.
[22] SE Reed and N de Freitas. Neural programmer-interpreters. ICLR , 2016.


key: chroma:document
string_value:
[22] SE Reed and N de Freitas. Neural programmer-interpreters. ICLR , 2016.
[23] F. Scarselli, M. Gori, A.C. Tsoi, M. Hagenbuchner, and G. Monfardini. The graph neural network model.
IEEE Trans. Neural Networks , 20(1):61–80, 2009.
[24] J. Schmidhuber. Deep learning in neural networks: An overview. Neural Networks , 61:85–117, 2015.
[25] R Socher, E Huang, J Pennin, C Manning, and A Ng. Dynamic pooling and unfolding recursive autoencoders for paraphrase detection. In in NIPS , pages 801–809, 2011.
[26] E Spelke, K Breinlinger, J Macomber, and K Jacobson. Origins of knowledge. Psychol. Rev. , 99(4):605–
632, 1992.
[27] I Sutskever and GE Hinton. Using matrices to model symbolic relationship. In D. Koller, D. Schuurmans,
Y . Bengio, and L. Bottou, editors, NIPS , pages 1593–1600. 2009.


key: chroma:document
string_value:
Y . Bengio, and L. Bottou, editors, NIPS , pages 1593–1600. 2009.
[28] J.B. Tenenbaum, C. Kemp, T.L. Grifﬁths, and N.D. Goodman. How to grow a mind: Statistics, structure,
and abstraction. Science , 331(6022):1279, 2011.
[29] P Winston and B Horn. The psychology of computer vision , volume 73. McGraw-Hill New York, 1975.
[30] J Wu, I Yildirim, JJ Lim, B Freeman, and J Tenenbaum. Galileo: Perceiving physical object properties by
integrating a physics engine with deep learning. In NIPS , pages 127–135, 2015.
10
A Supplementary material
A.1 Experimental details
A.1.1 Physics engine details
Every simulated trajectory contained the states of the objects in the system on each frame of a sequence of
1000 one-millisecond time steps. The parameters of each system were chosen so that a diverse set of dynamics
could unfold within the trajectory. On each step, the physics engine took as input the current state of the
system, calculated the forces associated with each inter-entity interaction, and applied them to the individual


key: chroma:document
string_value:
system, calculated the forces associated with each inter-entity interaction, and applied them to the individual
entities as accelerations by dividing by the entity’s mass, parameterized as inverse-mass, a=Fm1in both the
engine and model input. This also allows static objects to be represented as having m1= 0. The previous
positions and velocities, and newly computed accelerations, were input to an Euler integrator to update the
current velocities and positions of the entities. By using a custom engine we were able to have perfect control
over the details of the simulation, and use one engine for all physical domains. It produced trajectory rollouts
that were indistinguishable from those of an off-the-shelf simulation engine, but was highly efﬁcient because it
could perform thousands of runs in parallel on a GPU, allowing millions of simulated steps to be generated each
second.
A.1.2 Physical domains
n-body All objects in n-body systems exerted gravitational forces on each other, which were a function of
the objects’ pairwise distances and masses, giving rise to very complex dynamics that are highly sensitive to
initial conditions. Across simulation runs, the objects’ masses varied, while all other non-dynamic variables
were held constant. The gravitational constant was set to ensure that objects could move across several hundred


key: chroma:document
string_value:
were held constant. The gravitational constant was set to ensure that objects could move across several hundred
meters within the 1000 step rollout. The training scenes always included 6 objects, and for testing we used 3,
6, and 12 objects. The masses were uniformly sampled from [0:02;9]kg, their shapes were points, and their
initial positions were randomly sampled from all angles, with a distance in [10;100] m. We included two classes
of scenes. The ﬁrst, orbit systems, had one object (the star), initialized at position (0;0), with zero velocity
and a mass of 100kg. The planets’ velocities were initialized such that they would have stable orbits around
the star, if not for their interactions with other planets. The second, non-orbit systems, sampled initial x- and
y-velocity components from [3;3]m/s. The objects would typically begin attracting, and gave rise to complex
semi-periodic behavior.
An n-body system is a highly nonlinear (chaotic) dynamical system, which means they are highly sensitive to
initial conditions and extended predictions of their states are not possible under even small perturbations. The
relations between the objects corresponded to gravitational attraction. Between simulation runs, the masses of


key: chroma:document
string_value:
initial conditions and extended predictions of their states are not possible under even small perturbations. The
relations between the objects corresponded to gravitational attraction. Between simulation runs, the masses of
the objects were varied, while all other non-dynamic variables were held constant (e.g., gravitational constant) or
were not meaningful (e.g., object scales and shapes). The gravitational force from object itojwas computed as,
Fij=Gmimj(xixj)
kxixjk3, where Gis the gravitational constant. The denominator was clipped so that forces could
not go too high as the distances between objects went to zero. All forces applied to each object were summed to
yield the per-object total forces.
Bouncing balls The bouncing balls domain still had all-to-all object relations–collisions–and any object
could collide with any other, including the walls. But colliding objects are more difﬁcult to simulate than the
gravitational interactions in n-body systems, and our bouncing balls domain also included more variability in
object attributes, such as shape, scale, and mass (as before), as well as relation attributes, such as the coefﬁcient
of restitution. The data distribution was much more challenging: for more than 99% of the time steps, a ball was


key: chroma:document
string_value:
of restitution. The data distribution was much more challenging: for more than 99% of the time steps, a ball was
not in contact with any others, and its next-step velocity equaled its current velocity. For the remaining steps,
however, collisions caused next-step velocities that was a complex function of its state and the state of the object
it collides with. The training scene contained 6 balls inside a box with 4 walls, and test scenes contained either 3,
6, or 9 balls. The balls’ radii were sampled uniformly from [0:1;0:3]m, and masses from [0:75;1:25]kg. The
walls were static rectangular boxes, positioned so that the horizontal and vertical lengths of area they enclosed
varied independently between [1;3]m. The balls’ initial positions were randomly sampled so that all balls ﬁt
within the box and did not interpenetrate any other object, with initial x- andy-velocity components sampled
uniformly from [5;5]m/s. The restitutions, an attribute of their collision relations, was sampled uniformly
from[0:4;1].
Rigid body collision systems are highly nonlinear (chaotic) dynamical systems, as well. Collision forces were


key: chroma:document
string_value:
from[0:4;1].
Rigid body collision systems are highly nonlinear (chaotic) dynamical systems, as well. Collision forces were
applied between objects when they began to interpenetrate, using two-step process: collision detection between
all objects, and resolution of detected collisions. A detected collision between two objects meant that their
shapes overlapped and their velocities were causing them to approach each other. To resolve a partially inelastic
collision, the post-collision velocities of each object were computed, and forces appropriate to effect these
velocities on the subsequent time step were then calculated and applied the the objects. This resulted in realistic
bouncing ball trajectories.
11
Table 3: Prediction experiments - Generalization MSEs
n-body Balls String
Model 3 12 3 9 5, 1 30, 1 15, 0 15, 2
Const. vel. 45 166 0.054 0.084 0.0080 0.010 0.020 0.012
IN (within) 0.21 5.2 0.0053 0.0019 0.0030 0.00066 0.0010 0.00097
IN (transfer) 0.052 7.8 0.0053 0.004 0.00052 0.00074 0.0014 0.00097


key: chroma:document
string_value:
IN (transfer) 0.052 7.8 0.0053 0.004 0.00052 0.00074 0.0014 0.00097
String Our string domain used multiple types of relations (springs and collisions), relation structures that were
more sparse and speciﬁc than all-to-all, and variable external effects. Each scene contained one string, comprised
of point masses connected by springs, and one static, rigid circle positioned below the string. Gravitational
acceleration, varied across simulation runs, was applied so that the string always fell, usually colliding with
the static object. Each training scene contained a string with 15 point masses, and test scenes contained either
5, 15, or 30 mass strings. In training, one of the point masses at the end of the string, chosen at random, was
always held static, as if pinned to the wall, while the other masses were free to move. In the test conditions, we
also included strings that had both ends pinned, and no ends pinned, to evaluate generalization. The string’s
masses were sampled from [0:05;0:15]kg. The springs’ rest lengths were set to 0:2m, spring constants to 100,


key: chroma:document
string_value:
masses were sampled from [0:05;0:15]kg. The springs’ rest lengths were set to 0:2m, spring constants to 100,
and damping factors to 0:001. The static object’s x-position was sampled from [0:5;0:5]m,y-position was
sampled from [1;0:5]m, and radius from [0:2;0:4]m. The string mass-rigid object coefﬁcient of restitution
was sampled from [0;1]. The gravitational acceleration applied to all non-static objects was sampled from
[30;5]m/s2.
The spring force from object itojwas computed using Hooke’s law as, F=Cs(1L
kxixjk)(xixj),
where Csis the spring constant and Lis the spring’s rest length. A damping factor, which was proportional to
the difference in the objects’ velocities, was also applied. Collision forces between the string’s masses and the
static, rigid object were calculated as described in the bouncing balls section above.
A.2 Results details
The prediction experiment’s generalization performance MSEs are shown Table 3. The URL links for the videos


key: chroma:document
string_value:
A.2 Results details
The prediction experiment’s generalization performance MSEs are shown Table 3. The URL links for the videos
associated with all still frames in Figure 2 are in Table 1.
12


key: chroma:document
string_value:
Mask R-CNN
Kaiming He Georgia Gkioxari Piotr Doll ´ar Ross Girshick
Facebook AI Research (FAIR)
Abstract
We present a conceptually simple, ﬂexible, and general
framework for object instance segmentation. Our approach
efﬁciently detects objects in an image while simultaneously
generating a high-quality segmentation mask for each instance. The method, called Mask R-CNN, extends Faster
R-CNN by adding a branch for predicting an object mask in
parallel with the existing branch for bounding box recognition. Mask R-CNN is simple to train and adds only a small
overhead to Faster R-CNN, running at 5 fps. Moreover,
Mask R-CNN is easy to generalize to other tasks, e.g., allowing us to estimate human poses in the same framework.
We show top results in all three tracks of the COCO suite
of challenges, including instance segmentation, boundingbox object detection, and person keypoint detection. Without bells and whistles, Mask R-CNN outperforms all existing, single-model entries on every task, including the
COCO 2016 challenge winners. We hope our simple and
effective approach will serve as a solid baseline and help
ease future research in instance-level recognition. Code
has been made available at: https://github.com/


key: chroma:document
string_value:
effective approach will serve as a solid baseline and help
ease future research in instance-level recognition. Code
has been made available at: https://github.com/
facebookresearch/Detectron .
1. Introduction
The vision community has rapidly improved object detection and semantic segmentation results over a short period of time. In large part, these advances have been driven
by powerful baseline systems, such as the Fast/Faster RCNN [12, 36] and Fully Convolutional Network (FCN) [30]
frameworks for object detection and semantic segmentation, respectively. These methods are conceptually intuitive
and offer ﬂexibility and robustness, together with fast training and inference time. Our goal in this work is to develop a
comparably enabling framework for instance segmentation .
Instance segmentation is challenging because it requires
the correct detection of all objects in an image while also
precisely segmenting each instance. It therefore combines
elements from the classical computer vision tasks of object detection , where the goal is to classify individual objects and localize each using a bounding box, and semantic
RoIAlignRoIAlignclass
box
convconv
 convconvFigure 1. The Mask R-CNN framework for instance segmentation.
segmentation , where the goal is to classify each pixel into


key: chroma:document
string_value:
box
convconv
 convconvFigure 1. The Mask R-CNN framework for instance segmentation.
segmentation , where the goal is to classify each pixel into
a ﬁxed set of categories without differentiating object instances.1Given this, one might expect a complex method
is required to achieve good results. However, we show that
a surprisingly simple, ﬂexible, and fast system can surpass
prior state-of-the-art instance segmentation results.
Our method, called Mask R-CNN , extends Faster R-CNN
[36] by adding a branch for predicting segmentation masks
on each Region of Interest (RoI), in parallel with the existing branch for classiﬁcation and bounding box regression (Figure 1). The mask branch is a small FCN applied
to each RoI, predicting a segmentation mask in a pixel-topixel manner. Mask R-CNN is simple to implement and
train given the Faster R-CNN framework, which facilitates
a wide range of ﬂexible architecture designs. Additionally,
the mask branch only adds a small computational overhead,
enabling a fast system and rapid experimentation.
In principle Mask R-CNN is an intuitive extension of
Faster R-CNN, yet constructing the mask branch properly


key: chroma:document
string_value:
enabling a fast system and rapid experimentation.
In principle Mask R-CNN is an intuitive extension of
Faster R-CNN, yet constructing the mask branch properly
is critical for good results. Most importantly, Faster RCNN was not designed for pixel-to-pixel alignment between network inputs and outputs. This is most evident in
how RoIPool [18, 12], the de facto core operation for attending to instances, performs coarse spatial quantization
for feature extraction. To ﬁx the misalignment, we propose a simple, quantization-free layer, called RoIAlign , that
faithfully preserves exact spatial locations. Despite being
1Following common terminology, we use object detection to denote
detection via bounding boxes , not masks, and semantic segmentation to
denote per-pixel classiﬁcation without differentiating instances. Yet we
note that instance segmentation is both semantic and a form of detection.
1arXiv:1703.06870v3  [cs.CV]  24 Jan 2018
dining table.96person1.00person1.00 person1.00 person1.00 person1.00person1.00
person1.00person.94
bottle.99
bottle.99
bottle.99
motorcycle1.00motorcycle1.00person1.00
person1.00


key: chroma:document
string_value:
bottle.99
bottle.99
bottle.99
motorcycle1.00motorcycle1.00person1.00
person1.00
person.96person1.00person.83person.96
person.98 person.90person.92person.99person.91
bus.99
person1.00
person1.00person1.00
backpack.93person1.00person.99
person1.00
backpack.99
person.99person.98person.89person.95
person1.00
person1.00
car1.00traffic light .96
person.96truck1.00person.99
car.99person.85
motorcycle.95car.99car.92person.99person1.00traffic light .92 traffic light .84traffic light .95
car.93person.87
person1.00
person1.00umbrella.98
umbrella.98
backpack1.00
handba g.96
elephant1.00person1.00person1.00 person.99
sheep1. 00person1.00
sheep.99sheep.91 sheep1. 00
sheep.99sheep.99sheep.95person.99


key: chroma:document
string_value:
sheep1. 00person1.00
sheep.99sheep.91 sheep1. 00
sheep.99sheep.99sheep.95person.99
sheep1. 00sheep.96sheep.99sheep.99
sheep.96
sheep.96sheep.96sheep.86
sheep.82sheep.93
dining table.99
chair.99chair.90
chair.99chair.98
chair.96chair.86chair.99
bowl.81chair.96tv.99
bottle.99
wine glass.99wine glass1.00bowl.85
knife.83wine glass1.00 wine glass.93wine glass.97
fork.95Figure 2. Mask R-CNN results on the COCO test set. These results are based on ResNet-101 [19], achieving a mask AP of 35.7 and
running at 5 fps. Masks are shown in color, and bounding box, category, and conﬁdences are also shown.
a seemingly minor change, RoIAlign has a large impact: it
improves mask accuracy by relative 10% to 50%, showing
bigger gains under stricter localization metrics. Second, we
found it essential to decouple mask and class prediction: we
predict a binary mask for each class independently, without


key: chroma:document
string_value:
bigger gains under stricter localization metrics. Second, we
found it essential to decouple mask and class prediction: we
predict a binary mask for each class independently, without
competition among classes, and rely on the network’s RoI
classiﬁcation branch to predict the category. In contrast,
FCNs usually perform per-pixel multi-class categorization,
which couples segmentation and classiﬁcation, and based
on our experiments works poorly for instance segmentation.
Without bells and whistles, Mask R-CNN surpasses all
previous state-of-the-art single-model results on the COCO
instance segmentation task [28], including the heavilyengineered entries from the 2016 competition winner. As
a by-product, our method also excels on the COCO object
detection task. In ablation experiments, we evaluate multiple basic instantiations, which allows us to demonstrate its
robustness and analyze the effects of core factors.
Our models can run at about 200ms per frame on a GPU,
and training on COCO takes one to two days on a single
8-GPU machine. We believe the fast train and test speeds,
together with the framework’s ﬂexibility and accuracy, will
beneﬁt and ease future research on instance segmentation.


key: chroma:document
string_value:
together with the framework’s ﬂexibility and accuracy, will
beneﬁt and ease future research on instance segmentation.
Finally, we showcase the generality of our framework
via the task of human pose estimation on the COCO keypoint dataset [28]. By viewing each keypoint as a one-hot
binary mask, with minimal modiﬁcation Mask R-CNN can
be applied to detect instance-speciﬁc poses. Mask R-CNN
surpasses the winner of the 2016 COCO keypoint competition, and at the same time runs at 5 fps. Mask R-CNN,
therefore, can be seen more broadly as a ﬂexible framework
forinstance-level recognition and can be readily extended
to more complex tasks.
We have released code to facilitate future research.2. Related Work
R-CNN: The Region-based CNN (R-CNN) approach [13]
to bounding-box object detection is to attend to a manageable number of candidate object regions [42, 20] and evaluate convolutional networks [25, 24] independently on each
RoI. R-CNN was extended [18, 12] to allow attending to
RoIs on feature maps using RoIPool, leading to fast speed
and better accuracy. Faster R-CNN [36] advanced this


key: chroma:document
string_value:
RoIs on feature maps using RoIPool, leading to fast speed
and better accuracy. Faster R-CNN [36] advanced this
stream by learning the attention mechanism with a Region
Proposal Network (RPN). Faster R-CNN is ﬂexible and robust to many follow-up improvements ( e.g., [38, 27, 21]),
and is the current leading framework in several benchmarks.
Instance Segmentation: Driven by the effectiveness of RCNN, many approaches to instance segmentation are based
onsegment proposals . Earlier methods [13, 15, 16, 9] resorted to bottom-up segments [42, 2]. DeepMask [33] and
following works [34, 8] learn to propose segment candidates, which are then classiﬁed by Fast R-CNN. In these
methods, segmentation precedes recognition, which is slow
and less accurate. Likewise, Dai et al. [10] proposed a complex multiple-stage cascade that predicts segment proposals
from bounding-box proposals, followed by classiﬁcation.
Instead, our method is based on parallel prediction of masks
and class labels, which is simpler and more ﬂexible.
Most recently, Li et al. [26] combined the segment proposal system in [8] and object detection system in [11] for


key: chroma:document
string_value:
Most recently, Li et al. [26] combined the segment proposal system in [8] and object detection system in [11] for
“fully convolutional instance segmentation” (FCIS). The
common idea in [8, 11, 26] is to predict a set of positionsensitive output channels fully convolutionally. These
channels simultaneously address object classes, boxes, and
masks, making the system fast. But FCIS exhibits systematic errors on overlapping instances and creates spurious
edges (Figure 6), showing that it is challenged by the fundamental difﬁculties of segmenting instances.
2
Another family of solutions [23, 4, 3, 29] to instance segmentation are driven by the success of semantic segmentation. Starting from per-pixel classiﬁcation results ( e.g.,
FCN outputs), these methods attempt to cut the pixels of
the same category into different instances. In contrast to the
segmentation-ﬁrst strategy of these methods, Mask R-CNN
is based on an instance-ﬁrst strategy. We expect a deeper incorporation of both strategies will be studied in the future.
3. Mask R-CNN
Mask R-CNN is conceptually simple: Faster R-CNN has
two outputs for each candidate object, a class label and a


key: chroma:document
string_value:
3. Mask R-CNN
Mask R-CNN is conceptually simple: Faster R-CNN has
two outputs for each candidate object, a class label and a
bounding-box offset; to this we add a third branch that outputs the object mask. Mask R-CNN is thus a natural and intuitive idea. But the additional mask output is distinct from
the class and box outputs, requiring extraction of much ﬁner
spatial layout of an object. Next, we introduce the key elements of Mask R-CNN, including pixel-to-pixel alignment,
which is the main missing piece of Fast/Faster R-CNN.
Faster R-CNN: We begin by brieﬂy reviewing the Faster
R-CNN detector [36]. Faster R-CNN consists of two stages.
The ﬁrst stage, called a Region Proposal Network (RPN),
proposes candidate object bounding boxes. The second
stage, which is in essence Fast R-CNN [12], extracts features using RoIPool from each candidate box and performs
classiﬁcation and bounding-box regression. The features
used by both stages can be shared for faster inference. We
refer readers to [21] for latest, comprehensive comparisons
between Faster R-CNN and other frameworks.
Mask R-CNN: Mask R-CNN adopts the same two-stage


key: chroma:document
string_value:
refer readers to [21] for latest, comprehensive comparisons
between Faster R-CNN and other frameworks.
Mask R-CNN: Mask R-CNN adopts the same two-stage
procedure, with an identical ﬁrst stage (which is RPN). In
the second stage, in parallel to predicting the class and box
offset, Mask R-CNN also outputs a binary mask for each
RoI. This is in contrast to most recent systems, where classiﬁcation depends on mask predictions ( e.g. [33, 10, 26]).
Our approach follows the spirit of Fast R-CNN [12] that
applies bounding-box classiﬁcation and regression in parallel (which turned out to largely simplify the multi-stage
pipeline of original R-CNN [13]).
Formally, during training, we deﬁne a multi-task loss on
each sampled RoI as L=Lcls+Lbox+Lmask . The classiﬁcation loss Lclsand bounding-box loss Lboxare identical as those deﬁned in [12]. The mask branch has a Km2dimensional output for each RoI, which encodes Kbinary
masks of resolution mm, one for each of the Kclasses.


key: chroma:document
string_value:
masks of resolution mm, one for each of the Kclasses.
To this we apply a per-pixel sigmoid, and deﬁne Lmask as
the average binary cross-entropy loss. For an RoI associated
with ground-truth class k,Lmask is only deﬁned on the k-th
mask (other mask outputs do not contribute to the loss).
Our deﬁnition of Lmask allows the network to generate
masks for every class without competition among classes;
we rely on the dedicated classiﬁcation branch to predict the
Figure 3. RoIAlign: The dashed grid represents a feature map, the solid lines an RoI
(with 22 bins in this example), and the dots
the 4 sampling points in each bin. RoIAlign
computes the value of each sampling point
by bilinear interpolation from the nearby grid
points on the feature map. No quantization is
performed on any coordinates involved in the
RoI, its bins, or the sampling points.
class label used to select the output mask. This decouples
mask and class prediction. This is different from common
practice when applying FCNs [30] to semantic segmentation, which typically uses a per-pixel softmax and a multinomial cross-entropy loss. In that case, masks across classes


key: chroma:document
string_value:
practice when applying FCNs [30] to semantic segmentation, which typically uses a per-pixel softmax and a multinomial cross-entropy loss. In that case, masks across classes
compete; in our case, with a per-pixel sigmoid and a binary
loss, they do not. We show by experiments that this formulation is key for good instance segmentation results.
Mask Representation: A mask encodes an input object’s
spatial layout. Thus, unlike class labels or box offsets
that are inevitably collapsed into short output vectors by
fully-connected ( fc) layers, extracting the spatial structure
of masks can be addressed naturally by the pixel-to-pixel
correspondence provided by convolutions.
Speciﬁcally, we predict an mmmask from each RoI
using an FCN [30]. This allows each layer in the mask
branch to maintain the explicit mmobject spatial layout without collapsing it into a vector representation that
lacks spatial dimensions. Unlike previous methods that resort to fclayers for mask prediction [33, 34, 10], our fully
convolutional representation requires fewer parameters, and
is more accurate as demonstrated by experiments.
This pixel-to-pixel behavior requires our RoI features,
which themselves are small feature maps, to be well aligned


key: chroma:document
string_value:
is more accurate as demonstrated by experiments.
This pixel-to-pixel behavior requires our RoI features,
which themselves are small feature maps, to be well aligned
to faithfully preserve the explicit per-pixel spatial correspondence. This motivated us to develop the following
RoIAlign layer that plays a key role in mask prediction.
RoIAlign: RoIPool [12] is a standard operation for extracting a small feature map ( e.g., 77) from each RoI. RoIPool
ﬁrstquantizes a ﬂoating-number RoI to the discrete granularity of the feature map, this quantized RoI is then subdivided into spatial bins which are themselves quantized, and
ﬁnally feature values covered by each bin are aggregated
(usually by max pooling). Quantization is performed, e.g.,
on a continuous coordinate xby computing [x=16], where
16 is a feature map stride and []is rounding; likewise, quantization is performed when dividing into bins ( e.g., 77).
These quantizations introduce misalignments between the
RoI and the extracted features. While this may not impact
classiﬁcation, which is robust to small translations, it has a
large negative effect on predicting pixel-accurate masks.


key: chroma:document
string_value:
classiﬁcation, which is robust to small translations, it has a
large negative effect on predicting pixel-accurate masks.
To address this, we propose an RoIAlign layer that removes the harsh quantization of RoIPool, properly aligning
the extracted features with the input. Our proposed change
is simple: we avoid any quantization of the RoI boundaries
3
or bins ( i.e., we use x=16instead of [x=16]). We use bilinear interpolation [22] to compute the exact values of the
input features at four regularly sampled locations in each
RoI bin, and aggregate the result (using max or average),
see Figure 3 for details. We note that the results are not sensitive to the exact sampling locations, or how many points
are sampled, as long as no quantization is performed.
RoIAlign leads to large improvements as we show in
x4.2. We also compare to the RoIWarp operation proposed
in [10]. Unlike RoIAlign, RoIWarp overlooked the alignment issue and was implemented in [10] as quantizing RoI
just like RoIPool. So even though RoIWarp also adopts
bilinear resampling motivated by [22], it performs on par


key: chroma:document
string_value:
just like RoIPool. So even though RoIWarp also adopts
bilinear resampling motivated by [22], it performs on par
with RoIPool as shown by experiments (more details in Table 2c), demonstrating the crucial role of alignment.
Network Architecture: To demonstrate the generality of
our approach, we instantiate Mask R-CNN with multiple
architectures. For clarity, we differentiate between: (i) the
convolutional backbone architecture used for feature extraction over an entire image, and (ii) the network head
for bounding-box recognition (classiﬁcation and regression)
and mask prediction that is applied separately to each RoI.
We denote the backbone architecture using the nomenclature network-depth-features . We evaluate ResNet [19]
and ResNeXt [45] networks of depth 50 or 101 layers. The
original implementation of Faster R-CNN with ResNets
[19] extracted features from the ﬁnal convolutional layer
of the 4-th stage, which we call C4. This backbone with
ResNet-50, for example, is denoted by ResNet-50-C4. This
is a common choice used in [19, 10, 21, 39].


key: chroma:document
string_value:
ResNet-50, for example, is denoted by ResNet-50-C4. This
is a common choice used in [19, 10, 21, 39].
We also explore another more effective backbone recently proposed by Lin et al. [27], called a Feature Pyramid Network (FPN). FPN uses a top-down architecture with
lateral connections to build an in-network feature pyramid
from a single-scale input. Faster R-CNN with an FPN backbone extracts RoI features from different levels of the feature pyramid according to their scale, but otherwise the
rest of the approach is similar to vanilla ResNet. Using a
ResNet-FPN backbone for feature extraction with Mask RCNN gives excellent gains in both accuracy and speed. For
further details on FPN, we refer readers to [27].
For the network head we closely follow architectures
presented in previous work to which we add a fully convolutional mask prediction branch. Speciﬁcally, we extend the Faster R-CNN box heads from the ResNet [19]
and FPN [27] papers. Details are shown in Figure 4. The
head on the ResNet-C4 backbone includes the 5-th stage of
ResNet (namely, the 9-layer ‘res5’ [19]), which is computeintensive. For FPN, the backbone already includes res5 and


key: chroma:document
string_value:
ResNet (namely, the 9-layer ‘res5’ [19]), which is computeintensive. For FPN, the backbone already includes res5 and
thus allows for a more efﬁcient head that uses fewer ﬁlters.
We note that our mask branches have a straightforward
structure. More complex designs have the potential to improve performance but are not the focus of this work.
ave
RoI
RoI14×14
×2567×7
×256
14×14
×2561024
28×28
×2561024
mask14×14
×256class
box2048RoI res57×7
×10247×7
×2048
×4class
box
14×14
×80
mask28×28
×80Faster R-CNN
w/ ResNet [19]Faster R-CNN
w/ FPN [27]
Figure 4. Head Architecture : We extend two existing Faster RCNN heads [19, 27]. Left/Right panels show the heads for the
ResNet C4 and FPN backbones, from [19] and [27], respectively,
to which a mask branch is added. Numbers denote spatial resolution and channels. Arrows denote either conv, deconv, or fclayers
as can be inferred from context (conv preserves spatial dimension


key: chroma:document
string_value:
to which a mask branch is added. Numbers denote spatial resolution and channels. Arrows denote either conv, deconv, or fclayers
as can be inferred from context (conv preserves spatial dimension
while deconv increases it). All convs are 3 3, except the output
conv which is 1 1, deconvs are 2 2 with stride 2, and we use
ReLU [31] in hidden layers. Left: ‘res5’ denotes ResNet’s ﬁfth
stage, which for simplicity we altered so that the ﬁrst conv operates on a 7 7 RoI with stride 1 (instead of 14 14 / stride 2 as in
[19]). Right : ‘4’ denotes a stack of four consecutive convs.
3.1. Implementation Details
We set hyper-parameters following existing Fast/Faster
R-CNN work [12, 36, 27]. Although these decisions were
made for object detection in original papers [12, 36, 27], we
found our instance segmentation system is robust to them.
Training: As in Fast R-CNN, an RoI is considered positive
if it has IoU with a ground-truth box of at least 0.5 and
negative otherwise. The mask loss Lmask is deﬁned only on
positive RoIs. The mask target is the intersection between


key: chroma:document
string_value:
negative otherwise. The mask loss Lmask is deﬁned only on
positive RoIs. The mask target is the intersection between
an RoI and its associated ground-truth mask.
We adopt image-centric training [12]. Images are resized
such that their scale (shorter edge) is 800 pixels [27]. Each
mini-batch has 2 images per GPU and each image has N
sampled RoIs, with a ratio of 1:3 of positive to negatives
[12]. Nis 64 for the C4 backbone (as in [12, 36]) and 512
for FPN (as in [27]). We train on 8 GPUs (so effective minibatch size is 16) for 160k iterations, with a learning rate of
0.02 which is decreased by 10 at the 120k iteration. We
use a weight decay of 0.0001 and momentum of 0.9. With
ResNeXt [45], we train with 1 image per GPU and the same
number of iterations, with a starting learning rate of 0.01.
The RPN anchors span 5 scales and 3 aspect ratios, following [27]. For convenient ablation, RPN is trained separately and does not share features with Mask R-CNN, unless speciﬁed. For every entry in this paper, RPN and Mask
R-CNN have the same backbones and so they are shareable.


key: chroma:document
string_value:
R-CNN have the same backbones and so they are shareable.
Inference: At test time, the proposal number is 300 for the
C4 backbone (as in [36]) and 1000 for FPN (as in [27]). We
run the box prediction branch on these proposals, followed
by non-maximum suppression [14]. The mask branch is
then applied to the highest scoring 100 detection boxes. Although this differs from the parallel computation used in
training, it speeds up inference and improves accuracy (due
to the use of fewer, more accurate RoIs). The mask branch
4
horse1.00horse1.00horse1.00
bus1.00bus1.00
car.98truck.88
car.93
car.78car.98
car.91 car.96car.99
car.94car.99car.98 truck.86
car.99car.95car1.00car.93 car.98car.95
car.97car.87
car.99
car.82car.78car.93
car.95
car.97
person.99traffic light .73
person1.00
person.99person.95person.93
person.93
person1.00person.98
skateboard.82
suitcase1.00
suitcase.99suitcase.96suitcase1.00


key: chroma:document
string_value:
person.93
person1.00person.98
skateboard.82
suitcase1.00
suitcase.99suitcase.96suitcase1.00
suitcase.93suitcase.98
suitcase.88suitcase.72stop sign.88
person1.00 person1.00
person1.00person1.00
person.99person.99
bench.76skateboard.91
skateboard.83handba g.81
surfboard1.00person1.00person1.00 surfboard1.00person1.00person.98
surfboard1.00person1.00
surfboard.98surfboard1.00person.91
person.74
person1.00person1.00
person1.00person1.00person1.00 person1.00 person.98person.99
person1.00person.99 umbrella1.00
person.95umbrella.99umbrella.97umbrella.97
umbrella.96
umbrella1.00
backpack.96umbrella.98
backpack.95person.80
backpack.98
bicycle.93umbrella.89person.89handba g.97
handba g.85


key: chroma:document
string_value:
backpack.95person.80
backpack.98
bicycle.93umbrella.89person.89handba g.97
handba g.85
person1.00person1.00 person1.00person1.00person1.00person1.00
motorcycle.72kite.89
person.99kite.95
person.99
person1.00person.81person.72kite.93
person.89kite1.00
person.98
person1.00kite.84kite.97
person.80
handba g.80person.99kite.82
person.98 person.96kite.98
person.99person.82kite.81
person.95 person.84kite.98kite.72
kite.99kite.84kite.99
person.94person.72person.98kite.95
person.98 person.77kite.73
person.78 person.71 person.87kite.88kite.88
person.94kite.86kite.89
zebra.99
zebra1.00zebra1.00
zebra.99zebra1.00zebra.96
zebra.74
zebra.96zebra.99zebra.90


key: chroma:document
string_value:
zebra.99zebra1.00zebra.96
zebra.74
zebra.96zebra.99zebra.90
zebra.88zebra.76
dining table.91dining table.78
chair.97person.99
person.86
chair.94chair.98person.95
chair.95person.97
chair.92chair.99person.97
person.99person.94 person.99person.87
person.99
chair.83person.94person.99person.98
chair.87chair.95person.97
person.96
chair.99person.86person.89
chair.89
wine glass.93person.98person.88person.97
person.88person.88
person.91 chair.96person.95
person.77person.92
wine glass.94cup.83
wine glass.94
wine glass.83
cup.91chair.85 dining table.96
wine glass.91person.96
cup.98person.83
dining table.75
cup.96person.72
wine glass.80chair.98person.81person.82
dining table.81
chair.85chair.78
cup.75person.77
cup.71wine glass.80cup.79cup.93
cup.71


key: chroma:document
string_value:
dining table.81
chair.85chair.78
cup.75person.77
cup.71wine glass.80cup.79cup.93
cup.71
person.99person.99
person1.00person1.00
frisbee1.00
person.80person.82
elephant1.00elephant1.00elephant1.00
elephant.97
elephant.99
person1.00person1.00
dining table.95person1.00person.88
wine glass1.00bottle.97
wine glass1.00wine glass.99tv.98 tv.84
person1.00
bench.97person.98person1.00person1.00
handba g.73person.86 potted plant.92
bird.93person.76person.98person.78 person.78 backpack.88handba g.91
cell phone.77 clock.73
person.99person1.00person.98
person1.00person1.00 person1.00person.99
person.99 person.99 person1.00 person1.00person.98 person.99
handba g.88person1.00 person.98person.92
handba g.99person.97
person.95
handba g.88traffic light .99
person.95


key: chroma:document
string_value:
handba g.99person.97
person.95
handba g.88traffic light .99
person.95
person.87person.95traffic light .87
traffic light .71
person.80person.95 person.95 person.73person.74
tie.85
car.99
car.86car.97car1.00 car.95car.97traffic light 1.00traffic light .99
car.99person.99car.95
car.97 car.98car.98
car.91
car1.00car.96car.96
bicycle.86car.97car.97
car.97car.94car.95
car.94car.81
person.87
parking meter.98car.89
donut1.00donut.90
donut.88donut.81
donut.95
donut.96donut1.00 donut.98
donut.99donut.94donut.97donut.99
donut.98donut1.00
donut.95donut1.00
donut.98donut.98donut.99
donut.96
donut.89donut.96donut.95donut.98donut.89
donut.93donut.95


key: chroma:document
string_value:
donut.96
donut.89donut.96donut.95donut.98donut.89
donut.93donut.95
donut.90donut.89
donut.89donut.89
donut.86donut.86
person1.00person1.00 person1.00
person1.00person1.00person1.00
person1.00
dog1.00baseball bat.99
baseball bat.85
baseball bat.98
truck.92
truck.99truck.96truck.99truck.97
bus.99truck.93 bus.90
person1.00person1.00horse.77horse.99
cow.93person.96person1.00
person.99horse.97
person.98 person.97person.98
person.96
person1.00
tennis racket1.00chair.73person.90person.77
person.97
person.81person.87
person.71 person.96 person.99 person.98 person.94chair.97
chair.80
chair.71chair.94chair.92chair.99chair.93
chair.99
chair.91 chair.81 chair.98 chair.83chair.81chair.81
chair.93
sports ball.99


key: chroma:document
string_value:
chair.99
chair.91 chair.81 chair.98 chair.83chair.81chair.81
chair.93
sports ball.99
person1.00
couch.82person1.00
person.99person1.00person1.00person1.00 person.99skateboard.99
person.90person.98person.99person.91
person.99person1.00
person.80
skateboard.98Figure 5. More results of Mask R-CNN on COCO test images, using ResNet-101-FPN and running at 5 fps, with 35.7 mask AP (Table 1).
backbone AP AP 50 AP75 APS APM APL
MNC [10] ResNet-101-C4 24.6 44.3 24.8 4.7 25.9 43.6
FCIS [26] +OHEM ResNet-101-C5-dilated 29.2 49.5 - 7.1 31.3 50.0
FCIS+++ [26] +OHEM ResNet-101-C5-dilated 33.6 54.5 - - - Mask R-CNN ResNet-101-C4 33.1 54.9 34.8 12.1 35.6 51.1


key: chroma:document
string_value:
Mask R-CNN ResNet-101-FPN 35.7 58.0 37.8 15.5 38.1 52.4
Mask R-CNN ResNeXt-101-FPN 37.1 60.0 39.4 16.9 39.9 53.5
Table 1. Instance segmentation mask AP on COCO test-dev . MNC [10] and FCIS [26] are the winners of the COCO 2015 and 2016
segmentation challenges, respectively. Without bells and whistles, Mask R-CNN outperforms the more complex FCIS+++, which includes
multi-scale train/test, horizontal ﬂip test, and OHEM [38]. All entries are single-model results.
can predict Kmasks per RoI, but we only use the k-th mask,
where kis the predicted class by the classiﬁcation branch.
Themmﬂoating-number mask output is then resized to
the RoI size, and binarized at a threshold of 0.5.
Note that since we only compute masks on the top 100
detection boxes, Mask R-CNN adds a small overhead to its
Faster R-CNN counterpart ( e.g.,20% on typical models).
4. Experiments: Instance Segmentation


key: chroma:document
string_value:
Faster R-CNN counterpart ( e.g.,20% on typical models).
4. Experiments: Instance Segmentation
We perform a thorough comparison of Mask R-CNN to
the state of the art along with comprehensive ablations on
the COCO dataset [28]. We report the standard COCO metrics including AP (averaged over IoU thresholds), AP 50,
AP75, and AP S, AP M, AP L(AP at different scales). Unless noted, AP is evaluating using mask IoU. As in previous
work [5, 27], we train using the union of 80k train images
and a 35k subset of val images ( trainval35k ), and report ablations on the remaining 5k val images ( minival ).
We also report results on test-dev [28].4.1. Main Results
We compare Mask R-CNN to the state-of-the-art methods in instance segmentation in Table 1. All instantiations of our model outperform baseline variants of previous state-of-the-art models. This includes MNC [10]
and FCIS [26], the winners of the COCO 2015 and 2016
segmentation challenges, respectively. Without bells and
whistles, Mask R-CNN with ResNet-101-FPN backbone
outperforms FCIS+++ [26], which includes multi-scale


key: chroma:document
string_value:
whistles, Mask R-CNN with ResNet-101-FPN backbone
outperforms FCIS+++ [26], which includes multi-scale
train/test, horizontal ﬂip test, and online hard example mining (OHEM) [38]. While outside the scope of this work, we
expect many such improvements to be applicable to ours.
Mask R-CNN outputs are visualized in Figures 2 and 5.
Mask R-CNN achieves good results even under challenging conditions. In Figure 6 we compare our Mask R-CNN
baseline and FCIS+++ [26]. FCIS+++ exhibits systematic
artifacts on overlapping instances, suggesting that it is challenged by the fundamental difﬁculty of instance segmentation. Mask R-CNN shows no such artifacts.
5
person1.00
person1.00
person1.00person1.00umbrella1.00 umbrella.99
car.99car.93
giraffe1.00 giraffe1.00
person1.00person1.00person1.00person1.00
person.95
sports ball1.00sports ball.98person1.00
person1.00
person1.00
tie.95
tie1.00
FCIS Mask R-CNN
train1.00
train.99
train.80
person1.00 person1.00person1.00


key: chroma:document
string_value:
tie1.00
FCIS Mask R-CNN
train1.00
train.99
train.80
person1.00 person1.00person1.00
person1.00person1.00 person1.00
skateboard.98person.99 person.99
skateboard.99handba g.93
Figure 6. FCIS+++ [26] (top) vs. Mask R-CNN (bottom, ResNet-101-FPN). FCIS exhibits systematic artifacts on overlapping objects.
net-depth-features AP AP 50 AP75
ResNet-50-C4 30.3 51.2 31.5
ResNet-101-C4 32.7 54.2 34.3
ResNet-50-FPN 33.6 55.2 35.3
ResNet-101-FPN 35.4 57.3 37.5
ResNeXt-101-FPN 36.7 59.5 38.9
(a)Backbone Architecture : Better backbones bring expected gains: deeper networks
do better, FPN outperforms C4 features, and
ResNeXt improves on ResNet.AP AP 50 AP75
softmax 24.8 44.1 25.1
sigmoid 30.3 51.2 31.5
+5.5 +7.1 +6.4


key: chroma:document
string_value:
softmax 24.8 44.1 25.1
sigmoid 30.3 51.2 31.5
+5.5 +7.1 +6.4
(b)Multinomial vs. Independent Masks
(ResNet-50-C4): Decoupling via perclass binary masks (sigmoid) gives large
gains over multinomial masks (softmax).align? bilinear? agg. AP AP 50 AP75
RoIPool [12] max 26.9 48.8 26.4
RoIWarp [10]X max 27.2 49.2 27.1
X ave 27.1 48.9 27.1
RoIAlignX X max 30.2 51.0 31.8
X X ave 30.3 51.2 31.5
(c)RoIAlign (ResNet-50-C4): Mask results with various RoI
layers. Our RoIAlign layer improves AP by 3 points and
AP75by5 points. Using proper alignment is the only factor that contributes to the large gap between RoI layers.
AP AP 50 AP75 APbbAPbb
50APbb
75
RoIPool 23.6 46.5 21.6 28.2 52.7 26.9


key: chroma:document
string_value:
AP AP 50 AP75 APbbAPbb
50APbb
75
RoIPool 23.6 46.5 21.6 28.2 52.7 26.9
RoIAlign 30.9 51.8 32.1 34.0 55.3 36.4
+7.3 + 5.3 +10.5 +5.8 +2.6 +9.5
(d)RoIAlign (ResNet-50- C5,stride 32 ): Mask-level and box-level
AP using large-stride features. Misalignments are more severe than
with stride-16 features (Table 2c), resulting in big accuracy gaps.mask branch AP AP 50 AP75
MLP fc: 1024!1024!8028231.5 53.7 32.8
MLP fc: 1024!1024!1024!8028231.5 54.0 32.6
FCN conv: 256!256!256!256!256!80 33.6 55.2 35.3
(e)Mask Branch (ResNet-50-FPN): Fully convolutional networks (FCN) vs.
multi-layer perceptrons (MLP, fully-connected) for mask prediction. FCNs improve results as they take advantage of explicitly encoding spatial layout.


key: chroma:document
string_value:
multi-layer perceptrons (MLP, fully-connected) for mask prediction. FCNs improve results as they take advantage of explicitly encoding spatial layout.
Table 2. Ablations . We train on trainval35k , test on minival , and report mask AP unless otherwise noted.
4.2. Ablation Experiments
We run a number of ablations to analyze Mask R-CNN.
Results are shown in Table 2 and discussed in detail next.
Architecture: Table 2a shows Mask R-CNN with various
backbones. It beneﬁts from deeper networks (50 vs. 101)
and advanced designs including FPN and ResNeXt. We
note that notall frameworks automatically beneﬁt from
deeper or advanced networks (see benchmarking in [21]).
Multinomial vs. Independent Masks: Mask R-CNN decouples mask and class prediction: as the existing box
branch predicts the class label, we generate a mask for each
class without competition among classes (by a per-pixel sigmoid and a binary loss). In Table 2b, we compare this to
using a per-pixel softmax and a multinomial loss (as commonly used in FCN [30]). This alternative couples the tasks
of mask and class prediction, and results in a severe loss


key: chroma:document
string_value:
of mask and class prediction, and results in a severe loss
in mask AP (5.5 points). This suggests that once the instance has been classiﬁed as a whole (by the box branch),
it is sufﬁcient to predict a binary mask without concern for
the categories, which makes the model easier to train.
Class-Speciﬁc vs. Class-Agnostic Masks: Our default instantiation predicts class-speciﬁc masks, i.e., one mmmask per class. Interestingly, Mask R-CNN with classagnostic masks ( i.e., predicting a single mmoutput regardless of class) is nearly as effective: it has 29.7 mask AP
vs. 30.3 for the class-speciﬁc counterpart on ResNet-50-C4.
This further highlights the division of labor in our approach
which largely decouples classiﬁcation and segmentation.
RoIAlign: An evaluation of our proposed RoIAlign layer is
shown in Table 2c. For this experiment we use the ResNet50-C4 backbone, which has stride 16. RoIAlign improves
AP by about 3 points over RoIPool, with much of the gain
coming at high IoU (AP 75). RoIAlign is insensitive to


key: chroma:document
string_value:
AP by about 3 points over RoIPool, with much of the gain
coming at high IoU (AP 75). RoIAlign is insensitive to
max/average pool; we use average in the rest of the paper.
Additionally, we compare with RoIWarp proposed in
MNC [10] that also adopt bilinear sampling. As discussed
inx3, RoIWarp still quantizes the RoI, losing alignment
with the input. As can be seen in Table 2c, RoIWarp performs on par with RoIPool and much worse than RoIAlign.
This highlights that proper alignment is key.
We also evaluate RoIAlign with a ResNet-50-C5 backbone, which has an even larger stride of 32 pixels. We use
the same head as in Figure 4 (right), as the res5 head is not
applicable. Table 2d shows that RoIAlign improves mask
AP by a massive 7.3 points, and mask AP 75by 10.5 points
6
backbone APbbAPbb
50APbb
75APbb
SAPbb
MAPbb
L
Faster R-CNN+++ [19] ResNet-101-C4 34.9 55.7 37.4 15.6 38.7 50.9


key: chroma:document
string_value:
MAPbb
L
Faster R-CNN+++ [19] ResNet-101-C4 34.9 55.7 37.4 15.6 38.7 50.9
Faster R-CNN w FPN [27] ResNet-101-FPN 36.2 59.1 39.0 18.2 39.0 48.2
Faster R-CNN by G-RMI [21] Inception-ResNet-v2 [41] 34.7 55.5 36.7 13.5 38.1 52.0
Faster R-CNN w TDM [39] Inception-ResNet-v2-TDM 36.8 57.7 39.2 16.2 39.8 52.1
Faster R-CNN, RoIAlign ResNet-101-FPN 37.3 59.6 40.3 19.8 40.2 48.8
Mask R-CNN ResNet-101-FPN 38.2 60.3 41.7 20.1 41.1 50.2
Mask R-CNN ResNeXt-101-FPN 39.8 62.3 43.4 22.1 43.2 51.2


key: chroma:document
string_value:
Mask R-CNN ResNeXt-101-FPN 39.8 62.3 43.4 22.1 43.2 51.2
Table 3. Object detection single-model results (bounding box AP), vs. state-of-the-art on test-dev . Mask R-CNN using ResNet-101FPN outperforms the base variants of all previous state-of-the-art models (the mask output is ignored in these experiments). The gains of
Mask R-CNN over [27] come from using RoIAlign (+1.1 APbb), multitask training (+0.9 APbb), and ResNeXt-101 (+1.6 APbb).
(50% relative improvement ). Moreover, we note that with
RoIAlign, using stride-32 C5 features (30.9 AP) is more accurate than using stride-16 C4 features (30.3 AP, Table 2c).
RoIAlign largely resolves the long-standing challenge of
using large-stride features for detection and segmentation.
Finally, RoIAlign shows a gain of 1.5 mask AP and 0.5
box AP when used with FPN, which has ﬁner multi-level
strides. For keypoint detection that requires ﬁner alignment,
RoIAlign shows large gains even with FPN (Table 6).


key: chroma:document
string_value:
strides. For keypoint detection that requires ﬁner alignment,
RoIAlign shows large gains even with FPN (Table 6).
Mask Branch: Segmentation is a pixel-to-pixel task and
we exploit the spatial layout of masks by using an FCN.
In Table 2e, we compare multi-layer perceptrons (MLP)
and FCNs, using a ResNet-50-FPN backbone. Using FCNs
gives a 2.1 mask AP gain over MLPs. We note that we
choose this backbone so that the conv layers of the FCN
head are not pre-trained, for a fair comparison with MLP.
4.3. Bounding Box Detection Results
We compare Mask R-CNN to the state-of-the-art COCO
bounding-box object detection in Table 3. For this result,
even though the full Mask R-CNN model is trained, only
the classiﬁcation and box outputs are used at inference (the


key: chroma:document
string_value:
even though the full Mask R-CNN model is trained, only
the classiﬁcation and box outputs are used at inference (the
mask output is ignored). Mask R-CNN using ResNet-101FPN outperforms the base variants of all previous state-ofthe-art models, including the single-model variant of GRMI [21], the winner of the COCO 2016 Detection Challenge. Using ResNeXt-101-FPN, Mask R-CNN further improves results, with a margin of 3.0 points box AP over
the best previous single model entry from [39] (which used
Inception-ResNet-v2-TDM).
As a further comparison, we trained a version of Mask
R-CNN but without the mask branch, denoted by “Faster
R-CNN, RoIAlign” in Table 3. This model performs better
than the model presented in [27] due to RoIAlign. On the
other hand, it is 0.9 points box AP lower than Mask R-CNN.
This gap of Mask R-CNN on box detection is therefore due
solely to the beneﬁts of multi-task training.
Lastly, we note that Mask R-CNN attains a small gap
between its mask and box AP: e.g., 2.7 points between 37.1


key: chroma:document
string_value:
Lastly, we note that Mask R-CNN attains a small gap
between its mask and box AP: e.g., 2.7 points between 37.1
(mask, Table 1) and 39.8 (box, Table 3). This indicates that
our approach largely closes the gap between object detection and the more challenging instance segmentation task.4.4. Timing
Inference: We train a ResNet-101-FPN model that shares
features between the RPN and Mask R-CNN stages, following the 4-step training of Faster R-CNN [36]. This model
runs at 195ms per image on an Nvidia Tesla M40 GPU (plus
15ms CPU time resizing the outputs to the original resolution), and achieves statistically the same mask AP as the
unshared one. We also report that the ResNet-101-C4 variant takes 400ms as it has a heavier box head (Figure 4), so
we do not recommend using the C4 variant in practice.
Although Mask R-CNN is fast, we note that our design
is not optimized for speed, and better speed/accuracy tradeoffs could be achieved [21], e.g., by varying image sizes and
proposal numbers, which is beyond the scope of this paper.
Training: Mask R-CNN is also fast to train. Training with


key: chroma:document
string_value:
proposal numbers, which is beyond the scope of this paper.
Training: Mask R-CNN is also fast to train. Training with
ResNet-50-FPN on COCO trainval35k takes 32 hours
in our synchronized 8-GPU implementation (0.72s per 16image mini-batch), and 44 hours with ResNet-101-FPN. In
fact, fast prototyping can be completed in less than one day
when training on the train set. We hope such rapid training will remove a major hurdle in this area and encourage
more people to perform research on this challenging topic.
5. Mask R-CNN for Human Pose Estimation
Our framework can easily be extended to human pose
estimation. We model a keypoint’s location as a one-hot
mask, and adopt Mask R-CNN to predict Kmasks, one for
each of Kkeypoint types ( e.g., left shoulder, right elbow).
This task helps demonstrate the ﬂexibility of Mask R-CNN.
We note that minimal domain knowledge for human pose
is exploited by our system, as the experiments are mainly to
demonstrate the generality of the Mask R-CNN framework.
We expect that domain knowledge ( e.g., modeling structures [6]) will be complementary to our simple approach.
Implementation Details: We make minor modiﬁcations to


key: chroma:document
string_value:
We expect that domain knowledge ( e.g., modeling structures [6]) will be complementary to our simple approach.
Implementation Details: We make minor modiﬁcations to
the segmentation system when adapting it for keypoints.
For each of the Kkeypoints of an instance, the training
target is a one-hot mmbinary mask where only a single
pixel is labeled as foreground. During training, for each visible ground-truth keypoint, we minimize the cross-entropy
loss over an m2-way softmax output (which encourages a
7
Figure 7. Keypoint detection results on COCO test using Mask R-CNN (ResNet-50-FPN), with person segmentation masks predicted
from the same model. This model has a keypoint AP of 63.1 and runs at 5 fps.
APkpAPkp
50APkp
75APkp
MAPkp
L
CMU-Pose+++ [6] 61.8 84.9 67.5 57.1 68.2
G-RMI [32]y62.4 84.0 68.5 59.1 68.1
Mask R-CNN , keypoint-only 62.7 87.0 68.4 57.4 71.1


key: chroma:document
string_value:
Mask R-CNN , keypoint-only 62.7 87.0 68.4 57.4 71.1
Mask R-CNN , keypoint & mask 63.1 87.3 68.7 57.8 71.4
Table 4. Keypoint detection AP on COCO test-dev . Ours is a
single model (ResNet-50-FPN) that runs at 5 fps. CMU-Pose+++
[6] is the 2016 competition winner that uses multi-scale testing,
post-processing with CPM [44], and ﬁltering with an object detector, adding a cumulative 5 points (clariﬁed in personal communication).y: G-RMI was trained on COCO plus MPII [1] (25k images), using two models (Inception-ResNet-v2 for bounding box
detection and ResNet-101 for keypoints).
single point to be detected). We note that as in instance segmentation, the Kkeypoints are still treated independently.
We adopt the ResNet-FPN variant, and the keypoint head
architecture is similar to that in Figure 4 (right). The keypoint head consists of a stack of eight 3 3 512-d conv layers, followed by a deconv layer and 2 bilinear upscaling,


key: chroma:document
string_value:
producing an output resolution of 56 56. We found that
a relatively high resolution output (compared to masks) is
required for keypoint-level localization accuracy.
Models are trained on all COCO trainval35k images that contain annotated keypoints. To reduce overﬁtting, as this training set is smaller, we train using image
scales randomly sampled from [640, 800] pixels; inference
is on a single scale of 800 pixels. We train for 90k iterations,
starting from a learning rate of 0.02 and reducing it by 10 at
60k and 80k iterations. We use bounding-box NMS with a
threshold of 0.5. Other details are identical as in x3.1.
Main Results and Ablations: We evaluate the person keypoint AP (APkp) and experiment with a ResNet-50-FPN
backbone; more backbones will be studied in the appendix.
Table 4 shows that our result (62.7 APkp) is 0.9 points higher
than the COCO 2016 keypoint detection winner [6] that
uses a multi-stage processing pipeline (see caption of Table 4). Our method is considerably simpler and faster.
More importantly, we have a uniﬁed model that can si-APbb
person APmask
person APkp


key: chroma:document
string_value:
More importantly, we have a uniﬁed model that can si-APbb
person APmask
person APkp
Faster R-CNN 52.5 - Mask R-CNN, mask-only 53.6 45.8 Mask R-CNN, keypoint-only 50.7 - 64.2
Mask R-CNN, keypoint & mask 52.0 45.1 64.7
Table 5. Multi-task learning of box, mask, and keypoint about the
person category, evaluated on minival . All entries are trained
on the same data for fair comparisons. The backbone is ResNet50-FPN. The entries with 64.2 and 64.7 AP on minival have
test-dev AP of 62.7 and 63.1, respectively (see Table 4).
APkpAPkp
50APkp
75APkp
MAPkp
L
RoIPool 59.8 86.2 66.7 55.1 67.4
RoIAlign 64.2 86.6 69.7 58.7 73.0
Table 6. RoIAlign vs. RoIPool for keypoint detection on
minival . The backbone is ResNet-50-FPN.
multaneously predict boxes, segments, and keypoints while


key: chroma:document
string_value:
minival . The backbone is ResNet-50-FPN.
multaneously predict boxes, segments, and keypoints while
running at 5 fps. Adding a segment branch (for the person category) improves the APkpto 63.1 (Table 4) on
test-dev . More ablations of multi-task learning on
minival are in Table 5. Adding the mask branch to the
box-only ( i.e., Faster R-CNN) or keypoint-only versions
consistently improves these tasks. However, adding the
keypoint branch reduces the box/mask AP slightly, suggesting that while keypoint detection beneﬁts from multitask
training, it does not in turn help the other tasks. Nevertheless, learning all three tasks jointly enables a uniﬁed system
to efﬁciently predict all outputs simultaneously (Figure 7).
We also investigate the effect of RoIAlign on keypoint
detection (Table 6). Though this ResNet-50-FPN backbone
has ﬁner strides ( e.g., 4 pixels on the ﬁnest level), RoIAlign
still shows signiﬁcant improvement over RoIPool and increases APkpby 4.4 points. This is because keypoint detections are more sensitive to localization accuracy. This again


key: chroma:document
string_value:
indicates that alignment is essential for pixel-level localization, including masks and keypoints.
Given the effectiveness of Mask R-CNN for extracting
object bounding boxes, masks, and keypoints, we expect it
be an effective framework for other instance-level tasks.
8
training data AP [val] AP AP 50 person rider car truck bus train mcycle bicycle
InstanceCut [23] fine +coarse 15.8 13.0 27.9 10.0 8.0 23.7 14.0 19.5 15.2 9.3 4.7
DWT [4] fine 19.8 15.6 30.0 15.1 11.7 32.9 17.1 20.4 15.0 7.9 4.9
SAIS [17] fine - 17.4 36.7 14.6 12.9 35.7 16.0 23.2 19.0 10.3 7.8
DIN [3] fine +coarse - 20.0 38.8 16.5 16.7 25.7 20.6 30.0 23.4 17.1 10.1
SGN [29] fine +coarse 29.2 25.0 44.9 21.8 20.1 39.4 24.8 33.2 30.8 17.7 12.4


key: chroma:document
string_value:
Mask R-CNN fine 31.5 26.2 49.9 30.5 23.7 46.9 22.8 32.2 18.6 19.1 16.0
Mask R-CNN fine + COCO 36.4 32.0 58.1 34.8 27.0 49.1 30.1 40.9 30.9 24.1 18.7
Table 7. Results on Cityscapes val (‘AP [ val]’ column) and test (remaining columns) sets. Our method uses ResNet-50-FPN.
Appendix A: Experiments on Cityscapes
We further report instance segmentation results on the
Cityscapes [7] dataset. This dataset has fine annotations for 2975 train, 500 val, and 1525 test images. It has
20kcoarse training images without instance annotations,
which we do notuse. All images are 2048 1024 pixels.
The instance segmentation task involves 8 object categories,
whose numbers of instances on the fine training set are:
person rider car truck bus train mcycle bicycle
17.9k 1.8k 26.9k 0.5k 0.4k 0.2k 0.7k 3.7k
Instance segmentation performance on this task is measured


key: chroma:document
string_value:
Instance segmentation performance on this task is measured
by the COCO-style mask AP (averaged over IoU thresholds); AP 50(i.e., mask AP at an IoU of 0.5) is also reported.
Implementation: We apply our Mask R-CNN models with
the ResNet-FPN-50 backbone; we found the 101-layer
counterpart performs similarly due to the small dataset size.
We train with image scale (shorter side) randomly sampled
from [800, 1024], which reduces overﬁtting; inference is on
a single scale of 1024 pixels. We use a mini-batch size of
1 image per GPU (so 8 on 8 GPUs) and train the model
for 24k iterations, starting from a learning rate of 0.01 and
reducing it to 0.001 at 18k iterations. It takes 4 hours of
training on a single 8-GPU machine under this setting.
Results: Table 7 compares our results to the state of the
art on the val andtest sets. Without using the coarse
training set, our method achieves 26.2 AP on test , which
is over 30% relative improvement over the previous best entry (DIN [3]), and is also better than the concurrent work of
SGN’s 25.0 [29]. Both DIN and SGN use fine +coarse


key: chroma:document
string_value:
SGN’s 25.0 [29]. Both DIN and SGN use fine +coarse
data. Compared to the best entry using fine data only
(17.4 AP), we achieve a 50% improvement.
For the person andcarcategories, the Cityscapes dataset
exhibits a large number of within -category overlapping instances (on average 6 people and 9 cars per image). We
argue that within-category overlap is a core difﬁculty of instance segmentation. Our method shows massive improvement on these two categories over the other best entries (relative40% improvement on person from 21.8 to 30.5 and
20% improvement on carfrom 39.4 to 46.9), even though
our method does not exploit the coarse data.
A main challenge of the Cityscapes dataset is training
models in a low-data regime, particularly for the categories
oftruck ,bus, and train , which have about 200-500 traincar:1.00car:0.98
car:0.98 car:0.95 car:0.81car:0.52person:1.00person:1.00person:1.00
person:1.00 person:1.00person:1.00person:1.00
person:1.00 person:1.00person:1.00


key: chroma:document
string_value:
person:1.00 person:1.00person:1.00person:1.00
person:1.00 person:1.00person:1.00
person:1.00person:1.00
person:1.00
person:1.00person:0.99
person:0.99person:0.99 person:0.99person:0.98
person:0.98person:0.98person:0.98person:0.94 person:0.94person:0.82
person:0.82person:0.79
person:0.73person:0.67person:0.66person:0.59
truck:0.66bus:1.00
bus:0.95rider:0.59
bicycle:0.83
bicycle:0.56car:1.00car:1.00
car:1.00 car:1.00car:1.00car:1.00
car:1.00car:1.00 car:1.00car:0.99car:0.95
car:0.95car:0.95 car:0.69car:0.68 car:0.68car:0.64
car:0.57 car:0.52person:1.00


key: chroma:document
string_value:
car:0.57 car:0.52person:1.00
person:0.99person:0.99person:0.99 person:0.99person:0.98person:0.98 person:0.98 person:0.97 person:0.93person:0.92
person:0.91person:0.86 person:0.84person:0.82 person:0.73person:0.72 person:0.72person:0.72 person:0.63 rider:0.68
car:1.00
car:1.00car:1.00car:1.00
car:1.00car:1.00car:1.00car:1.00
car:1.00 car:1.00
car:1.00car:1.00
car:1.00car:1.00car:1.00car:1.00car:1.00
car:0.98car:0.97
car:0.88car:0.76car:0.72car:0.72 car:0.65car:0.50person:1.00
person:1.00 person:0.98person:0.93person:0.85person:0.78person:0.73 person:0.58
person:1.00


key: chroma:document
string_value:
person:1.00
person:1.00person:1.00
person:1.00
person:1.00person:1.00
person:1.00person:1.00person:1.00
person:1.00person:1.00person:1.00
person:1.00person:1.00
person:1.00person:1.00person:1.00
person:1.00person:1.00
person:1.00person:1.00
person:1.00person:0.99
person:0.99 person:0.98person:0.97
person:0.96person:0.92
person:0.91person:0.70 person:0.59
bicycle:0.99bicycle:0.97car:1.00
car:1.00 car:0.99
car:0.89person:1.00
person:1.00person:1.00
person:1.00
person:1.00person:0.96person:0.93
person:0.89person:0.88person:0.75
rider:0.94
car:1.00
car:1.00car:1.00
car:1.00car:1.00


key: chroma:document
string_value:
rider:0.94
car:1.00
car:1.00car:1.00
car:1.00car:1.00
car:1.00car:1.00 car:0.99car:0.89 car:0.67person:1.00
person:1.00
person:1.00 person:1.00 person:0.82
bus:0.75Figure 8. Mask R-CNN results on Cityscapes test (32.0 AP).
The bottom-right image shows a failure prediction.
ing samples each. To partially remedy this issue, we further
report a result using COCO pre-training. To do this, we initialize the corresponding 7 categories in Cityscapes from a
pre-trained COCO Mask R-CNN model ( rider being randomly initialized). We ﬁne-tune this model for 4k iterations
in which the learning rate is reduced at 3k iterations, which
takes1 hour for training given the COCO model.
The COCO pre-trained Mask R-CNN model achieves
32.0 AP on test , almost a 6 point improvement over the
fine -only counterpart. This indicates the important role
the amount of training data plays. It also suggests that


key: chroma:document
string_value:
32.0 AP on test , almost a 6 point improvement over the
fine -only counterpart. This indicates the important role
the amount of training data plays. It also suggests that
methods on Cityscapes might be inﬂuenced by their lowshot learning performance. We show that using COCO pretraining is an effective strategy on this dataset.
Finally, we observed a bias between the val andtest
AP, as is also observed from the results of [23, 4, 29]. We
found that this bias is mainly caused by the truck ,bus,
and train categories, with the fine -only model having
val/test AP of 28.8/22.8, 53.5/32.2, and 33.0/18.6, respectively. This suggests that there is a domain shift on
these categories, which also have little training data. COCO
pre-training helps to improve results the most on these categories; however, the domain shift persists with 38.0/30.1,
57.5/40.9, and 41.2/30.9 val/test AP, respectively. Note
that for the person andcarcategories we do not see any
such bias ( val/test AP are within1point).
Example results on Cityscapes are shown in Figure 8.
9
description backbone AP AP 50 AP75 APbbAPbb


key: chroma:document
string_value:
such bias ( val/test AP are within1point).
Example results on Cityscapes are shown in Figure 8.
9
description backbone AP AP 50 AP75 APbbAPbb
50APbb
75
original baseline X-101-FPN 36.7 59.5 38.9 39.6 61.5 43.2
+updated baseline X-101-FPN 37.0 59.7 39.0 40.5 63.0 43.7
+e2e training X-101-FPN 37.6 60.4 39.9 41.7 64.1 45.2
+ImageNet-5k X-101-FPN 38.6 61.7 40.9 42.7 65.1 46.6
+train-time augm. X-101-FPN 39.2 62.5 41.6 43.5 65.9 47.2
+deeper X-152-FPN 39.7 63.2 42.2 44.1 66.4 48.4
+Non-local [43] X-152-FPN-NL 40.3 64.4 42.8 45.0 67.8 48.9
+test-time augm. X-152-FPN-NL 41.8 66.0 44.8 47.3 69.3 51.5


key: chroma:document
string_value:
+test-time augm. X-152-FPN-NL 41.8 66.0 44.8 47.3 69.3 51.5
Table 8. Enhanced detection results of Mask R-CNN on COCO
minival . Each row adds an extra component to the above row.
We denote ResNeXt model by ‘X’ for notational brevity.
Appendix B: Enhanced Results on COCO
As a general framework, Mask R-CNN is compatible with complementary techniques developed for detection/segmentation, including improvements made to
Fast/Faster R-CNN and FCNs. In this appendix we describe some techniques that improve over our original results. Thanks to its generality and ﬂexibility, Mask R-CNN
was used as the framework by the three winning teams in
the COCO 2017 instance segmentation competition, which
all signiﬁcantly outperformed the previous state of the art.
Instance Segmentation and Object Detection
We report some enhanced results of Mask R-CNN in Table 8. Overall, the improvements increase mask AP 5.1
points (from 36.7 to 41.8) and box AP 7.7 points (from 39.6
to 47.3). Each model improvement increases both mask AP
and box AP consistently, showing good generalization of
the Mask R-CNN framework. We detail the improvements


key: chroma:document
string_value:
to 47.3). Each model improvement increases both mask AP
and box AP consistently, showing good generalization of
the Mask R-CNN framework. We detail the improvements
next. These results, along with future updates, can be reproduced by our released code at https://github.com/
facebookresearch/Detectron , and can serve as
higher baselines for future research.
Updated baseline: We start with an updated baseline
with a different set of hyper-parameters. We lengthen the
training to 180k iterations, in which the learning rate is reduced by 10 at 120k and 160k iterations. We also change
the NMS threshold to 0.5 (from a default value of 0.3). The
updated baseline has 37.0 mask AP and 40.5 box AP.
End-to-end training: All previous results used stagewise training, i.e., training RPN as the ﬁrst stage and Mask
R-CNN as the second. Following [37], we evaluate endto-end (‘e2e’) training that jointly trains RPN and Mask RCNN. We adopt the ‘approximate’ version in [37] that only
computes partial gradients in the RoIAlign layer by ignoring the gradient w.r.t. RoI coordinates. Table 8 shows that


key: chroma:document
string_value:
computes partial gradients in the RoIAlign layer by ignoring the gradient w.r.t. RoI coordinates. Table 8 shows that
e2e training improves mask AP by 0.6 and box AP by 1.2.
ImageNet-5k pre-training: Following [45], we experiment with models pre-trained on a 5k-class subset of ImageNet (in contrast to the standard 1k-class subset). This 5 
increase in pre-training data improves both mask and box 1
AP. As a reference, [40] used 250more images (300M)
and reported a 2-3 box AP improvement on their baselines.description backbone APkpAPkp
50APkp
75APkp
MAPkp
L
original baseline R-50-FPN 64.2 86.6 69.7 58.7 73.0
+updated baseline R-50-FPN 65.1 86.6 70.9 59.9 73.6
+deeper R-101-FPN 66.1 87.7 71.7 60.5 75.0
+ResNeXt X-101-FPN 67.3 88.0 73.3 62.2 75.6


key: chroma:document
string_value:
+ResNeXt X-101-FPN 67.3 88.0 73.3 62.2 75.6
+data distillation [35] X-101-FPN 69.1 88.9 75.3 64.1 77.1
+test-time augm. X-101-FPN 70.4 89.3 76.8 65.8 78.1
Table 9. Enhanced keypoint results of Mask R-CNN on COCO
minival . Each row adds an extra component to the above row.
Here we use only keypoint annotations but no mask annotations.
We denote ResNet by ‘R’ and ResNeXt by ‘X’ for brevity.
Train-time augmentation: Scale augmentation at train
time further improves results. During training, we randomly
sample a scale from [640, 800] pixels and we increase the
number of iterations to 260k (with the learning rate reduced
by 10 at 200k and 240k iterations). Train-time augmentation improves mask AP by 0.6 and box AP by 0.8.
Model architecture: By upgrading the 101-layer
ResNeXt to its 152-layer counterpart [19], we observe an
increase of 0.5 mask AP and 0.6 box AP. This shows a
deeper model can still improve results on COCO.


key: chroma:document
string_value:
increase of 0.5 mask AP and 0.6 box AP. This shows a
deeper model can still improve results on COCO.
Using the recently proposed non-local (NL) model [43],
we achieve 40.3 mask AP and 45.0 box AP. This result is
without test-time augmentation, and the method runs at 3fps
on an Nvidia Tesla P100 GPU at test time.
Test-time augmentation: We combine the model results
evaluated using scales of [400, 1200] pixels with a step of
100 and on their horizontal ﬂips. This gives us a singlemodel result of 41.8 mask AP and 47.3 box AP.
The above result is the foundation of our submission to
the COCO 2017 competition (which also used an ensemble,
not discussed here). The ﬁrst three winning teams for the
instance segmentation task were all reportedly based on an
extension of the Mask R-CNN framework.
Keypoint Detection
We report enhanced results of keypoint detection in Table 9. As an updated baseline, we extend the training schedule to 130k iterations in which the learning rate is reduced
by 10 at 100k and 120k iterations. This improves APkpby
about 1 point. Replacing ResNet-50 with ResNet-101 and


key: chroma:document
string_value:
by 10 at 100k and 120k iterations. This improves APkpby
about 1 point. Replacing ResNet-50 with ResNet-101 and
ResNeXt-101 increases APkpto 66.1 and 67.3, respectively.
With a recent method called data distillation [35], we are
able to exploit the additional 120k unlabeled images provided by COCO. In brief, data distillation is a self-training
strategy that uses a model trained on labeled data to predict annotations on unlabeled images, and in turn updates
the model with these new annotations. Mask R-CNN provides an effective framework for such a self-training strategy. With data distillation, Mask R-CNN APkpimprove by
1.8 points to 69.1. We observe that Mask R-CNN can beneﬁt from extra data, even if that data is unlabeled .
By using the same test-time augmentation as used for
instance segmentation, we further boost APkpto 70.4.
10
Acknowledgements: We would like to acknowledge Ilija
Radosavovic for contributions to code release and enhanced
results, and the Caffe2 team for engineering support.
References
[1] M. Andriluka, L. Pishchulin, P. Gehler, and B. Schiele. 2D


key: chroma:document
string_value:
References
[1] M. Andriluka, L. Pishchulin, P. Gehler, and B. Schiele. 2D
human pose estimation: New benchmark and state of the art
analysis. In CVPR , 2014. 8
[2] P. Arbel ´aez, J. Pont-Tuset, J. T. Barron, F. Marques, and
J. Malik. Multiscale combinatorial grouping. In CVPR ,
2014. 2
[3] A. Arnab and P. H. Torr. Pixelwise instance segmentation
with a dynamically instantiated network. In CVPR , 2017. 3,
9
[4] M. Bai and R. Urtasun. Deep watershed transform for instance segmentation. In CVPR , 2017. 3, 9
[5] S. Bell, C. L. Zitnick, K. Bala, and R. Girshick. Insideoutside net: Detecting objects in context with skip pooling
and recurrent neural networks. In CVPR , 2016. 5
[6] Z. Cao, T. Simon, S.-E. Wei, and Y . Sheikh. Realtime multiperson 2d pose estimation using part afﬁnity ﬁelds. In CVPR ,
2017. 7, 8


key: chroma:document
string_value:
2017. 7, 8
[7] M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler,
R. Benenson, U. Franke, S. Roth, and B. Schiele. The
Cityscapes dataset for semantic urban scene understanding.
InCVPR , 2016. 9
[8] J. Dai, K. He, Y . Li, S. Ren, and J. Sun. Instance-sensitive
fully convolutional networks. In ECCV , 2016. 2
[9] J. Dai, K. He, and J. Sun. Convolutional feature masking for
joint object and stuff segmentation. In CVPR , 2015. 2
[10] J. Dai, K. He, and J. Sun. Instance-aware semantic segmentation via multi-task network cascades. In CVPR , 2016. 2, 3,
4, 5, 6
[11] J. Dai, Y . Li, K. He, and J. Sun. R-FCN: Object detection via
region-based fully convolutional networks. In NIPS , 2016. 2
[12] R. Girshick. Fast R-CNN. In ICCV , 2015. 1, 2, 3, 4, 6


key: chroma:document
string_value:
[12] R. Girshick. Fast R-CNN. In ICCV , 2015. 1, 2, 3, 4, 6
[13] R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature hierarchies for accurate object detection and semantic
segmentation. In CVPR , 2014. 2, 3
[14] R. Girshick, F. Iandola, T. Darrell, and J. Malik. Deformable
part models are convolutional neural networks. In CVPR ,
2015. 4
[15] B. Hariharan, P. Arbel ´aez, R. Girshick, and J. Malik. Simultaneous detection and segmentation. In ECCV . 2014. 2
[16] B. Hariharan, P. Arbel ´aez, R. Girshick, and J. Malik. Hypercolumns for object segmentation and ﬁne-grained localization. In CVPR , 2015. 2
[17] Z. Hayder, X. He, and M. Salzmann. Shape-aware instance
segmentation. In CVPR , 2017. 9
[18] K. He, X. Zhang, S. Ren, and J. Sun. Spatial pyramid pooling
in deep convolutional networks for visual recognition. In


key: chroma:document
string_value:
[18] K. He, X. Zhang, S. Ren, and J. Sun. Spatial pyramid pooling
in deep convolutional networks for visual recognition. In
ECCV . 2014. 1, 2
[19] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning
for image recognition. In CVPR , 2016. 2, 4, 7, 10
[20] J. Hosang, R. Benenson, P. Doll ´ar, and B. Schiele. What
makes for effective detection proposals? PAMI , 2015. 2[21] J. Huang, V . Rathod, C. Sun, M. Zhu, A. Korattikara,
A. Fathi, I. Fischer, Z. Wojna, Y . Song, S. Guadarrama, et al.
Speed/accuracy trade-offs for modern convolutional object
detectors. In CVPR , 2017. 2, 3, 4, 6, 7
[22] M. Jaderberg, K. Simonyan, A. Zisserman, and
K. Kavukcuoglu. Spatial transformer networks. In
NIPS , 2015. 4
[23] A. Kirillov, E. Levinkov, B. Andres, B. Savchynskyy, and


key: chroma:document
string_value:
NIPS , 2015. 4
[23] A. Kirillov, E. Levinkov, B. Andres, B. Savchynskyy, and
C. Rother. Instancecut: from edges to instances with multicut. In CVPR , 2017. 3, 9
[24] A. Krizhevsky, I. Sutskever, and G. Hinton. ImageNet classiﬁcation with deep convolutional neural networks. In NIPS ,
2012. 2
[25] Y . LeCun, B. Boser, J. S. Denker, D. Henderson, R. E.
Howard, W. Hubbard, and L. D. Jackel. Backpropagation
applied to handwritten zip code recognition. Neural computation , 1989. 2
[26] Y . Li, H. Qi, J. Dai, X. Ji, and Y . Wei. Fully convolutional
instance-aware semantic segmentation. In CVPR , 2017. 2,
3, 5, 6
[27] T.-Y . Lin, P. Doll ´ar, R. Girshick, K. He, B. Hariharan, and
S. Belongie. Feature pyramid networks for object detection.
InCVPR , 2017. 2, 4, 5, 7


key: chroma:document
string_value:
S. Belongie. Feature pyramid networks for object detection.
InCVPR , 2017. 2, 4, 5, 7
[28] T.-Y . Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Doll ´ar, and C. L. Zitnick. Microsoft COCO: Common objects in context. In ECCV , 2014. 2, 5
[29] S. Liu, J. Jia, S. Fidler, and R. Urtasun. SGN: Sequential grouping networks for instance segmentation. In ICCV ,
2017. 3, 9
[30] J. Long, E. Shelhamer, and T. Darrell. Fully convolutional
networks for semantic segmentation. In CVPR , 2015. 1, 3, 6
[31] V . Nair and G. E. Hinton. Rectiﬁed linear units improve restricted boltzmann machines. In ICML , 2010. 4
[32] G. Papandreou, T. Zhu, N. Kanazawa, A. Toshev, J. Tompson, C. Bregler, and K. Murphy. Towards accurate multiperson pose estimation in the wild. In CVPR , 2017. 8


key: chroma:document
string_value:
[33] P. O. Pinheiro, R. Collobert, and P. Dollar. Learning to segment object candidates. In NIPS , 2015. 2, 3
[34] P. O. Pinheiro, T.-Y . Lin, R. Collobert, and P. Doll ´ar. Learning to reﬁne object segments. In ECCV , 2016. 2, 3
[35] I. Radosavovic, P. Doll ´ar, R. Girshick, G. Gkioxari, and
K. He. Data distillation: Towards omni-supervised learning.
arXiv:1712.04440 , 2017. 10
[36] S. Ren, K. He, R. Girshick, and J. Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. In NIPS , 2015. 1, 2, 3, 4, 7
[37] S. Ren, K. He, R. Girshick, and J. Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. In TPAMI , 2017. 10
[38] A. Shrivastava, A. Gupta, and R. Girshick. Training regionbased object detectors with online hard example mining. In
CVPR , 2016. 2, 5


key: chroma:document
string_value:
CVPR , 2016. 2, 5
[39] A. Shrivastava, R. Sukthankar, J. Malik, and A. Gupta. Beyond skip connections: Top-down modulation for object detection. arXiv:1612.06851 , 2016. 4, 7
[40] C. Sun, A. Shrivastava, S. Singh, and A. Gupta. Revisiting
unreasonable effectiveness of data in deep learning era. In
ICCV , 2017. 10
11
[41] C. Szegedy, S. Ioffe, and V . Vanhoucke. Inception-v4,
inception-resnet and the impact of residual connections on
learning. In ICLR Workshop , 2016. 7
[42] J. R. Uijlings, K. E. van de Sande, T. Gevers, and A. W.
Smeulders. Selective search for object recognition. IJCV ,
2013. 2
[43] X. Wang, R. Girshick, A. Gupta, and K. He. Non-local neural
networks. arXiv:1711.07971 , 2017. 10


key: chroma:document
string_value:
networks. arXiv:1711.07971 , 2017. 10
[44] S.-E. Wei, V . Ramakrishna, T. Kanade, and Y . Sheikh. Convolutional pose machines. In CVPR , 2016. 8
[45] S. Xie, R. Girshick, P. Doll ´ar, Z. Tu, and K. He. Aggregated
residual transformations for deep neural networks. In CVPR ,
2017. 4, 10
12


key: chroma:document
string_value:
Pixelwise Instance Segmentation with a Dynamically Instantiated Network
Anurag Arnab and Philip H.S Torr
University of Oxford
{anurag.arnab, philip.torr }@eng.ox.ac.uk
Abstract
Semantic segmentation and object detection research
have recently achieved rapid progress. However, the former
task has no notion of different instances of the same object,
and the latter operates at a coarse, bounding-box level. We
propose an Instance Segmentation system that produces a
segmentation map where each pixel is assigned an object
class and instance identity label. Most approaches adapt
object detectors to produce segments instead of boxes. In
contrast, our method is based on an initial semantic segmentation module, which feeds into an instance subnetwork.
This subnetwork uses the initial category-level segmentation, along with cues from the output of an object detector,
within an end-to-end CRF to predict instances. This part
of our model is dynamically instantiated to produce a variable number of instances per image. Our end-to-end approach requires no post-processing and considers the image holistically, instead of processing independent proposals. Therefore, unlike some related work, a pixel cannot belong to multiple instances. Furthermore, far more precise
segmentations are achieved, as shown by our substantial


key: chroma:document
string_value:
segmentations are achieved, as shown by our substantial
improvements at high APrthresholds.
1. Introduction
Semantic segmentation and object detection are wellstudied scene understanding problems, and have recently
witnessed great progress due to deep learning [ 22,13,7].
However, semantic segmentation – which labels every pixel
in an image with its object class – has no notion of different
instances of an object (Fig. 1). Object detection does localise different object instances, but does so at a very coarse,
bounding-box level. Instance segmentation localises objects at a pixel level, as shown in Fig. 1, and can be thought
of being at the intersection of these two scene understanding
tasks. Unlike the former, it knows about different instances
of the same object, and unlike the latter, it operates at a pixel
level. Accurate recognition and localisation of objects enables many applications, such as autonomous driving [ 9],
image-editing [ 53] and robotics [ 17].
Many recent approaches to instance segmentation arebased on object detection pipelines where objects are ﬁrst
localised with bounding boxes. Thereafter, each bounding


key: chroma:document
string_value:
Many recent approaches to instance segmentation arebased on object detection pipelines where objects are ﬁrst
localised with bounding boxes. Thereafter, each bounding
box is reﬁned into a segmentation [ 19,20,32,37,30]. Another related approach [ 12,56] is to use segment-based region proposals [ 10,41,42] instead of box-based proposals.
However, these methods do not consider the entire image,
but rather independent proposals. As a result, occlusions
between different objects are not handled. Furthermore,
many of these methods cannot easily produce segmentation
maps of the image, as shown in Fig. 1, since they process
numerous proposals independently. There are typically far
more proposals than actual objects in the image, and these
proposals can overlap and be assigned different class labels.
Finally, as these methods are based on an initial detection
step, they cannot recover from false detections.
Our proposed method is inspired by the fact that instance
segmentation can be viewed as a more complex form of semantic segmentation, since we are not only required to label the object class of each pixel, but also its instance identity. We produce a pixelwise segmentation of the image,


key: chroma:document
string_value:
where each pixel is assigned both a semantic class and instance label. Our end-to-end trained network, which outputs a variable number of instances per input image, begins with an initial semantic segmentation module. The
following, dynamic part of the network, then uses information from an object detector and a Conditional Random
Field (CRF) model to distinguish different instances. This
approach is robust to false-positive detections, as well as
poorly localised bounding boxes which do not cover the
entire object, in contrast to detection-based methods to instance segmentation. Moreover, as it considers the entire
image when making predictions, it attempts to resolve occlusions between different objects and can produce segmentation maps as in Fig. 1without any post-processing.
Furthermore, we note that the Average Precision (AP)
metric [ 14] used in evaluating object detection systems, and
itsAPrvariant [ 19] used for instance segmentation, considers individual, potentially overlapping, object predictions in
isolation, as opposed to the entire image. To evaluate methods such as ours, which produce complete segmentation
maps and reason about occlusions, we also evaluate using
1
arXiv:1704.02386v1  [cs.CV]  7 Apr 2017
(a) Object Detection (b) Semantic Segmentation (c) Instance Segmentation


key: chroma:document
string_value:
(a) Object Detection (b) Semantic Segmentation (c) Instance Segmentation
Figure 1: Object detection (a) localises the different people, but at a coarse, bounding-box level. Semantic segmentation
(b) labels every pixel, but has no notion of instances. Instance segmentation (c) labels each pixel of each person uniquely.
Our proposed method jointly produces both semantic and instance segmentations. Our method uses the output of an object
detector as a cue to identify instances, but is robust to false positive detections, poor bounding box localisation and occlusions.
Best viewed in colour.
the “Matching Intersection over Union” metric.
Our system, which is based on an initial semantic segmentation subnetwork, produces sharp and accurate instance segmentations. This is reﬂected by the substantial improvements we achieve over state-of-the-art methods
at highAPrthresholds on the Pascal VOC and Semantic
Boundaries datasets. Furthermore, our network improves
on the semantic segmentation task while being trained for
the related task of instance segmentation.
2. Related Work
An early work on instance segmentation was by Winn
and Shotton [ 51]. A per-pixel unary classiﬁer was trained


key: chroma:document
string_value:
2. Related Work
An early work on instance segmentation was by Winn
and Shotton [ 51]. A per-pixel unary classiﬁer was trained
to predict parts of an object. These parts were then encouraged to maintain a spatial ordering, that is characteristic of
an instance, using asymmetric pairwise potentials in a Conditional Random Field (CRF). Subsequent work [ 54], presented another approach where detection outputs of DPM
[15], with associated foreground masks, were assigned a
depth ordering using a generative, probabilistic model. This
depth ordering resolved occlusions.
However, instance segmentation has become more common after the “Simultaneous Detection and Segmentation”
(SDS) work of Hariharan et al. [19]. This system was based
on the R-CNN pipeline [ 16]: Region proposals, generated
by the method of [ 1], were classiﬁed into object categories
with a Convolutional Neural Network (CNN) before applying bounding-box regression as post-processing. A classspeciﬁc segmentation was then performed in this bounding


key: chroma:document
string_value:
with a Convolutional Neural Network (CNN) before applying bounding-box regression as post-processing. A classspeciﬁc segmentation was then performed in this bounding
box to simultaneously detect and segment the object. Numerous works [ 20,8,30] have extended this pipeline. However, approaches that segment instances by reﬁning detections [ 19,20,8,11,30] are inherently limited by the quality of the initial proposals. This problem is exacerbated by
the fact that this pipeline consists of several different modules trained with different objective functions. Furthermore,numerous post-processing steps such as “superpixel projection” and rescoring are performed. Dai et al. [12] addressed
some of these issues by designing one end-to-end trained
network that generates box-proposals, creates foreground
masks from these proposals and then classiﬁes these masks.
This network can be seen as an extension of the end-to-end
Faster-RCNN [ 44] detection framework, which generates
box-proposals and classiﬁes them. Additionally, Liu et al.
[37] formulated an end-to-end version of the SDS network
[19], whilst [ 32] iteratively reﬁned object proposals.
On a separate track, algorithms have also been developed


key: chroma:document
string_value:
[19], whilst [ 32] iteratively reﬁned object proposals.
On a separate track, algorithms have also been developed
that do not require object detectors. Zhang et al. [57,58]
segmented car instances by predicting the depth ordering
of each pixel in the image. Unlike the previous detectionbased approaches, this method reasoned globally about all
instances in the image simultaneously (rather than individual proposals) with an MRF-based formulation. However,
inference of this graphical model was not performed end-toend as shown to be possible in [ 60,2,5,34]. Furthermore,
although this method does not use object detections, it is
trained with ground truth depth and assumes a maximum
of nine cars in an image. Predicting all the instances in
an image simultaneously (rather than classifying individual
proposals) requires a model to be able to handle a variable
number of output instances per image. As a result, [ 45]
proposed a Recurrent Neural Network (RNN) for this task.
However, this model was only for a single object category.
Our proposed method not only outputs a variable number of
instances, but can also handle multiple object classes.
Liang et al. [33] developed another proposal-free method
based on the semantic segmentation network of [ 6]. The


key: chroma:document
string_value:
instances, but can also handle multiple object classes.
Liang et al. [33] developed another proposal-free method
based on the semantic segmentation network of [ 6]. The
category-level segmentation, along with CNN features, was
used to predict instance-level bounding boxes. The number
of instances of each class was also predicted to enable a ﬁnal
spectral clustering step. However, this additional information predicted by Liang’s network could have been obtained
 Semantic Segmentation subnetwork  Instance Segmentation subnetwork Instance Unary Potentials Pixelwise 
CNN Semantic CRF Detector Box 
Global 
Shape Instance CRF 
Input Image Instance Segmentation 
Semantic Segmentation End-to-end Network 
K+1 
D+1 
Figure 2: Network overview: Our end-to-end trained network consists of semantic- and instance-segmentation modules. The
intermediate category-level segmentation, along with the outputs of an object detector, are used to reason about instances.
This is done by instance unary terms which use information from the detector’s bounding boxes, the initial semantic segmentation and also the object’s shape. A ﬁnal CRF is used to combine all this information together to obtain an instance


key: chroma:document
string_value:
segmentation. The output of the semantic segmentation module is a ﬁxed size W×H×(K+ 1) tensor where Kis the
number of object classes, excluding background, in the dataset. The ﬁnal output, however, is of a variable W×H×(D+ 1)
dimensions where Dis the number of detected objects (and one background label).
from an object detector. Arnab et al. [3] also started with an
initial semantic segmentation network [ 2], and combined
this with the outputs of an object detector using a CRF to
reason about instances. This method was not trained endto-end though, and could not really recover from errors in
bounding-box localisation or occlusion.
Our method also has an initial semantic segmentation
subnetwork, and uses the outputs of an object detector.
However, in contrast to [ 3] it is trained end-to-end to improve on both semantic- and instance-segmentation performance (to our knowledge, this is the ﬁrst work to achieve
this). Furthermore, it can handle detector localisation errors and occlusions better due to the energy terms in our
end-to-end CRF. In contrast to detection-based approaches
[19,20,12,37], our network requires no additional postprocessing to create an instance segmentation map as in


key: chroma:document
string_value:
[19,20,12,37], our network requires no additional postprocessing to create an instance segmentation map as in
Fig. 1(c) and reasons about the entire image, rather than
independent proposals. This global reasoning allows our
method to produce more accurate segmentations. Our proposed system also handles a variable number of instances
per image, and thus does not assume a maximum number
of instances like [ 57,58].
3. Proposed Approach
Our network (Fig. 2) contains an initial semantic segmentation module. We use the semantic segmentation result, along with the outputs of an object detector, to compute
the unary potentials of a Conditional Random Field (CRF)
deﬁned over object instances. We perform mean ﬁeld inference in this random ﬁeld to obtain the Maximum a Posteriori (MAP) estimate, which is our labelling. Although ournetwork consists of two conceptually different parts – a semantic segmentation module, and an instance segmentation
network – the entire pipeline is fully differentiable, given
object detections, and trained end-to-end.
3.1. Semantic Segmentation subnetwork
Semantic Segmentation assigns each pixel in an image a
semantic class label from a given set, L. In our case, this
module uses the FCN8s architecture [ 38] which is based


key: chroma:document
string_value:
semantic class label from a given set, L. In our case, this
module uses the FCN8s architecture [ 38] which is based
on the VGG [ 47] ImageNet model. For better segmentation results, we include mean ﬁeld inference of a Conditional Random Field as the last layer of this module. This
CRF contains the densely-connected pairwise potentials described in [ 26] and is formulated as a recurrent neural network as in [ 60]. Additionally, we include the Higher Order
detection potential described in [ 2]. This detection potential has two primary beneﬁts: Firstly, it improves semantic
segmentation quality by encouraging consistency between
object detections and segmentations. Secondly, it also recalibrates detection scores. This detection potential is similar to the one previously proposed by [ 28], [48], [52] and
[55], but formulated for the differentiable mean ﬁeld inference algorithm. We employ this potential as we are already
using object detection information for identifying object instances in the next stage. We denote the output at the semantic segmentation module of our network as the tensor
Q, whereQi(l)denotes the probability (obtained by applying the softmax function on the network’s activations) of
pixelitaking on the label l∈ L.


key: chroma:document
string_value:
pixelitaking on the label l∈ L.
(a) Semantic Segmentation (b) Instance Segmentation
Figure 3: Instance segmentation using only the “Box” unary
potential. This potential is effective when we have a good
initial semantic segmentation (a). Occlusions between objects of the same class can be resolved by the pairwise term
based on appearance differences. Note that we can ignore
the conﬁdent, false-positive “bottle” detections (b). This is
in contrast to methods such as [ 8,19,20,30] which cannot
recover from detection errors.
3.2. Instance Segmentation subnetwork
At the input to our instance segmentation subnetwork,
we assume that we have two inputs available: The semantic
segmentation predictions, Q, for each pixel and label, and a
set of object detections. For each input image, we assume
that there are Dobject detections, and that the ithdetection
is of the form (li,si,Bi)whereli∈ Lis the detected class
label,si∈[0,1]is the conﬁdence score and Biis the set of
indices of the pixels falling within the detector’s bounding


key: chroma:document
string_value:
label,si∈[0,1]is the conﬁdence score and Biis the set of
indices of the pixels falling within the detector’s bounding
box. Note that the number Dvaries for every input image.
The problem of instance segmentation can then be
thought of as assigning every pixel to either a particular object detection, or the background label. This is based on
the assumption that every object detection speciﬁes a potential object instance. We deﬁne a multinomial random
variable,V, at each of the Npixels in the image, and
V= [V1V2...V N]T. Each variable at pixel i,Vi, is assigned a label corresponding to its instance. This label set,
{0,1,2,...,D }changes for each image since D, the number
of detections, varies for every image (0 is the background
label). In the case of instance segmentation of images, the
quality of a prediction is invariant to the permutations of the
instance labelling. For example, labelling the “blue person”
in Fig. 1(c) as “1” and the “purple person” as “2” is no different to labelling them as “2” and “1” respectively. This


key: chroma:document
string_value:
condition is handled by our loss function in Sec. 3.4.
Note that unlike works such as [ 57] and [ 58] we do not
assume a maximum number of possible instances and keep
a ﬁxed label set. Furthermore, since we are considering object detection outputs jointly with semantic segmentation
predictions, we have some robustness to high-scoring false
positive detections unlike methods such as [ 8,20,37] which
reﬁne object detections into segmentations.
We formulate a Conditional Random Field over our instance variables, V, which consists of unary and pairwise
(a) Only Box term (b) Box and Global terms
Figure 4: The “Global” unary potential (b) is particularly
effective in cases where the input detection bounding box
does not cover the entire extent of the object. Methods
which are based on reﬁning bounding-box detections such
as [19,20,8,12] cannot cope with poorly localised detections. Note, the overlaid detection boxes are an additional
input to our system.
energies. The energy of the assignment vto all the variables,V, is
E(V=v) =/summationdisplay
iU(vi) +/summationdisplay
i<jP(vi,vj). (1)


key: chroma:document
string_value:
E(V=v) =/summationdisplay
iU(vi) +/summationdisplay
i<jP(vi,vj). (1)
The unary energy is a sum of three terms, which take into
account the object detection bounding boxes, the initial semantic segmentation and shape information,
U(vi) =−ln[w1ψBox(vi) +w2ψGlobal (vi)+
w3ψShape (vi)], (2)
and are described further in Sections 3.2.1 through 3.2.3 .
w1,w2andw3are all weighting co-efﬁcients learned via
backpropagation.
3.2.1 Box Term
This potential encourages a pixel to be assigned to the instance corresponding to the kthdetection if it falls within
the detection’s bounding box. This potential is proportional
to the probability of the pixel’s semantic class being equal
to the detected class Qi(lk)and the detection score, sk.
ψBox(Vi=k) =/braceleftBigg
Qi(lk)skifi∈Bk
0 otherwise(3)
As shown in Fig. 3, this potential performs well when the
initial semantic segmentation is good. It is robust to false


key: chroma:document
string_value:
0 otherwise(3)
As shown in Fig. 3, this potential performs well when the
initial semantic segmentation is good. It is robust to false
positive detections, unlike methods which reﬁne bounding
boxes [ 8,19,20] since the detections are considered in light
of our initial semantic segmentation, Q. Together with the
pairwise term (Sec. 3.2.4 ), occlusions between objects of
the same class can be resolved if there are appearance differences in the different instances.
3.2.2 Global Term
This term does not rely on bounding boxes, but only the segmentation prediction at a particular pixel, Qi. It encodes the
intuition that if we only know there are dpossible instances
of a particular object class, and have no further localisation information, each instance is equally probable, and this
potential is proportional to the semantic segmentation conﬁdence for the detected object class at that pixel:
ψGlobal (Vi=k) =Qi(lk). (4)
As shown in Fig. 4, this potential overcomes cases where
the bounding box does not cover the entire extent of the object, as it assigns probability mass to a particular instance


key: chroma:document
string_value:
As shown in Fig. 4, this potential overcomes cases where
the bounding box does not cover the entire extent of the object, as it assigns probability mass to a particular instance
label throughout all pixels in the image. This is also beneﬁcial during training, as it ensures that the ﬁnal output is
dependent on the segmentation prediction at all pixels in the
image, leading to error gradients that are more stable across
batches and thus more amenable to backpropagation.
3.2.3 Shape Term
We also incorporate shape priors to help us reason about occlusions involving multiple objects of the same class, which
may have minimal appearance variation between them, as
shown in Fig. 5. In such cases, a prior on the expected shape
of an object category can help us to identify the foreground
instance within a bounding box. Previous approaches to incorporating shape priors in segmentation [ 23,8,50] have
involved generating “shape exemplars” from the training
dataset and, at inference time, matching these exemplars to
object proposals using the Chamfer distance [ 46,36].
We propose a fully differentiable method: Given a set of
shape templates, T, we warp each shape template using bilinear interpolation into ˜Tso that it matches the dimensions


key: chroma:document
string_value:
We propose a fully differentiable method: Given a set of
shape templates, T, we warp each shape template using bilinear interpolation into ˜Tso that it matches the dimensions
of thekthbounding box, Bk. We then select the shape prior
which matches the segmentation prediction for the detected
class within the bounding box, QBk(lk), the best according to the normalised cross correlation. Our shape prior is
then the Hadamard (elementwise) product ( ⊙) between the
segmentation unaries and the matched shape prior:
t∗= arg max
t∈˜T/summationtextQBk(lk)⊙t/vextenddouble/vextenddoubleQBk(lk)/vextenddouble/vextenddouble/bardblt/bardbl(5)
ψ(VBk=k) =QBk(lk)⊙t∗. (6)
Equations 5and6can be seen as a special case of maxpooling, and the numerator of Eq. 5is simply a convolution
that produces a scalar output since the two arguments are of
equal dimension. Additionally, during training, we can consider the shape priors Tas parameters of our “shape term”
layer and backpropagate through to the matched exemplar


key: chroma:document
string_value:
equal dimension. Additionally, during training, we can consider the shape priors Tas parameters of our “shape term”
layer and backpropagate through to the matched exemplar
t∗to update it. In practice, we initialised these parameters
(a) Without shape term (b) With Shape term
Figure 5: The “Shape” unary potential (b) helps us to distinguish between the green and purple sheep, which the other
two unary potentials cannot. Input detections are overlaid
on the images.
with the shape priors described in [ 50]. This consists of
roughly 250 shape templates for each of ﬁve different aspect ratios. These were obtained by clustering foreground
masks of object instances from the training set.
Here, we have only matched a single shape template to
a proposed instance. This method could be extended in future to matching multiple templates to an instance, in which
case each shape exemplar would correspond to a part of the
object such as in DPM [ 15].
3.2.4 Pairwise term
The pairwise term consists of densely-connected Gaussian
potentials [ 26] and encourages appearance and spatial consistency. The weights governing the importance of these
terms are also learnt via backpropagation, as in [ 60]. We


key: chroma:document
string_value:
potentials [ 26] and encourages appearance and spatial consistency. The weights governing the importance of these
terms are also learnt via backpropagation, as in [ 60]. We
ﬁnd that these priors are useful in the case of instance segmentation as well, since nearby pixels that have similar appearance often belong to the same object instance. They are
often able to resolve occlusions based on appearance differences between objects of the same class (Fig. 3).
3.3. Inference of our Dynamic Instance CRF
We use mean ﬁeld inference to approximately minimise
the Gibbs Energy in Eq. 1which corresponds to ﬁnding the
Maximum a Posteriori (MAP) labelling of the corresponding probability distribution, P(V=v) =1
Zexp (−E(v))
whereZis the normalisation factor. Mean ﬁeld inference is
differentiable, and this iterative algorithm can be unrolled
and seen as a recurrent neural network [ 60]. Following this
approach, we can incorporate mean ﬁeld inference of a CRF
as a layer of our neural network. This enables us to train our
entire instance segmentation network end-to-end.
Because we deal with a variable number of instances for
every image, our CRF needs to be dynamically instantiated


key: chroma:document
string_value:
entire instance segmentation network end-to-end.
Because we deal with a variable number of instances for
every image, our CRF needs to be dynamically instantiated
to have a different number of labels for every image, as observed in [ 3]. Therefore, unlike [ 60], none of our weights
are class-speciﬁc. This weight-sharing not only allows us to
deal with variable length inputs, but class-speciﬁc weights
(a) Original
ground truth, G(b) Prediction, P (c) “Matched”
ground truth, G∗
Figure 6: Due to the problem of label permutations, we
“match” the ground truth with our prediction before computing the loss when training.
also do not make sense in the case of instance segmentation
since a class label has no particular semantic meaning.
3.4. Loss Function
When training for instance segmentation, we have a
single loss function which we backpropagate through our
instance- and semantic-segmentation modules to update all
the parameters. As discussed previously, we need to deal
with different permutations of our ﬁnal labelling which
could have the same ﬁnal result. The works of [ 57] and [ 58]
order instances by depth to break this symmetry. However,


key: chroma:document
string_value:
could have the same ﬁnal result. The works of [ 57] and [ 58]
order instances by depth to break this symmetry. However,
this requires ground-truth depth maps during training which
we do not assume that we have. Proposal-based methods
[12,19,20,37] do not have this issue since they consider a
single proposal at a time, rather than the entire image. Our
approach is similar to [ 45] in that we match the original
ground truth to our instance segmentation prediction based
on the Intersection over Union (IoU) [ 14] of each instance
prediction and ground truth, as shown in Fig. 6.
More formally, we denote the ground-truth labelling of
an image, G, to be a set of rsegments, {g1,g2,...,g r},
where each segment (set of pixels) is an object instance
and has an associated semantic class label. Our prediction, which is the output of our network, P, is a set of s
segments, {p1,p2,...,p s}, also where each segment corresponds to an instance label and also has an associated class
label. Note that randsmay be different since we may predict greater or fewer instances than actually present. Let
Mdenote the set of all permutations of the ground-truth,


key: chroma:document
string_value:
label. Note that randsmay be different since we may predict greater or fewer instances than actually present. Let
Mdenote the set of all permutations of the ground-truth,
G. As can be seen in Fig. 6, different permutations of the
ground-truth correspond to the same qualitative result. We
deﬁne the “matched” ground-truth, G∗, as the permutation
of the original ground-truth labelling which maximises the
IoU between the prediction, P, and ground truth:
G∗= arg max
m∈MIoU(m,P). (7)
Once we have the “matched” ground truth, G∗, (Fig. 6) foran image, we can apply any loss function to train our network for segmentation. In our case, we use the common
cross-entropy loss function. We found that this performed
better than the approximate IoU loss proposed in [ 27,45].
Crucially, we do not need to evaluate all permutations
of the ground truth to compute Eq. 7, since it can be formulated as a maximum-weight bipartite matching problem.
The edges in our bipartite graph connect ground-truth and
predicted segments. The edge weights are given by the
IoU between the ground truth and predicted segments if


key: chroma:document
string_value:
The edges in our bipartite graph connect ground-truth and
predicted segments. The edge weights are given by the
IoU between the ground truth and predicted segments if
they share the same semantic class label, and zero otherwise. Leftover segments are matched to “dummy” nodes
with zero overlap.
Additionally, the ordering of the instances in our network
are actually determined by the object detector, which remains static during training. As a result, the ordering of
our predictions does not ﬂuctuate much during training – it
only changes in cases where there are multiple detections
overlapping an object.
3.5. Network Training
We ﬁrst train a network for semantic segmentation with
the standard cross-entropy loss. In our case, this network is
FCN8s [ 38] with a CRF whose inference is unrolled as an
RNN and trained end-to-end, as described in [ 60] and [ 2].
To this pretrained network, we append our instance segmentation subnetwork, and ﬁnetune with instance segmentation
annotations and only the loss detailed in Sec. 3.4. For the
semantic segmentation subnetwork, we train with an initial
learning rate of 10−8, momentum of 0.9 and batch size of
20. The learning rate is low since we do not normalise the


key: chroma:document
string_value:
learning rate of 10−8, momentum of 0.9 and batch size of
20. The learning rate is low since we do not normalise the
loss by the number of pixels. This is so that images with
more pixels contribute a higher loss. The normalised learning rate is approximately 2×10−3. When training our instance segmentation network as well, we lower the learning
rate to 10−12and use a batch size of 1 instead. Decreasing the batch size gave empirically better results. We also
clipped gradients (a technique common in training RNNs
[40]) withℓ2norms above 109. This threshold was set by
observing “normal” gradient magnitudes during training.
The relatively high magnitude is due to the fact that our
loss is not normalised. In our complete network, we have
two CRF inference modules which are RNNs (one each in
the semantic- and instance-segmentation subnetworks), and
gradient clipping facilitated successful training.
3.6. Discussion
Our network is able to compute a semantic and instance
segmentation of the input image in a single forward pass.
We do not require any post-processing, such as the patch
aggregation of [ 37], “mask-voting” of [ 12], “superpixel


key: chroma:document
string_value:
We do not require any post-processing, such as the patch
aggregation of [ 37], “mask-voting” of [ 12], “superpixel
projection” of [ 19,20,30] or spectral clustering of [ 33].
The fact that we compute an initial semantic segmentation
means that we have some robustness to errors in the object detector (Fig. 3). Furthermore, we are not necessarily
limited by poorly localised object detections either (Fig. 4).
Our CRF model allows us to reason about the entire image
at a time, rather than consider independent object proposals,
as done in [ 19,20,12,37,30]. Although we do not train our
object detector jointly with the network, it also means that
our segmentation network and object detector do not succumb to the same failure cases. Moreover, it ensures that
our instance labelling does not “switch” often during training, which makes learning more stable. Finally, note that
although we perform mean ﬁeld inference of a CRF within
our network, we do not optimise the CRF’s likelihood, but
rather a cross-entropy loss (Sec 3.4).
4. Experimental Evaluation
Sections 4.1to4.6describe our evaluation on the Pascal VOC Validation Set [ 14] and the Semantic Boundaries


key: chroma:document
string_value:
4. Experimental Evaluation
Sections 4.1to4.6describe our evaluation on the Pascal VOC Validation Set [ 14] and the Semantic Boundaries
Dataset (SBD) [ 18] (which provides per-pixel annotations
to 11355 previously unlaballed images from Pascal VOC).
Section 4.7details results on Cityscapes [ 9].
4.1. Experimental Details
We ﬁrst train a network for semantic segmentation, therafter we ﬁnetune it to the task of instance segmentation,
as described in Sec. 3.5. Our training data for the semantic segmentation pretraining consists of images from Pascal VOC [ 14], SBD [ 18] and Microsoft COCO [ 35]. Finally, when ﬁnetuning for instance segmentation, we use
only training data from either the VOC dataset, or from the
SBD dataset. We train separate models for evaluating on
the VOC Validation Set, and the SBD Validation Set. In
each case, we remove validation set images from the initial
semantic segmentation pretraining set. We use the publicly
available R-FCN object detection framework [ 13], and ensure that the images used to train the detector do not fall
into our test sets for instance segmentation.
4.2. Evaluation Metrics
We report the mean Average Precision over regions


key: chroma:document
string_value:
into our test sets for instance segmentation.
4.2. Evaluation Metrics
We report the mean Average Precision over regions
(APr) as deﬁned by [ 19]. The difference between APrand
the AP metric used in object detection [ 14] is that the Intersection over Union (IoU) is computed over predicted and
ground-truth regions instead of bounding boxes. Furthermore, the standard AP metric uses an IoU threshold of 0.5
to determine whether a prediction is correct or not. Here,
we use a variety of IoU thresholds since larger thresholds
require more precise segmentations. Additionally, we report theAPr
volwhich is the average of the APrfor 9 IoU
thresholds ranging from 0.1 to 0.9 in increments of 0.1.
However, we also observe that the APrmetric requires
an algorithm to produce a ranked list of segments and their
object class. It does not require, nor evaluate, the ability ofTable 1: The effect of the different CRF unary potentials,
and end-to-end training with them, on the VOC 2012 Validation Set.
APr
APr
volmatch
IoU 0.5 0.7 0.9
Box Term
(piecewise)60.0 47.3 21.2 54.9 42.6
Box+Global


key: chroma:document
string_value:
IoU 0.5 0.7 0.9
Box Term
(piecewise)60.0 47.3 21.2 54.9 42.6
Box+Global
(piecewise)59.1 46.1 23.4 54.6 43.0
Box+Global+Shape
(piecewise)59.5 46.4 23.3 55.2 44.8
Box Term
(end-to-end)60.7 47.4 24.6 56.2 46.9
Box+Global
(end-to-end)60.9 48.1 25.5 56.7 47.1
Box+Global+Shape
(end-to-end)61.7 48.6 25.1 57.5 48.3
an algorithm to produce a globally coherent segmentation
map of the image, for example Fig. 1c. To measure this, we
propose the “Matching IoU” which matches the predicted
image and ground truth, and then calculates the corresponding IoU as deﬁned in [ 14]. This matching procedure is the
same as described in Sec. 3.4. This measure was originally
proposed in [ 54], but has not been used since in evaluating
instance segmentation systems.
4.3. Effect of Instance Potentials and End-to-End
training


key: chroma:document
string_value:
proposed in [ 54], but has not been used since in evaluating
instance segmentation systems.
4.3. Effect of Instance Potentials and End-to-End
training
We ﬁrst perform ablation studies on the VOC 2012 Validation set. This dataset, consisting of 1464 training and
1449 validation images has very high-quality annotations
with detailed object delineations which makes it the most
suited for evaluating pixel-level segmentations.
In Tab. 1, we examine the effect of each of our unary potentials in our Instance subnetwork on overall performance.
Furthermore, we examine the effect of end-to-end training
the entire network as opposed to piecewise training. Piecewise training refers to freezing the pretrained semantic segmentation subnetwork’s weights and only optimising the instance segmentation subnetwork’s parameters. Note that
when training with only the “Box” (Eq. 3) unary potential and pairwise term, we also have to add in an additional
“Background” detection which encompasses the entire image. Otherwise, we cannot classify the background label.
We can see that each unary potential improves overall instance segmentation results, both in terms of APr
voland the
Matching IoU. The “Global” term (Eq. 4) shows particular


key: chroma:document
string_value:
voland the
Matching IoU. The “Global” term (Eq. 4) shows particular
improvement over the “Box” term at the high APrthreshold of 0.9. This is because it can overcome errors in bounding box localisation (Fig. 4) and leverage our semantic segmentation network’s accurate predictions to produce precise
Table 2: Comparison of Instance Segmentation performance to recent methods on the VOC 2012 Validation Set
MethodAPr
APr
vol0.5 0.6 0.7 0.8 0.9
SDS [ 19] 43.8 34.5 21.3 8.7 0.9 –
Chen et al. [8] 46.3 38.2 27.0 13.5 2.6 –
PFN [ 33] 58.7 51.3 42.5 31.2 15.7 52.3
Arnab et al. [3] 58.3 52.4 45.4 34.9 20.1 53.1
MPA 1-scale [ 37] 60.3 54.6 45.9 34.3 17.3 54.5
MPA 3-scale [ 37]62.1 56.6 47.4 36.1 18.5 56.5


key: chroma:document
string_value:
MPA 3-scale [ 37]62.1 56.6 47.4 36.1 18.5 56.5
Ours 61.7 55.5 48.6 39.5 25.1 57.5
labellings. The “Shape” term’s improvement in the APr
volis
primarily due to an improvement in the APrat low thresholds. By using shape priors, we are able to recover instances
which were occluded and missed out. End-to-end training
also improves results at all APrthresholds. Training with
just the “Box” term shows a modest improvement in the
APr
volof 1.3%. Training with the “Global” and “Shape”
terms shows larger improvements of 2.1% and 2.3% respectively. This may be because the “Box” term only considers
the semantic segmentation at parts of the image covered by
object detections. Once we include the “Global” term, we
consider the semantic segmentation over the entire image
for the detected class. Training makes more efﬁcient use of
images, and error gradients are more stable in this case.
4.4. Results on VOC Validation Set
We then compare our best instance segmentation model


key: chroma:document
string_value:
images, and error gradients are more stable in this case.
4.4. Results on VOC Validation Set
We then compare our best instance segmentation model
to recent methods on the VOC Validation Set in Tab. 2. The
fact that our algorithm achieves the highest APrat thresholds above 0.7 indicates that our method produces more detailed and accurate segmentations.
At an IoU threshold of 0.9, our improvement over the
previous state-of-the-art (MPA [ 37]) is 6.6%, which is a
relative improvement of 36%. Unlike [ 37,19,8], our
network performs an initial semantic segmentation which
may explain our more accurate segmentations. Other
segmentation-based approaches, [ 3,33] are not fully endto-end trained. We also achieve the best APr
volof 57.5%.
The relatively small difference in APr
volto MPA [ 37] despite large improvements at high IoU thresholds indicates
that MPA performs better at low IoU thresholds. Proposalbased methods, such as [ 37,19] are more likely to perform
better at low IoU thresholds since they output more proposals than actual instances in an image (SDS evaluates 2000
proposals per image). Furthermore, note that whilst MPA


key: chroma:document
string_value:
better at low IoU thresholds since they output more proposals than actual instances in an image (SDS evaluates 2000
proposals per image). Furthermore, note that whilst MPA
takes 8.7s to process an image [ 37], our method requires approximately 1.5s on the same Titan X GPU. More detailed
qualitative and quantitative results, including success and
failure cases, are included in the supplementary material.Table 3: Comparison of Instance Segmentation performance on the SBD Dataset
MethodAPr
APr
volmatch
0.5 0.7 IoU
SDS [ 19] 49.7 25.3 41.4 –
MPA 1-scale [ 37] 55.5 – 48.3 –
Hypercolumn [ 20] 56.5 37.0 – –
IIS [30] 60.1 38.7 – –
CFM [ 11] 60.7 39.6 – –
Hypercolumn rescore [ 20] 60.0 40.4 – –
MPA 3-scale rescore [ 37] 61.8 – 52.0 –
MNC [ 12] 63.5 41.5 – 39.0
MNC, Instance FCN [ 10] 61.5 43.0 – –
IIS sp. projection, rescore [ 30]63.6 43.3 – –


key: chroma:document
string_value:
MNC, Instance FCN [ 10] 61.5 43.0 – –
IIS sp. projection, rescore [ 30]63.6 43.3 – –
Ours (piecewise) 59.1 42.1 52.3 41.8
Ours (end-to-end) 62.0 44.8 55.4 47.3
4.5. Results on SBD Dataset
We also evaluate our model on the SBD dataset, which
consists of 5623 training and 5732 validation images, as
shown in Tab. 3. Following other works, we only report
APrresults at IoU thresholds of 0.5 and 0.7. However, we
provide more detailed results in our supplementary material. Once again, we show signiﬁcant improvements over
other work at high APrthresholds. Here, our APrat 0.7
improves by 1.5% over the previous state-of-the-art [ 30].
Note that [ 30,37,20] perform additional post-processing
where their results are rescored using an additional object
detector. In contrast, our results are obtained by a single
forward pass through our network. We have also improved
substantially on the APr
volmeasure (3.4%) compared to


key: chroma:document
string_value:
forward pass through our network. We have also improved
substantially on the APr
volmeasure (3.4%) compared to
other works which have reported it. We also used the publicly available source code1, model and default parameters
of MNC [ 12] to evaluate the “Matching IoU”. Our method
improves this by 8.3%. This metric is a stricter measure of
segmentation performance, and our method, which is based
on an initial semantic segmentation and includes a CRF as
part of training therefore performs better.
4.6. Improvement in Semantic Segmentation
Finetuning our network for instance segmentation, with
the loss described in Sec. 3.4improves semantic segmentation performance on both the VOC and SBD dataset, as
shown in Tab. 4. The improvement is 0.9% on VOC, and
1% on SBD. The tasks of instance segmentation and semantic segmentation are highly related – in fact, instance
segmentation can be thought of as a more speciﬁc case of
semantic segmentation. As a result, ﬁnetuning for one task
improves the other.
1https://github.com/daijifeng001/MNC
Table 4: Semantic Segmentation performance before and


key: chroma:document
string_value:
improves the other.
1https://github.com/daijifeng001/MNC
Table 4: Semantic Segmentation performance before and
after ﬁnetuning for Instance Segmentation on the VOC and
SBD Validation Sets
DatasetMean IoU [%] before
Instance ﬁnetuningMean IoU [%] after
Instance ﬁnetuning
VOC 74.2 75.1
SBD 71.5 72.5
4.7. Results on Cityscapes
Finally, we evaluate our algorithm on the Cityscapes
road-scene understanding dataset [ 9]. This dataset consists
of 2975 training images, and the held-out test set consisting
of 1525 images are evaluated on an online server. None of
the 500 validation images were used for training. We use
an initial semantic segmentation subnetwork that is based
on the ResNet-101 architecture [ 59], and all of the instance
unary potentials described in Sec. 3.2.
As shown in Tab. 5, our method sets a new state-of-theart on Cityscapes, surpassing concurrent work [ 21] and the
best previous published work [ 49] by signiﬁcant margins.
Table 5: Results on Cityscapes Test Set. Evaluation metrics and results of competing methods obtained from the


key: chroma:document
string_value:
best previous published work [ 49] by signiﬁcant margins.
Table 5: Results on Cityscapes Test Set. Evaluation metrics and results of competing methods obtained from the
online server. The “AP” metric of Cityscapes is similar to
ourAPr
volmetric.
Method AP AP at 0.5 AP 100m AP 50m
Ours 20.0 38.8 32.6 37.6
SAIS [ 21] 17.4 36.7 29.3 34.0
DWT [ 4] 15.6 30.0 26.2 31.8
InstanceCut [ 24] 13.0 27.9 22.1 26.1
Graph Decomp. [ 29] 9.8 23.2 16.8 20.3
RecAttend [ 43] 9.5 18.9 16.8 20.9
Pixel Encoding [ 49] 8.9 21.1 15.3 16.7
R-CNN [ 9] 4.6 12.9 7.7 10.3
5. Conclusion and Future Work
We have presented an end-to-end instance segmentation
approach that produces intermediate semantic segmentations, and shown that ﬁnetuning for instance segmentation


key: chroma:document
string_value:
We have presented an end-to-end instance segmentation
approach that produces intermediate semantic segmentations, and shown that ﬁnetuning for instance segmentation
improves our network’s semantic segmentations. Our approach differs from other methods which derive their architectures from object detection networks [ 12,37,20] in that
our approach is more similar to a semantic segmentation
network. As a result, our system produces more accurate
and detailed segmentations as shown by our substantial improvements at high APrthresholds. Moreover, our system
produces segmentation maps naturally, and in contrast to
other published work, does not require any post-processing.
Finally, our network produces a variable number of outputs,depending on the number of instances in the image. Our future work is to incorporate an object detector into the endto-end training of our system to create a network that performs semantic segmentation, object detection and instance
segmentation jointly. Possible techniques for doing this are
suggested by [ 25] and [ 39].
Acknowledgements We thank Bernardino RomeraParedes and Stuart Golodetz for insightful discussions
and feedback. This work was supported by the EPSRC,
Clarendon Fund, ERC grant ERC-2012-AdG 321162HELIOS, EPRSRC grant Seebibyte EP/M013774/1 and


key: chroma:document
string_value:
EPSRC/MURI grant EP/N019474/1.
References
[1] P. Arbelaez, J. Pont-Tuset, J. Barron, F. Marques, and J. Malik. Multiscale combinatorial grouping. In CVPR , pages
328–335. IEEE, 2014. 2
[2] A. Arnab, S. Jayasumana, S. Zheng, and P. H. S. Torr. Higher
order conditional random ﬁelds in deep neural networks. In
ECCV , 2016. 2,3,6
[3] A. Arnab and P. H. S. Torr. Bottom-up instance segmentation
with deep higher order crfs. In BMVC , 2016. 3,5,8,20
[4] M. Bai and R. Urtasun. Deep watershed transform for instance segmentation. In arXiv preprint arXiv:1611.08303 ,
2016. 9
[5] L. Chen, A. Schwing, A. Yuille, and R. Urtasun. Learning
deep structured models. In ICML , Lille, France, 2015. 2
[6] L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and


key: chroma:document
string_value:
[6] L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and
A. L. Yuille. Semantic image segmentation with deep convolutional nets and fully connected crfs. ICLR , 2015. 2
[7] L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and
A. L. Yuille. Deeplab: Semantic image segmentation with
deep convolutional nets, atrous convolution, and fully connected crfs. arXiv preprint arXiv:1606.00915 , 2016. 1
[8] Y.-T. Chen, X. Liu, and M.-H. Yang. Multi-instance object segmentation with occlusion handling. In CVPR , pages
3470–3478, 2015. 2,4,5,8,20
[9] M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler,
R. Benenson, U. Franke, S. Roth, and B. Schiele. The
cityscapes dataset for semantic urban scene understanding.
InCVPR , 2016. 1,7,9


key: chroma:document
string_value:
cityscapes dataset for semantic urban scene understanding.
InCVPR , 2016. 1,7,9
[10] J. Dai, K. He, Y. Li, S. Ren, and J. Sun. Instance-sensitive
fully convolutional networks. In ECCV , 2016. 1,8
[11] J. Dai, K. He, and J. Sun. Convolutional feature masking for
joint object and stuff segmentation. In CVPR , 2015. 2,8
[12] J. Dai, K. He, and J. Sun. Instance-aware semantic segmentation via multi-task network cascades. In CVPR , 2016. 1,2,
3,4,6,7,8,9,11,14,15
[13] J. Dai, Y. Li, K. He, and J. Sun. R-fcn: Object detection via
region-based fully convolutional networks. In NIPS , 2016.
1,7
[14] M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and
A. Zisserman. The pascal visual object classes (voc) challenge. IJCV , 2010. 1,6,7


key: chroma:document
string_value:
A. Zisserman. The pascal visual object classes (voc) challenge. IJCV , 2010. 1,6,7
[15] P. F. Felzenszwalb, R. B. Girshick, D. McAllester, and D. Ramanan. Object detection with discriminatively trained partbased models. PAMI , 32(9):1627–1645, 2010. 2,5
[16] R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature hierarchies for accurate object detection and semantic
segmentation. In CVPR , 2014. 2
[17] S. Gupta, R. Girshick, P. Arbelaez, and J. Malik. Learning
rich features from RGB-D images for object detection and
segmentation. In ECCV . 2014. 1
[18] B. Hariharan, P. Arbel ´aez, L. Bourdev, S. Maji, and J. Malik.
Semantic contours from inverse detectors. In ICCV , pages
991–998. IEEE, 2011. 7
[19] B. Hariharan, P. Arbel ´aez, R. Girshick, and J. Malik. Simultaneous detection and segmentation. In ECCV , pages 297–


key: chroma:document
string_value:
312. Springer, 2014. 1,2,3,4,6,7,8,20
[20] B. Hariharan, P. Arbel ´aez, R. Girshick, and J. Malik. Hypercolumns for object segmentation and ﬁne-grained localization. CVPR , 2015. 1,2,3,4,6,7,8,9
[21] Z. Hayder, X. He, and M. Salzmann. Shape-aware instance
segmentation. In arXiv preprint arXiv:1612.03129 , 2016. 9
[22] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning
for image recognition. In CVPR , 2016. 1
[23] X. He and S. Gould. An Exemplar-based CRF for Multiinstance Object Segmentation. In CVPR , 2014. 5
[24] A. Kirillov, E. Levinkov, B. Andres, B. Savchynskyy, and
C. Rother. Instancecut: from edges to instances with multicut. In arXiv preprint arXiv:1611.08272 , 2016. 9
[25] I. Kokkinos. Ubernet: Training a universal convolutional


key: chroma:document
string_value:
[25] I. Kokkinos. Ubernet: Training a universal convolutional
neural network for low-, mid-, and high-level vision using
diverse datasets and limited memory. In CVPR , 2017. 9
[26] P. Kr ¨ahenb ¨uhl and V. Koltun. Efﬁcient inference in fully
connected CRFs with Gaussian edge potentials. In NIPS ,
2011. 3,5
[27] P. Kr ¨ahenb ¨uhl and V. Koltun. Parameter learning and convergent inference for dense random ﬁelds. In ICML , 2013.
6
[28] L. Ladick ´y, P. Sturgess, K. Alahari, C. Russell, and P. H. S.
Torr. What, where and how many? combining object detectors and crfs. In ECCV , pages 424–437, 2010. 3
[29] E. Levinkov, S. Tang, E. Insafutdinov, and B. Andres. Joint
graph decomposition and node labeling by local search.
arXiv preprint arXiv:1611.04399 , 2016. 9


key: chroma:document
string_value:
graph decomposition and node labeling by local search.
arXiv preprint arXiv:1611.04399 , 2016. 9
[30] K. Li, B. Hariharan, and J. Malik. Iterative Instance Segmentation. In CVPR , 2016. 1,2,4,6,7,8,21
[31] Y. Li, H. Qi, J. Dai, X. Ji, and Y. Wei. Fully convolutional
instance-aware semantic segmentation. In CVPR , 2017. 11,
16
[32] X. Liang, Y. Wei, X. Shen, Z. Jie, J. Feng, L. Lin, and S. Yan.
Reversible recursive instance-level object segmentation. In
CVPR , 2016. 1,2
[33] X. Liang, Y. Wei, X. Shen, J. Yang, L. Lin, and S. Yan.
Proposal-free network for instance-level object segmentation. arXiv preprint arXiv:1509.02636 , 2015. 2,6,8,18,
20
[34] G. Lin, C. Shen, and I. Reid. Efﬁcient piecewise training of
deep structured models for semantic segmentation. In CVPR ,


key: chroma:document
string_value:
[34] G. Lin, C. Shen, and I. Reid. Efﬁcient piecewise training of
deep structured models for semantic segmentation. In CVPR ,
2016. 2[35] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Doll ´ar, and C. L. Zitnick. Microsoft coco: Common objects in context. In ECCV , pages 740–755. Springer,
2014. 7,16
[36] M.-Y. Liu, O. Tuzel, A. Veeraraghavan, and R. Chellappa.
Fast directional chamfer matching. In CVPR , pages 1696–
1703. IEEE, 2010. 5
[37] S. Liu, X. Qi, J. Shi, H. Zhang, and J. Jia. Multi-scale patch
aggregation (mpa) for simultaneous detection and segmentation. In CVPR , 2016. 1,2,3,4,6,7,8,9,18,20
[38] J. Long, E. Shelhamer, and T. Darrell. Fully convolutional
networks for semantic segmentation. In CVPR , 2015. 3,6


key: chroma:document
string_value:
[38] J. Long, E. Shelhamer, and T. Darrell. Fully convolutional
networks for semantic segmentation. In CVPR , 2015. 3,6
[39] I. Misra, A. Shrivastava, A. Gupta, and M. Hebert. Crossstitch Networks for Multi-task Learning. In CVPR , 2016. 9
[40] R. Pascanu, T. Mikolov, and Y. Bengio. On the difﬁculty of
training recurrent neural networks. In ICML , pages 1310–
1318, 2013. 6
[41] P. O. Pinheiro, R. Collobert, and P. Dollar. Learning to segment object candidates. In NIPS , 2015. 1
[42] P. O. Pinheiro, T.-Y. Lin, R. Collobert, and P. Dollar. Learning to reﬁne object segments. In ECCV , 2016. 1
[43] M. Ren and R. S. Zemel. End-to-end instance segmentation and counting with recurrent attention. In arXiv preprint
arXiv:1605.09410 , 2016. 9


key: chroma:document
string_value:
arXiv:1605.09410 , 2016. 9
[44] S. Ren, K. He, R. Girshick, and J. Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. In NIPS , 2015. 2
[45] B. Romera-Paredes and P. H. Torr. Recurrent instance segmentation. In ECCV , 2016. 2,6
[46] J. Shotton, A. Blake, and R. Cipolla. Contour-based learning
for object detection. In ICCV , volume 1, pages 503–510.
IEEE, 2005. 5
[47] K. Simonyan and A. Zisserman. Very deep convolutional
networks for large-scale image recognition. In ICLR , 2015.
3
[48] M. Sun, B.-s. Kim, P. Kohli, and S. Savarese. Relating things
and stuff via object property interactions. PAMI , 36(7):1370–
1383, 2014. 3
[49] J. Uhrig, M. Cordts, U. Franke, and T. Brox. Pixel-level encoding and depth layering for instance-level semantic labeling. In German Conference on Pattern Recognition (GCPR) ,
2016. 9


key: chroma:document
string_value:
2016. 9
[50] D. Weiss and B. Taskar. Scalpel: Segmentation cascades
with localized priors and efﬁcient learning. In CVPR , pages
2035–2042, 2013. 5
[51] J. Winn and J. Shotton. The layout consistent random ﬁeld
for recognizing and segmenting partially occluded objects.
InCVPR , 2006. 2
[52] C. Wojek and B. Schiele. A dynamic conditional random
ﬁeld model for joint labeling of object and scene classes. In
ECCV , pages 733–747. Springer, 2008. 3
[53] N. Xu, B. Price, S. Cohen, J. Yang, and T. Huang. Deep
interactive object selection. In CVPR , 2016. 1
[54] Y. Yang, S. Hallman, D. Ramanan, and C. C. Fowlkes. Layered object models for image segmentation. PAMI , 2012. 2,
7
[55] J. Yao, S. Fidler, and R. Urtasun. Describing the Scene as
a Whole: Joint Object Detection, Scene Classiﬁcation and
Semantic Segmentation. In CVPR , pages 702–709, 2012. 3


key: chroma:document
string_value:
a Whole: Joint Object Detection, Scene Classiﬁcation and
Semantic Segmentation. In CVPR , pages 702–709, 2012. 3
[56] S. Zagoruyko, A. Lerer, T.-Y. Lin, P. O. Pinheiro, S. Gross,
S. Chintala, and P. Doll ´ar. A multipath network for object
detection. In BMVC , 2016. 1
[57] Z. Zhang, S. Fidler, and R. Urtasun. Instance-level segmentation for autonomous driving with deep densely connected
mrfs. In CVPR , 2016. 2,3,4,6
[58] Z. Zhang, A. G. Schwing, S. Fidler, and R. Urtasun. Monocular object instance segmentation and depth ordering with
cnns. In ICCV , pages 2614–2622, 2015. 2,3,4,6
[59] H. Zhao, J. Shi, X. Qi, X. Wang, and J. Jia. Pyramid scene
parsing network. In arXiv preprint arXiv:1612.01105 , 2017.
9
[60] S. Zheng, S. Jayasumana, B. Romera-Paredes, V. Vineet,


key: chroma:document
string_value:
9
[60] S. Zheng, S. Jayasumana, B. Romera-Paredes, V. Vineet,
Z. Su, D. Du, C. Huang, and P. Torr. Conditional random
ﬁelds as recurrent neural networks. In ICCV , 2015. 2,3,5,6
Appendix
In this supplementary material, we include more detailed
qualitative and quantitative results on the VOC and SBD
datasets. Furthermore, we also show the runtime of our algorithm.
Figures 7and8show success and failure cases of our
algorithm. Figure 9compares the results of our algorithm
to the publicly available model for MNC [ 12]. Figure 10
compares our results to those of FCIS [ 31], concurrent work
which won the COCO 2016 challenge. Figure 11presents
some qualitative results on the Cityscapes dataset.
Section Ashows more detailed results on the VOC
dataset. Figure 12shows a visualisation of our results at
differentAPrthresholds, and Tables 7to9show per-class
APrresults at thresholds of 0.5, 0.7 and 0.9.
Section Bshows more detailed results on the SBD


key: chroma:document
string_value:
APrresults at thresholds of 0.5, 0.7 and 0.9.
Section Bshows more detailed results on the SBD
dataset. Table 6shows our mean APrresults at thresholds from 0.5 to 0.9, whilst Tables 10and11show per-class
APrresults at thresholds of 0.7 and 0.5 respectively.
Input image Semantic Segmentation Instance Segmentation Ground truth
Figure 7: Success cases of our method. First and second row: Our algorithm can leverage good initial semantic segmentations, and detections, to produce an instance segmentation. Third row: Notice that we have ignored three false-positive
detections. Additionally, the red bounding box does not completely encompass the person, but our algorithm is still able to
associate pixels “outside-the-box” with the correct detection (also applies to row 2). Fourth row: Our system is able to deal
with the heavily occluded sheep, and ignore the false-positive detection. Fifth row: We have not been able to identify one
bicycle on the left since it was not detected, but otherwise have performed well. Sixth row: Although subjective, the train has
not been annotated in the dataset, but both our initial semantic segmentation and object detection networks have identiﬁed it.


key: chroma:document
string_value:
not been annotated in the dataset, but both our initial semantic segmentation and object detection networks have identiﬁed it.
Note that the ﬁrst three images are from the VOC dataset, and the last three from SBD. Annotations in the VOC dataset are
more detailed, and also make more use of the grey “ignore” label to indicate uncertain areas in the image. The ﬁrst column
shows the input image, and the results of our object detector which are another input to our network. Best viewed in colour.
Input image Semantic Segmentation Instance Segmentation Ground truth
Figure 8: Failure cases of our method. First row: Both our initial detector, and semantic segmentation system did not
identify a car in the background. Additionally, the “brown” person prediction actually consists of two people that have been
merged together. This is because the detector did not ﬁnd the background person. Second row: Our initial semantic segmentation identiﬁed the table, but it is not there in the Instance Segmentation. This is because there was no “table detection” to
associate these pixels with. Using heuristics, we could propose additional detections in cases like these. However, we have


key: chroma:document
string_value:
associate these pixels with. Using heuristics, we could propose additional detections in cases like these. However, we have
not done this in our work. Third row: A difﬁcult case where we have segmented most of the people. However, sometimes
two people instances are joined together as one person instance. This problem is because we do not have a detection for
each person in the image. Fourth row: Due to our initial semantic segmentation, we have not been able to segment the green
person and table correctly. Fifth row: We have failed to segment a bird although it was detected. Sixth row: The occluding
cows, which all appear similar, pose a challenge, even with our shape priors. The ﬁrst column shows the input image, and
the results of our object detector which are another input to our network. Best viewed in colour.
Input image MNC [ 12] Ours Ground truth
Figure 9: Comparison to MNC [ 12]The above examples emphasise the advantages in our method over MNC [ 12]. Unlike
proposal-based approaches such as MNC, our method can handle false-positive detections, poor bounding box localisation,
reasons globally about the image and also produces more precise segmentations due to the initial semantic segmentation


key: chroma:document
string_value:
reasons globally about the image and also produces more precise segmentations due to the initial semantic segmentation
module which includes a differentiable CRF. Row 1 shows a case where MNC, which scores segment-based proposals, is
fooled by a false-positive detection and segments an imaginary human (yellow segment). Our method is robust to falsepositive detections due to the initial semantic segmentation module which does not have the same failure modes as the
detector. Rows 2, 3 and 4 show how MNC [ 12] cannot deal with poorly localised bounding boxes. The horizontal boundaries
of the red person in Row 2, and light-blue person in Row 4 correspond to the limits of the proposal processed by MNC. Our
method, in contrast, can segment “outside the detection bounding box” due to the global instance unary potential (Eq. 4). As
MNC does not reason globally about the image, it cannot handle cases of overlapping bounding boxes well, and produces
more instances than there actually are. The ﬁrst column shows the input image, and the results of our object detector which
are another input to our network. MNC does not use these detections, but does internally produce box-based proposals which
are not shown. Best viewed in colour.
Input image MNC [ 12] Ours Ground truth


key: chroma:document
string_value:
are not shown. Best viewed in colour.
Input image MNC [ 12] Ours Ground truth
Figure 9 continued: Comparison to MNC [ 12]The above examples show that our method produces more precise segmentations than MNC, that adhere to the boundaries of the objects. However, in Rows 3, 4 and 5, we see that MNC is able
to segment instances that our method misses out. In Row 3 , our algorithm does not segment the baby, although there is a
detection for it. This suggests that our shape prior which was formulated to overcome such occlusions could be better. As
MNC processes individual instances, it does not have a problem with dealing with small, occluding instances. In Row 4 ,
MNC has again identiﬁed a person that our algorithm could not. However, this is because we did not have a detection for
this person. In Row 5 , MNC has segmented the horses on the right better than our method. The ﬁrst column shows the input
image, and the results of our object detector which are another input to our network. MNC does not use these detections, but
does internally produce box-based proposals which are not shown. We used the publicly available code, models and default
parameters of MNC to produce this ﬁgure. Best viewed in colour.
Input image FCIS [ 31] Ours


key: chroma:document
string_value:
parameters of MNC to produce this ﬁgure. Best viewed in colour.
Input image FCIS [ 31] Ours
Figure 10: Comparison to FCIS [ 31]The above images compare our method to the concurrent work, FCIS [ 31], which was
trained on COCO [ 35] and won the COCO 2016 challenge. Unlike proposal-based methods such as FCIS, our method can
handle false-positive detections and poor bounding-box localisation. Furthermore, as our method reasons globally about the
image, one pixel can only be assigned to a single instance, which is not the case with FCIS. Our method also produces more
precise segmentations, as it includes a differentiable CRF, and it is based off a semantic segmentation network. The results of
FCIS are obtained from their publicly available results on the COCO test set ( https://github.com/daijifeng001/
TA-FCN ). Note that FCIS is trained on COCO, and our model is trained on Pascal VOC which does not have as many classes
as COCO, such as “umbrella” and “suitcase” among others. As a result, we are not able to detect these objects. The ﬁrst


key: chroma:document
string_value:
column shows the input image, and the results of our object detector which are another input to our network. FCIS does not
use these detections, but does internally produce proposals which are not shown. Best viewed in colour.
Input image Semantic Segmentation Instance Segmentation
Figure 11: Sample results on the Cityscapes dataset The above images show how our method can handle the large numbers
of instances present in the Cityscapes dataset. Unlike other recent approaches, our algorithm can deal with objects that are
not continuous – such as the car in the ﬁrst row which is occluded by a pole. Best viewed in colour.
A. Detailed results on the VOC dataset
Figure 12shows a visualisation of the AProbtained by
our method for each class across nine different thresholds.
Each “column” of Fig. 12corresponds to the APrfor each
class at a given IoU threshold. It is therefore an alternate
representation for the results tables (Tables 7to9). We can
see that our method struggles with classes such as “bicycle”, “chair”, “dining table” and “potted plant”. This may
be explained by the fact that current semantic segmentation


key: chroma:document
string_value:
be explained by the fact that current semantic segmentation
systems (including ours) struggle with these classes. All recent methods on the Pascal VOC leaderboard2obtain an
IoU for these classes which is lower than the mean IoU for
all classes. In fact the semantic segmentation IoU for the
“chair” class is less than half of the mean IoU for all the
classes for 16 out of the 20 most recent submissions on the
VOC leaderboard at the time of writing.
Tables 7to9show per-class instance segmentation results on the VOC dataset, at IoU thresholds of 0.9, 0.7 and
0.5 respectively. At an IoU threshold of 0.9, our method
achieves the highest APrfor 16 of the 20 object classes.
At the threshold of 0.7, we achieve the highest APrin 15
classes. Finally, at an IoU threshold of 0.5, our method,
MPA 3-scale [ 37] and PFN [ 33] each achieve the highest
APrfor 6 categories.
B. Detailed results on the SBD dataset
Once again, we show a visualisation of the AProbtained
by our method for each class across nine different thresholds (Fig. 13). The trend is quite similar to the VOC dataset
in that our algorithm struggles on the same object classes


key: chroma:document
string_value:
by our method for each class across nine different thresholds (Fig. 13). The trend is quite similar to the VOC dataset
in that our algorithm struggles on the same object classes
(“chair”, “dining table”, “potted plant”, “bottle”). Note that
ourAPrfor the “bicycle” class has improved compared to
the VOC dataset. This is probably because the VOC dataset
has more detailed annotations. In the VOC dataset, each
spoke of a bicycle’s wheel is often labelled, whilst in SBD,
the entire wheel is labelled as a single circle with the “bicycle” label. Therefore, the SBD dataset’s coarser labelling
makes it easier for an algorithm to perform well on objects
with ﬁne details.
Table 6shows our mean AProver all classes at thresholds ranging from 0.5 to 0.9. Our APrat 0.9 is low compared to the result which we obtained on the VOC dataset.
This could be for a number of reasons: As the SBD dataset
is not as ﬁnely annotated as the VOC dataset, it might not
be suited for measuring the APrat such high thresholds.
Additionally, the training data is not as good for training


key: chroma:document
string_value:
be suited for measuring the APrat such high thresholds.
Additionally, the training data is not as good for training
our system which includes a CRF and is therefore able to
delineate sharp boundaries. Finally, as the SBD dataset has
5732 validation images (compared to the 1449 in VOC), it
leaves less data for pretraining our initial semantic segmen2http://host.robots.ox.ac.uk:8080/leaderboard/
displaylb.php?challengeid=11&compid=6tation module. This may hinder our network in being able
to produce precise segmentations.
Table 6: Comparison of Instance Segmentation performance at multiple APrthesholds on the VOC 2012 Validation Set
MethodAPr
APr
vol0.5 0.6 0.7 0.8 0.9
Ours (piecewise) 59.1 51.9 42.1 29.4 12.0 52.3
Ours (end-to-end ) 62.0 54.0 44.8 32.3 13.8 55.4
Tables 10and11show per-class instance segmentation
results on the SBD dataset, at IoU thresholds of 0.7 and
0.5 respectively. We can only compare results at these two
thresholds since these are the only thresholds which other


key: chroma:document
string_value:
0.5 respectively. We can only compare results at these two
thresholds since these are the only thresholds which other
work has reported.
Figure 12: A visualisation of the AProbtained for each of the 20 classes on the VOC dataset, at nine different IoU thresholds.
The x-axis represents the IoU threshold, and the y-axis each of the Pascal classes. Therefore, each “column” of this ﬁgure
corresponds to the APrper class at a particular threshold, and is thus an alternate representation to the results tables. Best
viewed in colour.
Figure 13: A visualisation of the AProbtained for each of the 20 classes on the SBD dataset, at nine different IoU thresholds.
The x-axis represents the IoU threshold, and the y-axis each of the Pascal classes. Therefore, each “column” of this ﬁgure
corresponds to the APrper class at a particular threshold, and is thus an alternate representation to the results tables. Best
viewed in colour.
Table 7: Comparison of mean APr, achieved by different published methods, at an IoU threshold of 0.9, for all twenty classes in the VOC dataset.
MethodMean


key: chroma:document
string_value:
Table 7: Comparison of mean APr, achieved by different published methods, at an IoU threshold of 0.9, for all twenty classes in the VOC dataset.
MethodMean
APr(%)aeroplanebike bird boatbottlebus car cat chair cow table dog horse mbikepersonplant sheep sofa train tv
Our method 25.1 56.6 0.03 36.8 14.4 9.9 39.0 17.2 47.1 1.3 29.0 9.5 47.2 29.8 20.0 14.8 2.3 25.9 23.8 45.7 32.3
MPA 3-scale [ 37] 18.5 – – – – – – – – – – – – – – – – – – – –
MPA 1-scale [ 37] 17.3 – – – – – – – – – – – – – – – – – – – –
Arnab et al. [3] 20.1 43.7 0.03 30.0 13.2 11.4 47.3 10.9 34.5 0.7 19.6 12.1 35.6 24.3 13.3 10.7 0.4 20.7 20.9 35.0 17.4


key: chroma:document
string_value:
PFN [ 33] 15.7 43.9 0.1 24.5 7.8 4.1 32.5 6.3 42.0 0.6 25.7 3.2 31.8 13.4 8.1 5.9 1.6 14.8 14.3 25.0 8.5
Chen et al. [8] 2.6 0.6 0 0.6 0.5 4.9 9.8 1.1 8.3 0.1 1.1 1.2 1.7 0.3 0.8 0.6 0.3 0.8 7.6 4.3 6.2
SDS [ 19] 0.9 0 0 0.2 0.3 2.0 3.8 0.2 0.9 0.1 0.2 1.5 0 0 0 0.1 0.1 0 2.3 0.2 5.8
Table 8: Comparison of mean APr, achieved by different published methods, at an IoU threshold of 0.7, for all twenty classes in the VOC dataset.
MethodMean
APr(%)aeroplanebike bird boatbottlebus car cat chair cow table dog horse mbikepersonplant sheep sofa train tv


key: chroma:document
string_value:
MethodMean
APr(%)aeroplanebike bird boatbottlebus car cat chair cow table dog horse mbikepersonplant sheep sofa train tv
Our method 48.6 69.6 1.4 68.2 45.1 25.2 61.1 38.7 72.1 11.2 56.3 30.0 73.3 60.7 64.3 46.8 17.1 49.1 44.6 75.0 62.0
MPA 3-scale [ 37] 47.4 – – – – – – – – – – – – – – – – – – – –
MPA 1-scale [ 37] 45.9 – – – – – – – – – – – – – – – – – – – –
Arnab et al. [3] 45.4 68.9 0.84 65.1 38.3 26.3 64.7 31.8 72.7 6.7 45.4 32.9 67.9 60.0 63.7 41.1 13.4 43.9 41.1 74.6 48.1


key: chroma:document
string_value:
PFN [ 33] 42.5 68.5 5.6 60.4 34.8 14.9 61.4 19.2 78.6 4.2 51.1 28.2 69.6 60.7 60.5 26.5 9.8 35.1 43.9 71.2 45.6
Chen et al. [8] 27.0 40.8 0.07 40.1 16.2 19.6 56.2 26.5 46.1 2.6 25.2 16.4 36.0 22.1 20.0 22.6 7.7 27.5 19.5 47.7 46.7
SDS [ 19] 21.3 17.8 0 32.5 7.2 19.2 47.7 22.8 42.3 1.7 18.9 16.9 20.6 14.4 12.0 15.7 5.0 23.7 15.2 40.5 51.4
Table 9: Comparison of mean APr, achieved by different published methods, at an IoU threshold of 0.5, for all twenty classes in the VOC dataset.
MethodMean
APr(%)aeroplanebike bird boatbottlebus car cat chair cow table dog horse mbikepersonplant sheep sofa train tv


key: chroma:document
string_value:
MethodMean
APr(%)aeroplanebike bird boatbottlebus car cat chair cow table dog horse mbikepersonplant sheep sofa train tv
Our method 61.7 80.2 19.3 76.4 69.0 35.3 74.5 50.8 84.5 22.8 70.9 43.3 87.7 71.3 76.2 65.6 37.2 61.3 50.3 90.5 67.2
MPA 3-scale [ 37] 62.1 79.7 11.5 71.6 54.6 44.7 80.9 62.0 85.4 26.5 64.5 46.6 87.6 71.7 77.9 72.1 48.8 57.4 48.8 78.9 70.8
MPA 1-scale [ 37] 60.3 79.2 13.4 71.6 59.0 41.5 73.8 52.3 87.3 23.3 61.2 42.5 83.1 70.0 77.0 67.6 50.7 56.0 45.9 80.0 70.5


key: chroma:document
string_value:
Arnab et al. [3] 58.4 80.4 7.9 74.4 59.8 32.7 76.6 39.6 84.6 19.3 62.7 44.1 81.0 74.7 72.0 58.6 32.0 59.6 50.5 87.4 68.4
PFN [ 33] 58.7 76.4 15.6 74.2 54.1 26.3 73.8 31.4 92.1 17.4 73.7 48.1 82.2 81.7 72.0 48.4 23.7 57.7 64.4 88.9 72.3
Chen et al. [8] 46.3 63.6 0.3 61.5 43.9 33.8 67.3 46.9 74.4 8.6 52.3 31.3 63.5 48.8 47.9 48.3 26.3 40.1 33.5 66.7 67.8
SDS [ 19] 43.8 58.8 0.5 60.1 34.4 29.5 60.6 40.0 73.6 6.5 52.4 31.7 62.0 49.1 45.6 47.9 22.6 43.5 26.9 66.2 66.1


key: chroma:document
string_value:
Table 10: Comparison of mean APr, achieved by different published methods, at an IoU threshold of 0.7, for all twenty classes in the SBD dataset.
MethodMean
APr(%)aeroplanebike bird boatbottlebus car cat chair cow table dog horse mbikepersonplant sheep sofa train tv
Our method 44.8 69.0 27.4 52.7 26.4 22.4 70.3 46.0 74.7 9.6 46.8 16.9 71.6 48.4 46.3 40.3 14.8 47.6 36.5 69.7 58.2
IIS sp, rescore[
30]43.3 61.9 35.1 44.4 26.4 29.6 74.0 48.7 66.8 10.9 48.4 13.6 64.0 53.0 46.8 33.0 19.0 51.0 23.7 62.2 53.9
IIS raw [ 30] 38.7 61.8 31.5 42.0 22.0 22.7 72.4 44.8 65.4 7.2 37.6 10.4 60.4 39.6 41.9 32.5 12.0 40.9 19.9 58.8 50.8


key: chroma:document
string_value:
Table 11: Comparison of mean APr, achieved by different published methods, at an IoU threshold of 0.5, for all twenty classes in the SBD dataset.
MethodMean
APr(%)aeroplanebike bird boatbottlebus car cat chair cow table dog horse mbikepersonplant sheep sofa train tv
Our method 62.0 80.3 52.8 68.5 47.4 39.5 79.1 61.5 87.0 28.1 68.3 35.5 86.1 73.9 66.1 63.8 32.9 65.3 50.4 81.4 71.4
IIS sp, rescore[
30]63.6 79.2 67.9 70.0 47.9 45.3 81.6 68.8 84.1 30.4 65.5 31.8 83.6 75.5 74.5 66.6 37.7 70.6 44.7 77.7 68.7
IIS raw [ 30] 60.1 77.3 65.3 65.5 42.5 35.4 80.3 62.2 83.9 27.2 61.6 32.4 82.3 70.9 71.4 63.1 31.3 63.6 44.9 78.3 62.4


key: chroma:document
string_value:
Object Detection in Video with
Spatiotemporal Sampling Networks
Gedas Bertasius1, Lorenzo Torresani2, and Jianbo Shi1
1University of Pennsylvania,2Dartmouth College
Abstract. We propose a Spatiotemporal Sampling Network (STSN)
that uses deformable convolutions across time for object detection in
videos. Our STSN performs object detection in a video frame by learning to spatially sample features from the adjacent frames. This naturally
renders the approach robust to occlusion or motion blur in individual
frames. Our framework does not require additional supervision, as it optimizes sampling locations directly with respect to object detection performance. Our STSN outperforms the state-of-the-art on the ImageNet
VID dataset and compared to prior video object detection methods it
uses a simpler design, and does not require optical ﬂow data for training.
1 Introduction
In recent years, deep convolutional networks have achieved remarkable results
in many computer vision tasks [1,2,3,4,5,6,7,8], including object detection in images [9,10,11,12,13,14,15,16,17,18,19]. However, directly applying these imagelevel models to object detection in video is diﬃcult due to motion blur, video


key: chroma:document
string_value:
defocus, unusual poses, or object occlusions (see Figure 1). Despite these challenges, it is natural to assume that video object detectors should be more powerful than still image detectors because video contains richer information about
the same object instance (e.g., its appearance in diﬀerent poses, and from different viewpoints). The key challenge then is designing a model that eﬀectively
exploits temporal information in videos.
Prior work [20,21,22,23] has proposed to exploit such temporal information
in videos by means of various post-processing steps aimed at making object detections coherent across time. However, since temporal coherence is enforced in
a second stage, typically these methods cannot be trained end-to-end. To overcome this limitation, recent work [24] has introduced a ﬂow-based aggregation
network that is trainable end-to-end. It exploits optical ﬂow to ﬁnd correspondences across time and it then aggregates features across temporal correspondences to smooth object detections over adjacent frames. However, one of the
downsides of this new model is that in addition to performing object detection,
it also needs to predict motion. This is disadvantageous due to the following reasons: 1) designing an eﬀective ﬂow network architecture is not trivial, 2) training


key: chroma:document
string_value:
it also needs to predict motion. This is disadvantageous due to the following reasons: 1) designing an eﬀective ﬂow network architecture is not trivial, 2) training
such a model requires large amounts of ﬂow data, which may be diﬃcult and
costly to obtain, 3) integrating a ﬂow network and a detection network into a
single model may be challenging due to factors such as diﬀerent loss functions,
diﬀering training procedures for each network, etc. arXiv:1803.05549v2  [cs.CV]  24 Jul 2018
2 G. Bertasius, L. Torresani, J. Shi
Motion Blur
Occlusion
Unusual Pose
Video DefocusChallengingEasierChallengingEasier
Fig.1: An illustration of the common challenges associated with object detection
in video. These include video defocus, motion blur, occlusions and unusual poses.
Theboundingboxesdenotetheobjectsthatwewanttodetectintheseexamples.
To address these shortcomings, in this work, we introduce a simple, yet effective Spatiotemporal Sampling Network (STSN) that uses deformable convolutions [25] across space and time to leverage temporal information for object


key: chroma:document
string_value:
detection in video. Our STSN learns to spatially sample useful feature points
from nearby video frames such that object detection accuracy in a given video
frame is maximized. To achieve this, we train our STSN end-to-end on a large set
of video frames labeled with bounding boxes. We show that this leads to a better accuracy compared to the state-of-the-art on the ImageNet VID dataset [26],
without requiring complex ﬂow network design, or the need to train the network
on large amounts of ﬂow data.
2 Related Work
2.1 Object Detection in Images
Modernobjectdetectors[9,10,11,12,13,14,15,16,17,18,19]arepredominantlybuilt
on some form of deep CNNs [1,3,5]. One of the earliest deep CNN object detection systems was R-CNN [14], which involved a two-stage pipeline where object
proposals were extracted in the ﬁrst stage, and then each proposal was classiﬁed
using a CNN. To reduce the computational burden, the methods in [9], and [13]
leveraged ROI pooling, which led to more eﬃcient learning. Furthermore, to
unify the entire object detection pipeline, Faster R-CNN [12] replaced various


key: chroma:document
string_value:
leveraged ROI pooling, which led to more eﬃcient learning. Furthermore, to
unify the entire object detection pipeline, Faster R-CNN [12] replaced various
region proposal methods by another network to make the entire system trainable end-to-end. Following this work, several methods [18,19] extended Faster
R-CNN into a system that runs in real time with small reduction in performance.
Additionally, recent work [17] introduced position sensitive ROI pooling, which
signiﬁcantly improved the detection eﬃciency compared to prior object detection systems. Finally, two recent methods, Mask R-CNN [10], and Deformable
CNNs [25], improved object detection results even further and they represent
the current state-of-the-art in object detection. Whereas Mask-RCNNs use an
additional branch that predicts a mask for each region of interest, Deformable
Object Detection in Video with Spatiotemporal Sampling Networks 3
CNNs employ deformable convolutions, which allow the network to condition
discriminatively its receptive ﬁeld on the input, and to also model deformations
of objects more robustly.
While the aforementioned methods work well on images, they are not designed to exploit temporal relationships in video. Instead, our Spatiotemporal


key: chroma:document
string_value:
of objects more robustly.
While the aforementioned methods work well on images, they are not designed to exploit temporal relationships in video. Instead, our Spatiotemporal
Sampling Network (STSN), is speciﬁcally designed for a video object detection
task. Unlike standard Deformable CNNs [25], which use deformable convolution
in the spatial domain, our STSN learns to sample features temporally across
diﬀerent video frames, which leads to improved video object detection accuracy.
2.2 Object Detection in Videos
Up until the introduction of the ImageNet VID challenge [26], there were no
large-scalebenchmarksforvideoobjectdetection.Thus,thereareonlyfewmethods that we can compare our work to. T-CNNs [20,21] use a video object detection pipeline that involves predicting optical ﬂow ﬁrst, then propagating imagelevel predictions according to the ﬂow, and ﬁnally using a tracking algorithm to
select temporally consistent high conﬁdence detections. Seq-NMS [22] constructs
a temporal graph from overlapping bounding box detections across the adjacent
frames, and then uses dynamic programming to select bounding box sequences
with the highest overall detection score. The work of Lee et al. [23] treats a video


key: chroma:document
string_value:
frames, and then uses dynamic programming to select bounding box sequences
with the highest overall detection score. The work of Lee et al. [23] treats a video
object detection task as a multi-object tracking problem. Finally, the method of
Feichtenhofer et al. [27] proposes a ConvNet architecture that solves detection
and tracking problems jointly, and then applies a Viterbi algorithm to link the
detections across time.
The approach most similar to our work is the method of Zhu et al. [24], who
proposed an end-to-end trainable network that jointly estimates optical ﬂow and
also detects objects in video. This is accomplished by using the predicted optical
ﬂow to align the features from the adjacent frames. The aggregated features are
then fed as input to the detection network.
OurmethodisbeneﬁcialoverthemethodsthatuseopticalﬂowCNNssuchas
themethodofZhuetal.[24].First,wenotethatpretrainedopticalﬂowCNNsdo
not always generalize to new datasets, which may hinder video object detection
performance. In contrast, our method has a learnable spatiotemporal sampling
module that is discriminatively trained from object detection labels, and thus, it


key: chroma:document
string_value:
performance. In contrast, our method has a learnable spatiotemporal sampling
module that is discriminatively trained from object detection labels, and thus, it
does not suﬀer from this issue. Furthermore, our STSN can be trained for video
object detection in a single stage end-to-end. In comparison, methods that rely
on optical ﬂow require an additional stage to train an optical ﬂow CNN, which
renders the training procedure more cumbersome and lengthy. For example, we
notethatitwouldtakeaboutfourdaystotrainanopticalﬂowCNNofFGFA[24]
from scratch and then four additional days to train FGFA [24] for video object
detection, making it eight days of total training time. In contrast, our STSN is
trained in a single stage in only 4 days. Finally, we point out that our STSN also
yields a gain —albeit moderate— in video object detection accuracy.
4 G. Bertasius, L. Torresani, J. Shi
3 Background: Deformable Convolution
Before describing our method, we ﬁrst review some background information on
deformable convolution [25], which is one of the key components of our STSN.
Let us ﬁrst note that a standard 2D convolution is comprised of two steps: 1)


key: chroma:document
string_value:
Let us ﬁrst note that a standard 2D convolution is comprised of two steps: 1)
sampling locations on a uniformly-spaced grid R, and 2) performing a weighted
summation of sampled values using weights w. For example, if we consider a
standard 2D convolution with a 33kernel, and a dilation factor of 1, the grid
Ris deﬁned asR=f(1;1);(1;0);:::; (0;1);(1;1)g. Under a standard 2D
convolution, to compute a new value at pixel location p0in the output feature
mapy, we would perform the following operation on the input feature map x:
y(p0) =X
pn2Rw(pn)x(p0+pn);(1)
Instead, in a deformable 2D convolution, the grid Ris augmented with dataconditioned oﬀsets fpnjn= 1;:::;Ng, whereN=jRj. We can then compute
a deformable convolution as:
y(p0) =X
pn2Rw(pn)x(p0+pn+pn)(2)


key: chroma:document
string_value:
a deformable convolution as:
y(p0) =X
pn2Rw(pn)x(p0+pn+pn)(2)
Since the oﬀset pnis typically fractional, the operation above is implemented using bilinear interpolation. Note that the oﬀsets are obtained by applying a separate convolutional layer to the activation tensor containing the feature
mapx.Thisyieldsanoﬀsetmapthathasthesamespatialresolutionastheinput
feature map. Also, note that the oﬀsets are shared across all feature channels of
a given activation tensor. During training, the weights for the deformable convolution kernel, and the oﬀsets kernel are learned jointly by propagating gradients
through the bilinear interpolation operator. We refer the reader to the original
work that introduced deformable convolutions [25] for further details.
4 Spatiotemporal Sampling Network
Our goal is to design a network architecture that incorporates temporal information for object detection in video.
Let us denote with Itthe frame at time tin the video. Let us consider one
of the scenarios depicted in Figure 1, e.g., a setting where Itis blurry, contains
an object in an unusual pose, or perhaps an occlusion. But let us assume that


key: chroma:document
string_value:
of the scenarios depicted in Figure 1, e.g., a setting where Itis blurry, contains
an object in an unusual pose, or perhaps an occlusion. But let us assume that
a nearby frame It+kincludes the same object clearly visible and in a relatively
standard pose. If we only had access to It, accurate object detection would be
very challenging. However, leveraging information from It+kmay enable more
robust detection in the frame It. Thus, the main challenge in this setting is
incorporating object-level information from the supporting frameIt+kfor an improved object detection accuracy in the reference frameIt. Note that in our
system each frame in the video is treated in turn as a reference frame in order to
produce object detection in every frame of the video. Furthermore, in practice
Object Detection in Video with Spatiotemporal Sampling Networks 5
Reference Frame (time t)
Supporting Frame (time t+k)
Def. Conv.OffsetsDef. Conv.Offsets
Sampled Feature Map
Concat
Fig.2: Our spatiotemporal sampling mechanism, which we use for video object
detection. Given the task of detecting objects in a particular video frame (i.e.,
a reference frame), our goal is to incorporate information from a nearby frame
of the same video (i.e., a supporting frame). First, we extract features from


key: chroma:document
string_value:
a reference frame), our goal is to incorporate information from a nearby frame
of the same video (i.e., a supporting frame). First, we extract features from
both frames via a backbone convolutional network (CNN). Next, we concatenate
the features from the reference and supporting frames, and feed them through
multiple deformable convolutional layers. The last of such layers produces oﬀsets
that are used to sample informative features from the supporting frame. Our
spatiotemporal sampling scheme allows us to produce accurate detections even
if objects in the reference frame appear blurry or occluded.
we use 2Ksupporting frames for detection in the reference frame, by taking
theKpreceding frames and the Ksubsequent frames as supporting frames, i.e.
fItK;It(K1);:::;I t1;It+1;:::;I t+(K1);It+Kg. However, for ease of explanation we introduce our STSN by considering a single supporting frame It+k.
To eﬀectively integrate temporal information we need two things: 1) powerful
object-level features from an image-level network, and 2) an ability to sample
useful object-level features from the supporting frames for the reference frame.


key: chroma:document
string_value:
object-level features from an image-level network, and 2) an ability to sample
useful object-level features from the supporting frames for the reference frame.
Weachievetheformerbyemployingastate-of-the-artbackbonenetwork.Forthe
latter, we design a spatiotemporal sampling scheme, which we describe below.
Our STSN can be summarized in four steps. First, a backbone convolutional
network computes object-level features for each video frame individually. Then,
spatiotemporal sampling blocks are applied to the object-level feature maps in
order to sample relevant features from nearby frames conditioned on the input
reference frame.Next,thesampledfeaturesfromeachvideoframearetemporally
aggregated into a single feature tensor for the reference frame using a per-pixel
weighted summation. Finally, the feature tensor is provided as input to the
detectionnetworktoproduceﬁnalobjectdetectionresultsforthegivenreference
frame. We note that our framework integrates these conceptually-distinct four
steps into a single architecture, which we train end-to-end.
6 G. Bertasius, L. Torresani, J. Shi
Backbone Architecture. Our backbone network is applied to each frame


key: chroma:document
string_value:
6 G. Bertasius, L. Torresani, J. Shi
Backbone Architecture. Our backbone network is applied to each frame
of the video. As backbone network, we use a Deformable CNN [25] based on
the ResNet-101 [5] architecture, which is one of the top-performing object detection systems at the moment. Similarly to [25], our backbone network employs
6deformable convolutional layers. We also note that even though we use a Deformable CNN architecture, our system can easily integrate other architectures
and thus it can beneﬁt from future improvements in still-image object detection.
Spatiotemporal Feature Sampling. Our main contribution is the design
of a spatiotemporal sampling mechanism, which seamlessly integrates temporal
information in a given video. As a ﬁrst step, we feed the reference frame Itand
the supporting frame It+kthrough our image-level backbone network, which
produces feature tensors ftandft+k, respectively. Note that ft;ft+k2Rchw
wherec;h, andware the number of channels, the height, and the width of the
activation tensor. The feature tensors ft, andft+kare then concatenated into a


key: chroma:document
string_value:
activation tensor. The feature tensors ft, andft+kare then concatenated into a
new feature tensor ft;t+k2R2chw. Note that this tensor ft;t+know has twice
as many channels as our initial tensors, and that it now contains object-level
information from both the reference and the supporting frame.
Next, we use the tensor ft;t+kto predict (x;y)location oﬀsets, which are
then used to sample the supporting tensor ft+k. The sampling mechanism is
implementedusingadeformableconvolutionallayer,whichtakes1)thepredicted
oﬀsets, and 2) the supporting tensor ft+kas its inputs, and then outputs a
newly sampled feature tensor gt;t+k, which can be used for object detection in
the reference frame. We use subscript t;t+kto denote the resampled tensor
because, although gis obtained by resampling the supporting tensor, the oﬀset
computation uses both the reference as well as the supporting frame. A detailed
illustration of our spatiotemporal sampling scheme is presented in Figure 2.
In practice, our spatiotemporal sampling block has 4deformable convolution


key: chroma:document
string_value:
illustration of our spatiotemporal sampling scheme is presented in Figure 2.
In practice, our spatiotemporal sampling block has 4deformable convolution
layers (only 2are shown in Figure 2). This means that the initially predicted
oﬀsetso(1)
t;t+kand the concatenated temporal features ft;t+kare ﬁrst used as
inputs to a deformable convolution layer that outputs a new feature map g(1)
t;t+k.
Next, we use g(1)
t;t+kto predict oﬀsets o(2)
t;t+k, and a new feature map g(2)
t;t+k. This
continues for 2 more layers until we obtain oﬀsets o(4)
t;t+k, which are then used
to sample the points out of the supporting feature map ft+k. The ﬁnal sampled
feature map g(4)
t;t+kis obtained via another deformable convolutional layer that
takes as inputs oﬀsets o(4)
t;t+kand the original supporting feature map ft+k.
Our proposed spatiotemporal sampling mechanism learns, which object-level
features in the supporting frame are useful for object detection in the reference


key: chroma:document
string_value:
Our proposed spatiotemporal sampling mechanism learns, which object-level
features in the supporting frame are useful for object detection in the reference
frame. Conceptually, it replaces the optical ﬂow used in [24] to establish temporal correspondences with a learnable module that is discriminatively trained
from object detection labels. In our experimental section, we show that such a
sampling scheme allows us to improve video object detection performance over
the still-image baseline and the ﬂow-based method of Zhu et al. [24] without
training our model on optical ﬂow data.
Feature Aggregation. The spatiotemporal sampling procedure is applied
for all the supporting frames in the selected range. Note, that this includes a
Object Detection in Video with Spatiotemporal Sampling Networks 7
special case, when the reference frame is treated as a supporting frame to itself to
produceg(4)
t;t, which is a feature tensor computed from only the reference frame.
The resulting feature tensors have the following form: g(4)
t;t+k2Rc(4)hw.
Thesefeaturetensorsareaggregatedintoanoutputfeaturetensor gagg
t2Rc(4)hw
for the reference frame. This tensor captures information from the reference


key: chroma:document
string_value:
t2Rc(4)hw
for the reference frame. This tensor captures information from the reference
frame, itsKpreceding frames and its Ksubsequent frames. The output tensor
valuegagg
t(p)for frametat pixelpis computed as a weighted summation:
gagg
t(p) =KX
k=Kwt;t+k(p)g(4)
t;t+k(p) (3)
Inspired by strong results presented in [24], we use their proposed feature
aggregation method where the weights windicate the importance of each supporting frame to the reference frame. To compute the weights w, we attach a
3-layer subnetwork S(x)to the features g(4)
t;t+kand then compute their intermediate feature representations S(g(4)
t;t+k). We then obtain the weights wby applying
an exponential function on the cosine similarity between each corresponding
feature point in a reference frame and a supporting frame:
wt;t+k(p) = exp 
S(g(4)
t;t)(p)S(g(4)
t;t+k)(p)
jS(g(4)
t;t)(p)jjS(g(4)


key: chroma:document
string_value:
t;t+k)(p)
jS(g(4)
t;t)(p)jjS(g(4)
t;t+k)(p)j!
(4)
Finally,allweights warefedintothesoftmaxlayer,toensurethattheweights
sum up to 1at each pixel location p(i.e.,PK
k=Kwt;t+k(p) = 18p).
Object Detection. Finally, the aggregated feature tensor gagg
tis used as
input to the detection network, which outputs the ﬁnal bounding box predictions
and their object class probabilities. We describe more details related to the
detection network in the next section along with other implementation details.
4.1 Implementation Details
For our experiments we use the MXNet [28] library. Below we provide details
related to our STSN architecture, and our training and inference procedures.
Architecture. For our backbone network we adopt a state-of-the-art Deformable CNN [25] based on the ResNet-101 [5] architecture. Our spatiotemporal sampling block consists of four 33deformable convolutional layers each
with 1024output channels. In addition, it also has four 33convolutional layers


key: chroma:document
string_value:
with 1024output channels. In addition, it also has four 33convolutional layers
predicting (x;y)oﬀsets. To implement a subnetwork S(x)that predicts feature
aggregation weights, we use a sequence of 11,33and11convolutional
layers with 512;512and2048output channels respectively. Our detection network is implemented based on the deformable R-FCN design [17,29,25]. When
feeding the aggregated feature gagg
tto the detection network, we split its 1024
channels into two parts, and feed the ﬁrst and the last 512channels to the
RPN and R-FCN sub-networks respectively. For the RPN, we use 9anchors and
300proposals for each image. Furthermore, for the R-FCN, we use deformable
position-sensitive ROI pooling with 77groups.
8 G. Bertasius, L. Torresani, J. Shi
Training. Our entire STSN model is fully diﬀerentiable, and thus, trainable
end-to-end. During training, we resize all input images to a shorter side of 600
pixels, and use T= 3frames to train our model (i.e., K= 1). More speciﬁcally,


key: chroma:document
string_value:
pixels, and use T= 3frames to train our model (i.e., K= 1). More speciﬁcally,
we randomly sample one supporting frame before and one supporting frame after
the reference frame. We observed that using more supporting frames in training
does not lead to a higher accuracy.
For the rest of our training procedure, we follow the protocol outlined in [24].
Speciﬁcally, we train our model in two stages. First, we pre-train our full model
on the Imagenet DET dataset using the annotations of the 30object classes
that overlap with the Imagenet VID dataset. Note that Imagenet DET dataset
contains only images, and thus, we cannot sample meaningful supporting frames
in this case. Therefore, in the case of images, we use the reference frames as our
supportingframes.Afterwards,theentiremodelistrainedfor 120Kiterationson
4Tesla K40 GPUs with each GPU holding a single mini-batch. The learning rate
is set to 0:001and0:0001for the ﬁrst 80Kand the last 40Kiterations respectively. Afterwards, we ﬁnetune the entire model on the Imagenet VID dataset
for60Kiterations with a learning rate of 0:001and0:0001for the ﬁrst 40Kand


key: chroma:document
string_value:
for60Kiterations with a learning rate of 0:001and0:0001for the ﬁrst 40Kand
the last 20Kiterations respectively. Note that in the second stage of training
we sample the supporting frames randomly within a certain neighborhood of a
reference frame (as described above).
Inference. During inference, we use T= 27, meaning that we consider
K= 13supporting frames before and after the reference frame. To avoid GPU
memory issues, we ﬁrst extract features from the backbone network for each
image individually, and then cache these features in the memory. Afterwards,
we feed all these features into our spatiotemporal sampling block. At the end,
standard NMS with a threshold of 0:3is applied to reﬁne the detections. To
handle the ﬁrst and the last K= 13frames in the video —two boundary cases
that require sampling the neighboring frames beyond the video start and end,
we pad the start of a video with Kcopies of the ﬁrst frame, and the end of a
video with Kcopies of the last frame.
5 Experimental Results
In this section, we evaluate our approach for video object detection on the ImageNet VID [26] dataset, which has 3;862and555training and testing video


key: chroma:document
string_value:
5 Experimental Results
In this section, we evaluate our approach for video object detection on the ImageNet VID [26] dataset, which has 3;862and555training and testing video
clips respectively. Each video is annotated with bounding boxes. The frames
from each video are extracted at 2530fps. The dataset contains 30object
categories that are a subset of the 200categories in the ImageNet DET dataset.
5.1 Quantitative Results
To assess the eﬀectiveness of our method we compare it to several relevant baselines, mainly two state-of-the-art methods FGFA [24] and D&T [27]. First, to
verify that using temporal information from video is beneﬁcial, we include a
static image-level variant of our model (SSN) that uses only the reference frame
Object Detection in Video with Spatiotemporal Sampling Networks 9
Methods
D&T [27] Our SSN FGFA [24] Our STSN D&T+ [27] Our STSN+
No FlowNet? 3 - 7 3 3 3
Not Using Flow Data? 3 - 7 3 3 3
No Temporal Post-Processing? 3 - 3 3 7 7
mAP@0.5 75.8 76.0 78.8 78.9 79.8 80.4


key: chroma:document
string_value:
No Temporal Post-Processing? 3 - 3 3 7 7
mAP@0.5 75.8 76.0 78.8 78.9 79.8 80.4
Table 1: We use the ImageNet VID [26] dataset to compare our STSN to the
state-of-the-art FGFA [24] and D&T [27] methods. Note that SSN refers to our
static baseline, which is obtained by using only the reference frame for output
generation (no temporal info). Also note, that D&T+ and STSN+ refer to D&T
and STSN baselines with temporal post-processing applied on top of the CNN
outputs. Based on these results, we ﬁrst point out that unlike FGFA, our STSN
does not rely on the external optical ﬂow data, and still yields higher mAP
(78:9vs78:8). Furthermore, when no temporal post-processing is used, our
STSN produces superior performance in comparison to the D&T baseline ( 78:9
vs75:8). Finally, we demonstrate that if we use a simple Seq-NMS [22] temporal
post-processing scheme on top of our STSN predictions, we can further improve
our results and outperform all the other baselines.


key: chroma:document
string_value:
post-processing scheme on top of our STSN predictions, we can further improve
our results and outperform all the other baselines.
to make its predictions. Furthermore, we also want to show that our spatiotemporal sampling scheme works as eﬀectively as the optical ﬂow network in [24],
but without requiring optical ﬂow supervision. To do so, we replace the optical
ﬂow network from [24], with our spatiotemporal sampling mechanism. The rest
of the architecture and the training details are kept the same for both baselines.
Such an experimental design allows us to directly compare the eﬀectiveness of
our spatiotemporal sampling scheme and the optical ﬂow network of FGFA [24].
Finally, we demonstrate that our method performs better than the D&T [27]
method in two scenarios: 1) when we only useCNN-level outputs forvideo object
detection, and also 2) when we allow temporal post-processing techniques such
as Seq-NMS to be applied on top of the CNN outputs. We note that in Table 1,
D&T [27] and STSN refer to the CNN-level baselines whereas D&T+ [27] and


key: chroma:document
string_value:
D&T [27] and STSN refer to the CNN-level baselines whereas D&T+ [27] and
STSN+ denote these same methods but with temporal post-processing (i.e. SeqNMS[22],object-tubebasedlinking[27],etc)appliedontopoftheCNNoutputs.
We present our results in Table 1, where we assess each method according to
several criteria. In the ﬁrst row of Table 1, we list whether a given method requires integrating a separate ﬂow network into its training / prediction pipeline.
Ideally, we would want to eliminate this step because optical ﬂow prediction requiresdesigningahighlycomplexﬂownetworkarchitecture.Wealsolistwhether
a given method requires pre-training on the external optical ﬂow data, which we
would want to avoid since it makes the whole training pipeline more costly.
Additionally, we list, whether a given method uses any external temporal postprocessing steps, which we would want to eliminate because they typically make
the training / prediction pipeline disjoint and more complex. Finally, we assess
each method according to the standard mean average precision (mAP) metric
at intersection over union (IoU) threshold of 0:5.
10 G. Bertasius, L. Torresani, J. Shi


key: chroma:document
string_value:
at intersection over union (IoU) threshold of 0:5.
10 G. Bertasius, L. Torresani, J. Shi
0 2 6 10 14 18 22 26 30Number of Supporting Frames7676.57777.57878.579mAP
-10 -8 -6 -4 -2 0 2 4 6 8 10Time Delta w.r.t Reference Frame0.060.0650.070.0750.080.0850.090.0950.1Weight Magnitude
Fig.3: A ﬁgure illustrating some of our ablation experiments. Left:we plot mAP
as a function of the number of supporting frames used by our STSN. From this
plot, we notice that the video object detection accuracy improves as we use
more supporting frames. Right:To understand the contribution of each of the
supporting frames, we plot the average weight magnitudes wt;t+k(p)for diﬀerent
values ofk. Here,prepresents a point at the center of an object. From this plot,
we observe that the largest weights are associated with the supporting frames
that are near the reference frame. However, note that even supporting frames
that are further away from the reference frame (e.g. k= 9) contribute quite
substantially to the ﬁnal object detection predictions.


key: chroma:document
string_value:
that are further away from the reference frame (e.g. k= 9) contribute quite
substantially to the ﬁnal object detection predictions.
Based on our results in Table 1, we make the following conclusions. First, we
notethatourSTSNproducesbetterquantitativeresultsthanthestate-of-the-art
FGFA method ( 78:9vs78:8). We acknowledge that our accuracy improvement
over FGFA is moderate. However, we point out that our STSN operates in a
much more challenging setting than FGFA. Unlike FGFA, our STSN does not
use any optical ﬂow supervision. Instead, it is trained directly for video object
detection. The fact that STSN learns temporal correspondences without direct
optical ﬂow supervision, and still outperforms FGFA is quite impressive. Such
results also show the beneﬁt of discriminative end-to-end training with respect
to the ﬁnal video object detection task objective.
WenextcompareourSTSNtotheD&Tbaseline[27].Wenotethatunlikefor
the FGFA [24] baseline, it is much harder to make a direct comparison between


key: chroma:document
string_value:
the FGFA [24] baseline, it is much harder to make a direct comparison between
STSN and D&T. Whereas our STSN aims to produce powerful spatiotemporal features, the method of D&T [27] is targeted more for smoothing the ﬁnal
bounding box predictions across time. Thus, we believe that these two methods
are complementary, and it would be possible to integrate them together for the
model that produces both: temporally smooth features, as well as temporally
smooth bounding box predictions. We also note that our STSN and D&T [27]
use slightly diﬀerent architectures (both based on ResNet-101 though).
First, we compare STSN and D&T in a setting when no temporal postprocessing(i.e.Seq-NMS[22],object-tubelinking[27],etc)isused,andshowthat
ourSTSNoutperformstheD&Tbaselinebyasubstantialmargin( 78:9vs75:8).
Object Detection in Video with Spatiotemporal Sampling Networks 11
These results indicate, that our STSN is able to learn powerful spatiotemporal
features, and produce solid video object detection results even without temporal
post-processing algorithms that link bounding box detections over time.


key: chroma:document
string_value:
features, and produce solid video object detection results even without temporal
post-processing algorithms that link bounding box detections over time.
Afterwards, we show that integrating a simple temporal post-processing algorithm Seq-NMS [22] further improves our STSN’s results. Such a scheme allows
us to outperform the D&T+ baseline ( 80:4vs79:8), which uses a similar Viterbi
based temporal post-processing scheme.
5.2 Ablation Studies
Optimal Number of Supporting Frames. In the left subplot of Figure 3,
we also illustrate how the number of supporting frames aﬀects the video object
detection accuracy. We notice that the performance keeps increasing as we add
more supporting frames, and then plateaus at T= 27.
Increasing the Temporal Stride. We also investigate how the temporal
stridek, at which we sample the supporting frames, aﬀects STSN’s performance.
We report that temporal strides of k= 2andk= 4, yield mAP scores of 79:0
and77:9, respectively. Thus, k= 2yields a slight improvement over our original
78:9mAP score. However, increasing kto larger values reduces the accuracy.
Feature Aggregation Weight Analysis. To analyze how much each of


key: chroma:document
string_value:
78:9mAP score. However, increasing kto larger values reduces the accuracy.
Feature Aggregation Weight Analysis. To analyze how much each of
the supporting frame contributes to the ﬁnal object detections, we visualize the
average weight magnitudes wt;t+k(p)for diﬀerent values of k. This visualization
is presented in the right subplot of Figure 3. We note that in this case, the weight
magnitudes correspond to the point p, which is located at the center of an object.
From this plot, we can conclude that the largest contribution comes from the
supporting frames that are near the reference frame ( k=1;0;1). However,
note that even even supporting frames that are further away from the reference
frame (e.g. k=9;9) have non-zero weights, and contribute quite substantially
to the ﬁnal object detection predictions.
5.3 Qualitative Results
To understand how our STSN exploits temporal information from a given video,
we visualize in Figure 4, the average oﬀsets predicted by the STSN sampling
block. These oﬀsets are used by the STSN to decide, which object-level informationfromthesupportingframeshouldbeusedtodetectanobjectinthereference


key: chroma:document
string_value:
frame. The green square in the reference frame depicts a pixel, for which we want
to compute a convolution output. The red square in the supporting frame represents an average oﬀset, which is used to determine which feature points from the
supporting frame should be sampled. The yellow arrow indicates object’s motion between the reference frame and the supporting frame. Note that despite
a relatively large motion between the reference and the supporting frames, our
STSN samples features from the supporting frame right around the center of the
object, which is exactly what we want. Such spatiotemporal sampling allows us
to detect objects even if they appear blurry or occluded in the reference frame.
12 G. Bertasius, L. Torresani, J. Shi
Reference Frame (t)Supporting Frame (t+9)
Reference Frame (t)Supporting Frame (t+9)
Fig.4: An illustration of our spatiotemporal sampling scheme (zoom-in for a better view). The green square indicates a point in the reference frame, for which
we want to compute a new convolutional output. The red square indicates the
corresponding point predicted by our STSN in a supporting frame. The yellow
arrow illustrates the estimated object motion. Although our model is trained
discriminatively for object detection and notfor tracking or motion estimation,


key: chroma:document
string_value:
arrow illustrates the estimated object motion. Although our model is trained
discriminatively for object detection and notfor tracking or motion estimation,
our STSN learns to sample from the supporting frame at locations that coincide almost perfectly with the same object. This allows our method to perform
accurate object detection even if objects in the reference frame are blurry or
occluded.
In addition, based on the results in Figure 4, we observe that even without
an explicit optical ﬂow supervision, our STSN learns to accurately capture the
motion of the objects, which is another appealing property of our model. In
fact, in Figure 5, we illustrate several examples of using our STSN to track
objects in a given video. From Figure 5, we observe that despite a relatively
large motion in each sequence, our STSN accurately samples features around
objects in every supporting frame. Such results indicate that we may be able
to use our sampling mechanism for discriminative object tracking. In fact, we
note that the commonly used dense optical ﬂow methods are often redundant
because most applications do not require ﬂow prediction for every single pixel. In
comparison, we point out that our STSN captures a more discriminative form of
motion, which is learned to exclusively beneﬁt a video object detection task. In
our supplementary material, we include more of such results in the video form.


key: chroma:document
string_value:
motion, which is learned to exclusively beneﬁt a video object detection task. In
our supplementary material, we include more of such results in the video form.
In Figure 6, we also illustrate object detections of the static SSN baseline,
and those of our full STSN model (zoom-in to see the probabilities and class
predictions). In all of these cases, we observe that incorporating temporal information helps STSN to correct the mistakes made by the static baseline. For
instance, in the third row of Figure 6, a static SSN baseline incorrectly labels
an object in the reference frame as a bird, which happens due to the occluded
head of the lizard. However, STSN ﬁxes this mistake by looking at the supporting frames, and by sampling around the lizard body and its head (See Row 3,
Column 1 in Figure 6). Furthermore, in the last row, a static SSN baseline fails
to detect one of the bicycles because it is occluded in the reference frame. STSN
ﬁxes this error, by sampling around the missed bicycle in the supporting frame
Object Detection in Video with Spatiotemporal Sampling Networks 13
Reference Frame (t)Supporting Frame (t-9)Supporting Frame (t-4)Supporting Frame (t+4)Supporting Frame (t+9)


key: chroma:document
string_value:
Reference Frame (t)Supporting Frame (t-9)Supporting Frame (t-4)Supporting Frame (t+4)Supporting Frame (t+9)
Fig.5: An illustration of using our spatiotemporal sampling scheme in action.
The green square indicates a ﬁxed object location in the reference frame. The
red square depicts a location in a supporting frame, from which relevant features
are sampled. Even without optical ﬂow supervision, our STSN learns to track
these objects in video. In our supplementary material, we include more of such
examples in the video format.
where the bicycle is more clearly visible. Similar behavior also occurs in other
cases where STSN successfully resolves occlusion and blurriness issues.
6 Conclusion
Inthiswork,weintroducedtheSpatiotemporalSamplingNetwork(STSN)which
is a new architecture for object detection in video. Compared to the state-of-theart FGFA [24] method, our model involves a simpler design, it does not require
optical ﬂow computation and it produces higher video object detection accuracy.
Ourmodelisfullydiﬀerentiable,andunlikepriorvideoobjectdetectionmethods,
it does not necessitate optical ﬂow training data. This renders our model easy to


key: chroma:document
string_value:
it does not necessitate optical ﬂow training data. This renders our model easy to
train end-to-end. Our future work will include experimenting with more complex
design of spatiotemporal sampling blocks.
7 Acknowledgements
This work was funded in part by NSF award CNS-120552. We gratefully acknowledge NVIDIA and Facebook for the donation of GPUs used for portions
of this work.
14 G. Bertasius, L. Torresani, J. Shi
SSN (Static)STSN (Ours)
Sampled Points in the     Supporting Frame
Fig.6: A ﬁgure illustrating object detection examples where our spatiotemporal
sampling mechanism helps STSN to correct the mistakes made by a static SSN
baseline (please zoom-in to see the class predictions and their probabilities).
These mistakes typically occur due to occlusions, blurriness, etc. STSN ﬁxes
these errors by using relevant object level information from supporting frames.
InColumn 1weillustratethepointsinthesupportingframethatSTSNconsiders
relevant when computing the output for a point denoted by the green square in
Column 2.
Object Detection in Video with Spatiotemporal Sampling Networks 15
References


key: chroma:document
string_value:
relevant when computing the output for a point denoted by the green square in
Column 2.
Object Detection in Video with Spatiotemporal Sampling Networks 15
References
1. Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classiﬁcation with deep
convolutional neural networks. In: NIPS. (2012)
2. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D.,
Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions. In: Computer
Vision and Pattern Recognition (CVPR). (2015)
3. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale
image recognition. In: ICLR. (2015)
4. Bertasius, G., Shi, J., Torresani, L.: Deepedge: A multi-scale bifurcated deep
network for top-down contour detection. In: The IEEE Conference on Computer
Vision and Pattern Recognition (CVPR). (June 2015)


key: chroma:document
string_value:
network for top-down contour detection. In: The IEEE Conference on Computer
Vision and Pattern Recognition (CVPR). (June 2015)
5. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. 2016IEEEConferenceonComputerVisionandPatternRecognition(CVPR)
(2016) 770–778
6. Xie,S.,Girshick,R.,Dollár,P.,Tu,Z.,He,K.: Aggregatedresidualtransformations
for deep neural networks. In: CVPR. (2017)
7. Toshev, A., Szegedy, C.: Deeppose: Human pose estimation via deep neural networks. In: CVPR. (2014)
8. Bertasius, G., Torresani, L., Yu, S.X., Shi, J.: Convolutional random walk networks
for semantic image segmentation. In: The IEEE Conference on Computer Vision
and Pattern Recognition (CVPR). (July 2017)
9. He, K., Zhang, X., Ren, S., Sun, J.: Spatial pyramid pooling in deep convolutional
networks for visual recognition. In Fleet, D., Pajdla, T., Schiele, B., Tuytelaars,


key: chroma:document
string_value:
networks for visual recognition. In Fleet, D., Pajdla, T., Schiele, B., Tuytelaars,
T., eds.: Computer Vision – ECCV 2014. (2014)
10. He, K., Gkioxari, G., Dollár, P., Girshick, R.: Mask R-CNN. In: Proceedings of
the International Conference on Computer Vision (ICCV). (2017)
11. Lin, T.Y., Goyal, P., Girshick, R., He, K., Dollár, P.: Focal Loss for Dense Object
Detection. In: Proceedings of the International Conference on Computer Vision
(ICCV). (2017)
12. Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: Towards real-time object detection with region proposal networks. In: Neural Information Processing Systems
(NIPS). (2015)
13. Girshick, R.: Fast R-CNN. In: Proceedings of the International Conference on
Computer Vision (ICCV). (2015)
14. Girshick, R., Donahue, J., Darrell, T., Malik, J.: Rich feature hierarchies for accurate object detection and semantic segmentation. In: Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition (CVPR). (2014)


key: chroma:document
string_value:
Conference on Computer Vision and Pattern Recognition (CVPR). (2014)
15. Gupta,S.,Girshick,R.,Arbelaez,P.,Malik,J.: LearningrichfeaturesfromRGB-D
images for object detection and segmentation. In: ECCV. (2014)
16. Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.Y., Berg, A.C.:
Ssd: Single shot multibox detector. In: ECCV. (2016)
17. Dai, J., Li, Y., He, K., Sun, J.: R-fcn: Object detection via region-based fully
convolutional networks. In: Advances in Neural Information Processing Systems
29. Curran Associates, Inc. (2016) 379–387
18. Redmon,J.,Divvala,S.K.,Girshick,R.B.,Farhadi,A.: Youonlylookonce:Uniﬁed,
real-time object detection. In: 2016 IEEE Conference on Computer Vision and
Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016. (2016)
779–788
16 G. Bertasius, L. Torresani, J. Shi


key: chroma:document
string_value:
779–788
16 G. Bertasius, L. Torresani, J. Shi
19. Redmon, J., Farhadi, A.: YOLO9000: better, faster, stronger. In: 2017 IEEE
Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu,
HI, USA, July 21-26, 2017. (2017) 6517–6525
20. Kang, K., Li, H., Yan, J., Zeng, X., Yang, B., Xiao, T., Zhang, C., Wang, Z., Wang,
R., Wang, X., Ouyang, W.: T-CNN: tubelets with convolutional neural networks
for object detection from videos. IEEE TCSVT 2017 (2017)
21. Kang, K., Ouyang, W., Li, H., Wang, X.: Object detection from video tubelets
with convolutional neural networks. CoRR abs/1604.04053 (2016)
22. Han, W., Khorrami, P., Paine, T.L., Ramachandran, P., Babaeizadeh, M., Shi,
H., Li, J., Yan, S., Huang, T.S.: Seq-nms for video object detection. CoRR
abs/1602.08465 (2016)


key: chroma:document
string_value:
H., Li, J., Yan, S., Huang, T.S.: Seq-nms for video object detection. CoRR
abs/1602.08465 (2016)
23. Lee, B., Erdenee, E., Jin, S., Rhee, P.: Multi-class multi-object tracking using
changing point detection. CoRR abs/1608.08434 (2016)
24. Zhu, X., Wang, Y., Dai, J., Yuan, L., Wei, Y.: Flow-guided feature aggregation for
video object detection. In: International Conference on Computer Vision (ICCV).
(2017)
25. Dai, J., Qi, H., Xiong, Y., Li, Y., Zhang, G., Hu, H., Wei, Y.: Deformable convolutional networks. In: 2017 IEEE International Conference on Computer Vision
(ICCV). Volume 00. (Oct. 2017) 764–773
26. Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z.,
Karpathy, A., Khosla, A., Bernstein, M., Berg, A.C., Fei-Fei, L.: ImageNet Large
Scale Visual Recognition Challenge. International Journal of Computer Vision
(IJCV) 115(3) (2015) 211–252


key: chroma:document
string_value:
Scale Visual Recognition Challenge. International Journal of Computer Vision
(IJCV) 115(3) (2015) 211–252
27. Feichtenhofer, C., Pinz, A., Zisserman, A.: Detect to track and track to detect. In:
International Conference on Computer Vision (ICCV). (2017)
28. Chen, T., Li, M., Li, Y., Lin, M., Wang, N., Wang, M., Xiao, T., Xu, B., Zhang,
C., Zhang, Z.: Mxnet: A ﬂexible and eﬃcient machine learning library for heterogeneous distributed systems. CoRR abs/1512.01274 (2015)
29. Zhu, X., Xiong, Y., Dai, J., Yuan, L., Wei, Y.: Deep feature ﬂow for video recognition. In: CVPR. (2017)


key: chroma:document
string_value:
Relational inductive biases, deep learning, and graph networks
Peter W. Battaglia1,Jessica B. Hamrick1,Victor Bapst1,
Alvaro Sanchez-Gonzalez1,Vinicius Zambaldi1,Mateusz Malinowski1,
Andrea Tacchetti1,David Raposo1,Adam Santoro1,Ryan Faulkner1,
Caglar Gulcehre1,Francis Song1,Andrew Ballard1,Justin Gilmer2,
George Dahl2,Ashish Vaswani2,Kelsey Allen3,Charles Nash4,
Victoria Langston1,Chris Dyer1,Nicolas Heess1,
Daan Wierstra1,Pushmeet Kohli1,Matt Botvinick1,
Oriol Vinyals1,Yujia Li1,Razvan Pascanu1
1DeepMind;2Google Brain;3MIT;4University of Edinburgh
Abstract
Articial intelligence (AI) has undergone a renaissance recently, making major progress in
key domains such as vision, language, control, and decision-making. This has been due, in
part, to cheap data and cheap compute resources, which have t the natural strengths of deep
learning. However, many dening characteristics of human intelligence, which developed under


key: chroma:document
string_value:
part, to cheap data and cheap compute resources, which have t the natural strengths of deep
learning. However, many dening characteristics of human intelligence, which developed under
much dierent pressures, remain out of reach for current approaches. In particular, generalizing
beyond one's experiences|a hallmark of human intelligence from infancy|remains a formidable
challenge for modern AI.
The following is part position paper, part review, and part unication. We argue that
combinatorial generalization must be a top priority for AI to achieve human-like abilities, and that
structured representations and computations are key to realizing this objective. Just as biology
uses nature and nurture cooperatively, we reject the false choice between \hand-engineering"
and \end-to-end" learning, and instead advocate for an approach which benets from their
complementary strengths. We explore how using relational inductive biases within deep learning
architectures can facilitate learning about entities, relations, and rules for composing them. We
present a new building block for the AI toolkit with a strong relational inductive bias|the graph
network |which generalizes and extends various approaches for neural networks that operate
on graphs, and provides a straightforward interface for manipulating structured knowledge and
producing structured behaviors. We discuss how graph networks can support relational reasoning


key: chroma:document
string_value:
on graphs, and provides a straightforward interface for manipulating structured knowledge and
producing structured behaviors. We discuss how graph networks can support relational reasoning
and combinatorial generalization, laying the foundation for more sophisticated, interpretable,
and 
exible patterns of reasoning. As a companion to this paper, we have also released an
open-source software library for building graph networks, with demonstrations of how to use
them in practice.
1 Introduction
A key signature of human intelligence is the ability to make \innite use of nite means" (Humboldt,
1836; Chomsky, 1965), in which a small set of elements (such as words) can be productively
composed in limitless ways (such as into new sentences). This re
ects the principle of combinatorial
generalization , that is, constructing new inferences, predictions, and behaviors from known building
blocks. Here we explore how to improve modern AI's capacity for combinatorial generalization by
Corresponding author: peterbattaglia@google.com
1arXiv:1806.01261v3  [cs.LG]  17 Oct 2018
biasing learning towards structured representations and computations, and in particular, systems
that operate on graphs.


key: chroma:document
string_value:
biasing learning towards structured representations and computations, and in particular, systems
that operate on graphs.
Humans' capacity for combinatorial generalization depends critically on our cognitive mechanisms for representing structure and reasoning about relations. We represent complex systems as
compositions of entities and their interactions1(Navon, 1977; McClelland and Rumelhart, 1981;
Plaut et al., 1996; Marcus, 2001; Goodwin and Johnson-Laird, 2005; Kemp and Tenenbaum, 2008),
such as judging whether a haphazard stack of objects is stable (Battaglia et al., 2013). We use
hierarchies to abstract away from ne-grained dierences, and capture more general commonalities
between representations and behaviors (Botvinick, 2008; Tenenbaum et al., 2011), such as parts of
an object, objects in a scene, neighborhoods in a town, and towns in a country. We solve novel
problems by composing familiar skills and routines (Anderson, 1982), for example traveling to a
new location by composing familiar procedures and objectives, such as \travel by airplane", \to
San Diego", \eat at", and \an Indian restaurant". We draw analogies by aligning the relational
structure between two domains and drawing inferences about one based on corresponding knowledge


key: chroma:document
string_value:
San Diego", \eat at", and \an Indian restaurant". We draw analogies by aligning the relational
structure between two domains and drawing inferences about one based on corresponding knowledge
about the other (Gentner and Markman, 1997; Hummel and Holyoak, 2003).
Kenneth Craik's \The Nature of Explanation" (1943), connects the compositional structure of
the world to how our internal mental models are organized:
...[a human mental model] has a similar relation-structure to that of the process it
imitates. By `relation-structure' I do not mean some obscure non-physical entity which
attends the model, but the fact that it is a working physical model which works in the
same way as the process it parallels... physical reality is built up, apparently, from a few
fundamental types of units whose properties determine many of the properties of the
most complicated phenomena, and this seems to aord a sucient explanation of the
emergence of analogies between mechanisms and similarities of relation-structure among
these combinations without the necessity of any theory of objective universals. (Craik,
1943, page 51-55)
That is, the world is compositional, or at least, we understand it in compositional terms. When
learning, we either t new knowledge into our existing structured representations, or adjust the


key: chroma:document
string_value:
learning, we either t new knowledge into our existing structured representations, or adjust the
structure itself to better accommodate (and make use of) the new and the old (Tenenbaum et al.,
2006; Griths et al., 2010; Ullman et al., 2017).
The question of how to build articial systems which exhibit combinatorial generalization has
been at the heart of AI since its origins, and was central to many structured approaches, including
logic, grammars, classic planning, graphical models, causal reasoning, Bayesian nonparametrics, and
probabilistic programming (Chomsky, 1957; Nilsson and Fikes, 1970; Pearl, 1986, 2009; Russell and
Norvig, 2009; Hjort et al., 2010; Goodman et al., 2012; Ghahramani, 2015). Entire sub-elds have
focused on explicit entity- and relation-centric learning, such as relational reinforcement learning
(D zeroski et al., 2001) and statistical relational learning (Getoor and Taskar, 2007). A key reason
why structured approaches were so vital to machine learning in previous eras was, in part, because
data and computing resources were expensive, and the improved sample complexity aorded by
structured approaches' strong inductive biases was very valuable.


key: chroma:document
string_value:
data and computing resources were expensive, and the improved sample complexity aorded by
structured approaches' strong inductive biases was very valuable.
In contrast with past approaches in AI, modern deep learning methods (LeCun et al., 2015;
Schmidhuber, 2015; Goodfellow et al., 2016) often follow an \end-to-end" design philosophy which
emphasizes minimal a priori representational and computational assumptions, and seeks to avoid
explicit structure and \hand-engineering". This emphasis has t well with|and has perhaps been
armed by|the current abundance of cheap data and cheap computing resources, which make
1Whether this entails a \language of thought" (Fodor, 1975) is beyond the scope of this work.
2
trading o sample eciency for more 
exible learning a rational choice. The remarkable and rapid
advances across many challenging domains, from image classication (Krizhevsky et al., 2012;
Szegedy et al., 2017), to natural language processing (Sutskever et al., 2014; Bahdanau et al., 2015),
to game play (Mnih et al., 2015; Silver et al., 2016; Morav c k et al., 2017), are a testament to this


key: chroma:document
string_value:
to game play (Mnih et al., 2015; Silver et al., 2016; Morav c k et al., 2017), are a testament to this
minimalist principle. A prominent example is from language translation, where sequence-to-sequence
approaches (Sutskever et al., 2014; Bahdanau et al., 2015) have proven very eective without using
explicit parse trees or complex relationships between linguistic entities.
Despite deep learning's successes, however, important critiques (Marcus, 2001; Shalev-Shwartz
et al., 2017; Lake et al., 2017; Lake and Baroni, 2018; Marcus, 2018a,b; Pearl, 2018; Yuille and
Liu, 2018) have highlighted key challenges it faces in complex language and scene understanding,
reasoning about structured data, transferring learning beyond the training conditions, and learning
from small amounts of experience. These challenges demand combinatorial generalization, and so it
is perhaps not surprising that an approach which eschews compositionality and explicit structure
struggles to meet them.
When deep learning's connectionist (Rumelhart et al., 1987) forebears were faced with analogous
critiques from structured, symbolic positions (Fodor and Pylyshyn, 1988; Pinker and Prince, 1988),


key: chroma:document
string_value:
critiques from structured, symbolic positions (Fodor and Pylyshyn, 1988; Pinker and Prince, 1988),
there was a constructive eort (Bobrow and Hinton, 1990; Marcus, 2001) to address the challenges
directly and carefully. A variety of innovative sub-symbolic approaches for representing and reasoning
about structured objects were developed in domains such as analogy-making, linguistic analysis,
symbol manipulation, and other forms of relational reasoning (Smolensky, 1990; Hinton, 1990;
Pollack, 1990; Elman, 1991; Plate, 1995; Eliasmith, 2013), as well as more integrative theories for
how the mind works (Marcus, 2001). Such work also helped cultivate more recent deep learning
advances which use distributed, vector representations to capture rich semantic content in text
(Mikolov et al., 2013; Pennington et al., 2014), graphs (Narayanan et al., 2016, 2017), algebraic and
logical expressions (Allamanis et al., 2017; Evans et al., 2018), and programs (Devlin et al., 2017;
Chen et al., 2018b).
We suggest that a key path forward for modern AI is to commit to combinatorial generalization
as a top priority, and we advocate for integrative approaches to realize this goal. Just as biology does


key: chroma:document
string_value:
as a top priority, and we advocate for integrative approaches to realize this goal. Just as biology does
not choose between nature versus nurture|it uses nature and nurture jointly , to build wholes which
are greater than the sums of their parts|we, too, reject the notion that structure and 
exibility are
somehow at odds or incompatible, and embrace both with the aim of reaping their complementary
strengths. In the spirit of numerous recent examples of principled hybrids of structure-based methods
and deep learning (e.g., Reed and De Freitas, 2016; Garnelo et al., 2016; Ritchie et al., 2016; Wu
et al., 2017; Denil et al., 2017; Hudson and Manning, 2018), we see great promise in synthesizing
new techniques by drawing on the full AI toolkit and marrying the best approaches from today
with those which were essential during times when data and computation were at a premium.
Recently, a class of models has arisen at the intersection of deep learning and structured
approaches, which focuses on approaches for reasoning about explicitly structured data, in particular
graphs (e.g. Scarselli et al., 2009b; Bronstein et al., 2017; Gilmer et al., 2017; Wang et al., 2018c; Li
et al., 2018; Kipf et al., 2018; Gulcehre et al., 2018). What these approaches all have in common


key: chroma:document
string_value:
et al., 2018; Kipf et al., 2018; Gulcehre et al., 2018). What these approaches all have in common
is a capacity for performing computation over discrete entities and the relations between them.
What sets them apart from classical approaches is how the representations and structure of the
entities and relations|and the corresponding computations|can be learned, relieving the burden
of needing to specify them in advance. Crucially, these methods carry strong relational inductive
biases , in the form of specic architectural assumptions, which guide these approaches towards
learning about entities and relations (Mitchell, 1980), which we, joining many others (Spelke et al.,
1992; Spelke and Kinzler, 2007; Marcus, 2001; Tenenbaum et al., 2011; Lake et al., 2017; Lake and
Baroni, 2018; Marcus, 2018b), suggest are an essential ingredient for human-like intelligence.
3
Box 1: Relational reasoning
We dene structure as the product of composing a set of known building blocks. \Structured
representations" capture this composition (i.e., the arrangement of the elements) and \structured
computations" operate over the elements and their composition as a whole. Relational reasoning,
then, involves manipulating structured representations of entities and relations , using rules


key: chroma:document
string_value:
computations" operate over the elements and their composition as a whole. Relational reasoning,
then, involves manipulating structured representations of entities and relations , using rules
for how they can be composed. We use these terms to capture notions from cognitive science,
theoretical computer science, and AI, as follows:
Anentity is an element with attributes, such as a physical object with a size and mass.
Arelation is a property between entities. Relations between two objects might include
same size as ,heavier than , and distance from . Relations can have attributes as
well. The relation more than Xtimes heavier than takes an attribute, X, which
determines the relative weight threshold for the relation to be true vs.false . Relations
can also be sensitive to the global context. For a stone and a feather, the relation falls
with greater acceleration than depends on whether the context is in air vs.in a
vacuum . Here we focus on pairwise relations between entities.
Aruleis a function (like a non-binary logical predicate) that maps entities and relations
to other entities and relations, such as a scale comparison like is entity X large? and
is entity X heavier than entity Y? . Here we consider rules which take one or two
arguments (unary and binary), and return a unary property value.


key: chroma:document
string_value:
is entity X heavier than entity Y? . Here we consider rules which take one or two
arguments (unary and binary), and return a unary property value.
As an illustrative example of relational reasoning in machine learning, graphical models (Pearl,
1988; Koller and Friedman, 2009) can represent complex joint distributions by making explicit
random conditional independences among random variables. Such models have been very
successful because they capture the sparse structure which underlies many real-world generative
processes and because they support ecient algorithms for learning and reasoning. For example,
hidden Markov models constrain latent states to be conditionally independent of others given
the state at the previous time step, and observations to be conditionally independent given the
latent state at the current time step, which are well-matched to the relational structure of many
real-world causal processes. Explicitly expressing the sparse dependencies among variables
provides for various ecient inference and reasoning algorithms, such as message-passing, which
apply a common information propagation procedure across localities within a graphical model,
resulting in a composable, and partially parallelizable, reasoning procedure which can be applied
to graphical models of dierent sizes and shape.
In the remainder of the paper, we examine various deep learning methods through the lens of


key: chroma:document
string_value:
resulting in a composable, and partially parallelizable, reasoning procedure which can be applied
to graphical models of dierent sizes and shape.
In the remainder of the paper, we examine various deep learning methods through the lens of
their relational inductive biases, showing that existing methods often carry relational assumptions


key: chroma:document
string_value:
In the remainder of the paper, we examine various deep learning methods through the lens of
their relational inductive biases, showing that existing methods often carry relational assumptions
which are not always explicit or immediately evident. We then present a general framework for
entity- and relation-based reasoning|which we term graph networks |for unifying and extending
existing methods which operate on graphs, and describe key design principles for building powerful
architectures using graph networks as building blocks. We have also released an open-source library
for building graph networks, which can be found here: github.com/deepmind/graph nets .
2 Relational inductive biases
Many approaches in machine learning and AI which have a capacity for relational reasoning
4
Box 2: Inductive biases
Learning is the process of apprehending useful knowledge by observing and interacting with the
world. It involves searching a space of solutions for one expected to provide a better explanation
of the data or to achieve higher rewards. But in many cases, there are multiple solutions which
are equally good (Goodman, 1955). An inductive bias allows a learning algorithm to prioritize
one solution (or interpretation) over another, independent of the observed data (Mitchell,
1980). In a Bayesian model, inductive biases are typically expressed through the choice and
parameterization of the prior distribution (Griths et al., 2010). In other contexts, an inductive


key: chroma:document
string_value:
parameterization of the prior distribution (Griths et al., 2010). In other contexts, an inductive
bias might be a regularization term (McClelland, 1994) added to avoid overtting, or it might
be encoded in the architecture of the algorithm itself. Inductive biases often trade 
exibility
for improved sample complexity and can be understood in terms of the bias-variance tradeo
(Geman et al., 1992). Ideally, inductive biases both improve the search for solutions without
substantially diminishing performance, as well as help nd solutions which generalize in a
desirable way; however, mismatched inductive biases can also lead to suboptimal performance
by introducing constraints that are too strong.
Inductive biases can express assumptions about either the data-generating process or the space
of solutions. For example, when tting a 1D function to data, linear least squares follows
the constraint that the approximating function be a linear model, and approximation errors
should be minimal under a quadratic penalty. This re
ects an assumption that the datagenerating process can be explained simply, as a line process corrupted by additive Gaussian
noise. Similarly, L2 regularization prioritizes solutions whose parameters have small values,
and can induce unique solutions and global structure to otherwise ill-posed problems. This can


key: chroma:document
string_value:
noise. Similarly, L2 regularization prioritizes solutions whose parameters have small values,
and can induce unique solutions and global structure to otherwise ill-posed problems. This can
be interpreted as an assumption about the learning process: that searching for good solutions
is easier when there is less ambiguity among solutions. Note, these assumptions need not be
explicit|they re
ect interpretations of how a model or algorithm interfaces with the world.
(Box 1) use a relational inductive bias . While not a precise, formal denition, we use this term to
refer generally to inductive biases (Box 2) which impose constraints on relationships and interactions
among entities in a learning process.
Creative new machine learning architectures have rapidly proliferated in recent years, with
(perhaps not surprisingly given the thesis of this paper) practitioners often following a design pattern
of composing elementary building blocks to form more complex, deep2computational hierarchies and
graphs3. Building blocks such as \fully connected" layers are stacked into \multilayer perceptrons"
(MLPs), \convolutional layers" are stacked into \convolutional neural networks" (CNNs), and a
standard recipe for an image processing network is, generally, some variety of CNN composed with
a MLP. This composition of layers provides a particular type of relational inductive bias|that


key: chroma:document
string_value:
standard recipe for an image processing network is, generally, some variety of CNN composed with
a MLP. This composition of layers provides a particular type of relational inductive bias|that
of hierarchical processing|in which computations are performed in stages, typically resulting in
increasingly long range interactions among information in the input signal. As we explore below, the
building blocks themselves also carry various relational inductive biases (Table 1). Though beyond
the scope of this paper, various non-relational inductive biases are used in deep learning as well: for
example, activation non-linearities, weight decay, dropout (Srivastava et al., 2014), batch and layer
normalization (Ioe and Szegedy, 2015; Ba et al., 2016), data augmentation, training curricula, and
optimization algorithms all impose constraints on the trajectory and outcome of learning.
2This pattern of composition in depth is ubiquitous in deep learning, and is where the \deep" comes from.
3Recent methods (Liu et al., 2018) even automate architecture construction via learned graph editing procedures.
5
Component Entities Relations Rel. inductive bias Invariance
Fully connected Units All-to-all Weak Convolutional Grid elements Local Locality Spatial translation
Recurrent Timesteps Sequential Sequentiality Time translation
Graph network Nodes Edges Arbitrary Node, edge permutations


key: chroma:document
string_value:
Recurrent Timesteps Sequential Sequentiality Time translation
Graph network Nodes Edges Arbitrary Node, edge permutations
Table 1: Various relational inductive biases in standard deep learning components. See also Section 2.
To explore the relational inductive biases expressed within various deep learning methods, we
must identify several key ingredients, analogous to those in Box 1: what are the entities , what are
therelations , and what are the rules for composing entities and relations, and computing their
implications? In deep learning, the entities and relations are typically expressed as distributed
representations, and the rules as neural network function approximators; however, the precise forms
of the entities, relations, and rules vary between architectures. To understand these dierences
between architectures, we can further ask how each supports relational reasoning by probing:
The arguments to the rule functions (e.g., which entities and relations are provided as input).
How the rule function is reused , orshared , across the computational graph (e.g., across dierent
entities and relations, across dierent time or processing steps, etc.).
How the architecture denes interactions versus isolation among representations (e.g., by
applying rules to draw conclusions about related entities, versus processing them separately).
2.1 Relational inductive biases in standard deep learning building blocks
2.1.1 Fully connected layers


key: chroma:document
string_value:
applying rules to draw conclusions about related entities, versus processing them separately).
2.1 Relational inductive biases in standard deep learning building blocks
2.1.1 Fully connected layers
Perhaps the most common building block is a fully connected layer (Rosenblatt, 1961). Typically
implemented as a non-linear vector-valued function of vector inputs, each element, or \unit", of
the output vector is the dot product between a weight vector, followed by an added bias term, and
nally a non-linearity such as a rectied linear unit (ReLU). As such, the entities are the units in
the network, the relations are all-to-all (all units in layer iare connected to all units in layer j),
and the rules are specied by the weights and biases. The argument to the rule is the full input
signal, there is no reuse, and there is no isolation of information (Figure 1a). The implicit relational
inductive bias in a fully connected layer is thus very weak: all input units can interact to determine
any output unit's value, independently across outputs (Table 1).
2.1.2 Convolutional layers
Another common building block is a convolutional layer (Fukushima, 1980; LeCun et al., 1989). It is


key: chroma:document
string_value:
2.1.2 Convolutional layers
Another common building block is a convolutional layer (Fukushima, 1980; LeCun et al., 1989). It is
implemented by convolving an input vector or tensor with a kernel of the same rank, adding a bias
term, and applying a point-wise non-linearity. The entities here are still individual units (or grid
elements, e.g. pixels), but the relations are sparser. The dierences between a fully connected layer
and a convolutional layer impose some important relational inductive biases: locality and translation
invariance (Figure 1b). Locality re
ects that the arguments to the relational rule are those entities in
close proximity with one another in the input signal's coordinate space, isolated from distal entities.
Translation invariance re
ects reuse of the same rule across localities in the input. These biases
are very eective for processing natural image data because there is high covariance within local
6
(a) Fully connected
Sharing in space (b) Convolutional
Sharing in time (c) Recurrent
Figure 1: Reuse and sharing in common deep learning building blocks. (a) Fully connected layer,
in which all weights are independent, and there is no sharing. (b) Convolutional layer, in which


key: chroma:document
string_value:
in which all weights are independent, and there is no sharing. (b) Convolutional layer, in which
a local kernel function is reused multiple times across the input. Shared weights are indicated by
arrows with the same color. (c) Recurrent layer, in which the same function is reused across dierent
processing steps.
neighborhoods, which diminishes with distance, and because the statistics are mostly stationary
across an image (Table 1).
2.1.3 Recurrent layers
A third common building block is a recurrent layer (Elman, 1990), which is implemented over a
sequence of steps. Here, we can view the inputs and hidden states at each processing step as the
entities, and the Markov dependence of one step's hidden state on the previous hidden state and
the current input, as the relations. The rule for combining the entities takes a step's inputs and
hidden state as arguments to update the hidden state. The rule is reused over each step (Figure 1c),
which re
ects the relational inductive bias of temporal invariance (similar to a CNN's translational
invariance in space). For example, the outcome of some physical sequence of events should not
depend on the time of day. RNNs also carry a bias for locality in the sequence via their Markovian
structure (Table 1).


key: chroma:document
string_value:
depend on the time of day. RNNs also carry a bias for locality in the sequence via their Markovian
structure (Table 1).
2.2 Computations over sets and graphs
While the standard deep learning toolkit contains methods with various forms of relational inductive
biases, there is no \default" deep learning component which operates on arbitrary relational structure.
We need models with explicit representations of entities and relations, and learning algorithms which
nd rules for computing their interactions, as well as ways of grounding them in data. Importantly,
entities in the world (such as objects and agents) do not have a natural order; rather, orderings
can be dened by the properties of their relations. For example, the relations between the sizes of
a set of objects can potentially be used to order them, as can their masses, ages, toxicities, and
prices. Invariance to ordering|except in the face of relations|is a property that should ideally be
re
ected by a deep learning component for relational reasoning.
Sets are a natural representation for systems which are described by entities whose order is
undened or irrelevant; in particular, their relational inductive bias does not come from the presence
of something, but rather from the absence . For illustration, consider the task of predicting the center
7
H H
O
The brown 
dog jumped. The


key: chroma:document
string_value:
of something, but rather from the absence . For illustration, consider the task of predicting the center
7
H H
O
The brown 
dog jumped. The 
brown dog jumped (a) (b) 
(c) (d) 
(e) (f) 
Molecule Mass-Spring System 
n-body System Rigid Body System 
Sentence and Parse Tree Image and Fully-Connected Scene Graph Figure 2: Dierent graph representations. (a) A molecule, in which each atom is represented as a
node and edges correspond to bonds (e.g. Duvenaud et al., 2015). (b) A mass-spring system, in
which the rope is dened by a sequence of masses which are represented as nodes in the graph (e.g.
Battaglia et al., 2016; Chang et al., 2017). (c) A n-body system, in which the bodies are nodes and
the underlying graph is fully connected (e.g. Battaglia et al., 2016; Chang et al., 2017). (d) A rigid
body system, in which the balls and walls are nodes, and the underlying graph denes interactions
between the balls and between the balls and the walls (e.g. Battaglia et al., 2016; Chang et al., 2017).


key: chroma:document
string_value:
between the balls and between the balls and the walls (e.g. Battaglia et al., 2016; Chang et al., 2017).
(e) A sentence, in which the words correspond to leaves in a tree, and the other nodes and edges
could be provided by a parser (e.g. Socher et al., 2013). Alternately, a fully connected graph could
be used (e.g. Vaswani et al., 2017). (f) An image, which can be decomposed into image patches
corresponding to nodes in a fully connected graph (e.g. Santoro et al., 2017; Wang et al., 2018c).
of mass of a solar system comprised of nplanets, whose attributes (e.g., mass, position, velocity,
etc.) are denoted by fx1;x2;:::;xng. For such a computation, the order in which we consider the
planets does not matter because the state can be described solely in terms of aggregated, averaged
quantities. However, if we were to use a MLP for this task, having learned the prediction for a
particular input ( x1;x2;:::;xn) would not necessarily transfer to making a prediction for the same


key: chroma:document
string_value:
particular input ( x1;x2;:::;xn) would not necessarily transfer to making a prediction for the same
inputs under a dierent ordering ( xn;x1;:::;x2). Since there are n! such possible permutations, in
the worst case, the MLP could consider each ordering as fundamentally dierent, and thus require
an exponential number of input/output training examples to learn an approximating function.
A natural way to handle such combinatorial explosion is to only allow the prediction to depend
on symmetric functions of the inputs' attributes. This might mean computing shared per-object
8
featuresff(x1);:::;f (xn)gwhich are then aggregated in a symmetric way (for example, by taking
their mean). Such an approach is the essence of the Deep Sets and related models (Zaheer et al.,
2017; Edwards and Storkey, 2016; Pevn y and Somol, 2017), which we explore further in Section 4.2.3.
Of course, permutation invariance is not the only important form of underlying structure in
many problems. For example, each object in a set may be aected by pairwise interactions with
the other objects in the set (Hartford et al., 2018). In our planets scenario, consider now the


key: chroma:document
string_value:
the other objects in the set (Hartford et al., 2018). In our planets scenario, consider now the
task of predicting each individual planet's position after a time interval,  t. In this case, using
aggregated, averaged information is not enough because the movement of each planet depends on
the forces the other planets are exerting on it. Instead, we could compute the state of each object
asx0
i=f(xi;P
jg(xi;xj)), wheregcould compute the force induced by the j-th planet on the
i-th planet, and fcould compute the future state of the i-th planet which results from the forces
and dynamics. The fact that we use the same geverywhere is again a consequence of the global
permutation invariance of the system; however, it also supports a dierent relational structure
becausegnow takes two arguments rather than one.4
The above solar system examples illustrate two relational structures: one in which there are
no relations, and one which consists of all pairwise relations. Many real-world systems (such as
in Figure 2) have a relational structure somewhere in between these two extremes, however, with
some pairs of entities possessing a relation and others lacking one. In our solar system example, if
the system instead consists of the planets and their moons, one may be tempted to approximate


key: chroma:document
string_value:
some pairs of entities possessing a relation and others lacking one. In our solar system example, if
the system instead consists of the planets and their moons, one may be tempted to approximate
it by neglecting the interactions between moons of dierent planets. In practice, this means
computing interactions only between some pairs of objects, i.e. x0
i=f(xi;P
j2(i)g(xi;xj)), where
(i)f1;:::;ngis a neighborhood around node i. This corresponds to a graph, in that the i-th
object only interacts with a subset of the other objects, described by its neighborhood. Note, the
updated states still do not depend in the order in which we describe the neighborhood.5
Graphs, generally, are a representation which supports arbitrary (pairwise) relational structure, and computations over graphs aord a strong relational inductive bias beyond that which
convolutional and recurrent layers can provide.
3 Graph networks
Neural networks that operate on graphs, and structure their computations accordingly, have been
developed and explored extensively for more than a decade under the umbrella of \graph neural
networks" (Gori et al., 2005; Scarselli et al., 2005, 2009a; Li et al., 2016), but have grown rapidly


key: chroma:document
string_value:
networks" (Gori et al., 2005; Scarselli et al., 2005, 2009a; Li et al., 2016), but have grown rapidly
in scope and popularity in recent years. We survey the literature on these methods in the next
sub-section (3.1). Then in the remaining sub-sections, we present our graph networks framework,
which generalizes and extends several lines of work in this area.
3.1 Background
Models in the graph neural network family (Gori et al., 2005; Scarselli et al., 2005, 2009a; Li et al.,
2016) have been explored in a diverse range of problem domains, across supervised, semi-supervised,
unsupervised, and reinforcement learning settings. They have been eective at tasks thought to
have rich relational structure, such as visual scene understanding tasks (Raposo et al., 2017; Santoro
4We could extend this same analysis to increasingly entangled structures that depend on relations among triplets
(i.e.,g(xi;xj;xk)), quartets, and so on. We note that if we restrict these functions to only operate on subsets of xi
which are spatially close, then we end back up with something resembling CNNs. In the most entangled sense, where


key: chroma:document
string_value:
which are spatially close, then we end back up with something resembling CNNs. In the most entangled sense, where
there is a single relation function g(x1; : : : ;xn), we end back up with a construction similar to a fully connected layer.
5The invariance which this model enforces is the invariance under isomorphism of the graph.
9
et al., 2017) and few-shot learning (Garcia and Bruna, 2018). They have also been used to learn
the dynamics of physical systems (Battaglia et al., 2016; Chang et al., 2017; Watters et al., 2017;
van Steenkiste et al., 2018; Sanchez-Gonzalez et al., 2018) and multi-agent systems (Sukhbaatar
et al., 2016; Hoshen, 2017; Kipf et al., 2018), to reason about knowledge graphs (Bordes et al., 2013;
O~ noro-Rubio et al., 2017; Hamaguchi et al., 2017), to predict the chemical properties of molecules
(Duvenaud et al., 2015; Gilmer et al., 2017), to predict trac on roads (Li et al., 2017; Cui et al.,
2018), to classify and segment images and videos (Wang et al., 2018c; Hu et al., 2017) and 3D


key: chroma:document
string_value:
2018), to classify and segment images and videos (Wang et al., 2018c; Hu et al., 2017) and 3D
meshes and point clouds (Wang et al., 2018d), to classify regions in images (Chen et al., 2018a), to
perform semi-supervised text classication (Kipf and Welling, 2017), and in machine translation
(Vaswani et al., 2017; Shaw et al., 2018; Gulcehre et al., 2018). They have been used within both
model-free (Wang et al., 2018b) and model-based (Hamrick et al., 2017; Pascanu et al., 2017;
Sanchez-Gonzalez et al., 2018) continuous control, for model-free reinforcement learning (Hamrick
et al., 2018; Zambaldi et al., 2018), and for more classical approaches to planning (Toyer et al.,
2017).
Many traditional computer science problems, which involve reasoning about discrete entities and
structure, have also been explored with graph neural networks, such as combinatorial optimization
(Bello et al., 2016; Nowak et al., 2017; Dai et al., 2017), boolean satisability (Selsam et al., 2018),
program representation and verication (Allamanis et al., 2018; Li et al., 2016), modeling cellular


key: chroma:document
string_value:
program representation and verication (Allamanis et al., 2018; Li et al., 2016), modeling cellular
automata and Turing machines (Johnson, 2017), and performing inference in graphical models
(Yoon et al., 2018). Recent work has also focused on building generative models of graphs (Li et al.,
2018; De Cao and Kipf, 2018; You et al., 2018; Bojchevski et al., 2018), and unsupervised learning of
graph embeddings (Perozzi et al., 2014; Tang et al., 2015; Grover and Leskovec, 2016; Garc a-Dur an
and Niepert, 2017).
The works cited above are by no means an exhaustive list, but provide a representative crosssection of the breadth of domains for which graph neural networks have proven useful. We point
interested readers to a number of existing reviews which examine the body of work on graph neural
networks in more depth. In particular, Scarselli et al. (2009a) provides an authoritative overview of
early graph neural network approaches. Bronstein et al. (2017) provides an excellent survey of deep
learning on non-Euclidean data, and explores graph neural nets, graph convolution networks, and
related spectral approaches. Recently, Gilmer et al. (2017) introduced the message-passing neural


key: chroma:document
string_value:
related spectral approaches. Recently, Gilmer et al. (2017) introduced the message-passing neural
network (MPNN), which unied various graph neural network and graph convolutional network
approaches (Monti et al., 2017; Bruna et al., 2014; Hena et al., 2015; Deerrard et al., 2016;
Niepert et al., 2016; Kipf and Welling, 2017; Bronstein et al., 2017) by analogy to message-passing
in graphical models. In a similar vein, Wang et al. (2018c) introduced the non-local neural network
(NLNN), which unied various \self-attention"-style methods (Vaswani et al., 2017; Hoshen, 2017;
Veli ckovi c et al., 2018) by analogy to methods from computer vision and graphical models for
capturing long range dependencies in signals.
3.2 Graph network (GN) block
We now present our graph networks (GN) framework, which denes a class of functions for
relational reasoning over graph-structured representations. Our GN framework generalizes and
extends various graph neural network, MPNN, and NLNN approaches (Scarselli et al., 2009a; Gilmer
et al., 2017; Wang et al., 2018c), and supports constructing complex architectures from simple


key: chroma:document
string_value:
et al., 2017; Wang et al., 2018c), and supports constructing complex architectures from simple
building blocks. Note, we avoided using the term \neural" in the \graph network" label to re
ect
that they can be implemented with functions other than neural networks, though here our focus is
on neural network implementations.
The main unit of computation in the GN framework is the GN block , a \graph-to-graph" module
10
Box 3: Our denition of \graph"
Attributesviek


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:
Here we use \graph" to mean a directed, attributed multi-graph with a global attribute. In our
terminology, a node is denoted as vi, an edge as ek, and the global attributes as u. We also use
skandrkto indicate the indices of the sender and receiver nodes (see below), respectively, for
edgek. To be more precise, we dene these terms as:
Directed : one-way edges, from a \sender" node to a \receiver" node.
Attribute : properties that can be encoded as a vector, set, or even another graph.
Attributed : edges and vertices have attributes associated with them.
Global attribute : a graph-level attribute.
Multi-graph : there can be more than one edge between vertices, including self-edges.
Figure 2 shows a variety of dierent types of graphs corresponding to real data that we may be
interested in modeling, including physical systems, molecules, images, and text.
which takes a graph as input, performs computations over the structure, and returns a graph as
output. As described in Box 3, entities are represented by the graph's nodes , relations by the edges ,
and system-level properties by global attributes. The GN framework's block organization emphasizes
customizability and synthesizing new architectures which express desired relational inductive biases.


key: chroma:document
string_value:
and system-level properties by global attributes. The GN framework's block organization emphasizes
customizability and synthesizing new architectures which express desired relational inductive biases.
The key design principles are: Flexible representations (see Section 4.1); Congurable within-block
structure (see Section 4.2); and Composable multi-block architectures (see Section 4.3).
We introduce a motivating example to help make the GN formalism more concrete. Consider
predicting the movements a set of rubber balls in an arbitrary gravitational eld, which, instead of
bouncing against one another, each have one or more springs which connect them to some (or all) of
the others. We will refer to this running example throughout the denitions below, to motivate the
graph representation and the computations operating over it. Figure 2 depicts some other common
scenarios that can be represented by graphs and reasoned over using graph networks.
3.2.1 Denition of \graph"
Within our GN framework, a graph is dened as a 3-tuple G=(u;V;E )(see Box 3 for details of
graph representations). The uis a global attribute; for example, umight represent the gravitational


key: chroma:document
string_value:
graph representations). The uis a global attribute; for example, umight represent the gravitational
eld. TheV=fvigi=1:Nvis the set of nodes (of cardinality Nv), where each viis a node's attribute.
For example, Vmight represent each ball, with attributes for position, velocity, and mass. The
E=f(ek;rk;sk)gk=1:Neis the set of edges (of cardinality Ne), where each ekis the edge's attribute,
rkis the index of the receiver node, and skis the index of the sender node. For example, Emight
represent the presence of springs between dierent balls, and their corresponding spring constants.
11
Algorithm 1 Steps of computation in a full GN block.
function GraphNetwork (E,V,u)
fork2f1:::Negdo
e0
k e(ek;vrk;vsk;u) .1. Compute updated edge attributes
end for
fori2f1:::Nngdo
letE0
i=f(e0
k;rk;sk)grk=i;k=1:Ne
 e0
i e!v(E0
i) .2. Aggregate edge attributes per node
v0


key: chroma:document
string_value:
 e0
i e!v(E0
i) .2. Aggregate edge attributes per node
v0
i v( e0
i;vi;u) .3. Compute updated node attributes
end for
letV0=fv0gi=1:Nv
letE0=f(e0
k;rk;sk)gk=1:Ne
 e0 e!u(E0) .4. Aggregate edge attributes globally
 v0 v!u(V0) .5. Aggregate node attributes globally
u0 u( e0; v0;u) .6. Compute updated global attribute
return (E0;V0;u0)
end function
3.2.2 Internal structure of a GN block
A GN block contains three \update" functions, , and three \aggregation" functions, ,
e0
k=e(ek;vrk;vsk;u)
v0
i=v
 e0
i;vi;u
u0=u
 e0; v0;u e0
i=e!v
E0
i
 e0=e!u
E0


key: chroma:document
string_value:
i=e!v
E0
i
 e0=e!u
E0
 v0=v!u
V0(1)
whereE0
i=f(e0
k;rk;sk)grk=i;k=1:Ne,V0=fv0
igi=1:Nv, andE0=S
iE0
i=f(e0
k;rk;sk)gk=1:Ne.
Theeis mapped across all edges to compute per-edge updates, the vis mapped across all
nodes to compute per-node updates, and the uis applied once as the global update. The 
functions each take a set as input, and reduce it to a single element which represents the aggregated
information. Crucially, the functions must be invariant to permutations of their inputs, and should
take variable numbers of arguments (e.g., elementwise summation, mean, maximum, etc.).
3.2.3 Computational steps within a GN block
When a graph, G, is provided as input to a GN block, the computations proceed from the edge, to
the node, to the global level. Figure 3 shows a depiction of which graph elements are involved in


key: chroma:document
string_value:
the node, to the global level. Figure 3 shows a depiction of which graph elements are involved in
each of these computations, and Figure 4a shows a full GN block, with its update and aggregation
functions. Algorithm 1 shows the following steps of computation:
1.eis applied per edge, with arguments ( ek;vrk;vsk;u), and returns e0
k. In our springs example,
this might correspond to the forces or potential energies between two connected balls. The
set of resulting per-edge outputs for each node, i, is,E0
i=f(e0
k;rk;sk)grk=i;k=1:Ne. And
E0=S
iE0
i=f(e0
k;rk;sk)gk=1:Neis the set of all per-edge outputs.
2.e!vis applied to E0
i, and aggregates the edge updates for edges that project to vertex i, into
 e0
i, which will be used in the next step's node update. In our running example, this might
correspond to summing all the forces or potential energies acting on the ithball.
12
vi


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:
Figure 3: Updates in a GN block. Blue indicates the element that is being updated, and black
indicates other elements which are involved in the update (note that the pre-update value of the
blue element is also used in the update). See Equation 1 for details on the notation.
3.vis applied to each node i, to compute an updated node attribute, v0
i. In our running
example,vmay compute something analogous to the updated position, velocity, and kinetic
energy of each ball. The set of resulting per-node outputs is, V0=fv0
igi=1:Nv.
4.e!uis applied to E0, and aggregates all edge updates, into  e0, which will then be used in the
next step's global update. In our running example, e!umay compute the summed forces
(which should be zero, in this case, due to Newton's third law) and the springs' potential
energies.
5.v!uis applied to V0, and aggregates all node updates, into  v0, which will then be used in
the next step's global update. In our running example, v!umight compute the total kinetic
energy of the system.


key: chroma:document
string_value:
the next step's global update. In our running example, v!umight compute the total kinetic
energy of the system.
6.uis applied once per graph, and computes an update for the global attribute, u0. In our
running example, umight compute something analogous to the net forces and total energy
of the physical system.
Note, though we assume this sequence of steps here, the order is not strictly enforced: it is possible
to reverse the update functions to proceed from global, to per-node, to per-edge updates, for example.
Kearnes et al. (2016) computes edge updates from nodes in a similar manner.
3.2.4 Relational inductive biases in graph networks
Our GN framework imposes several strong relational inductive biases when used as components in
a learning process. First, graphs can express arbitrary relationships among entities, which means
the GN's input determines how representations interact and are isolated, rather than those choices
being determined by the xed architecture. For example, the assumption that two entities have a
relationship|and thus should interact|is expressed by an edge between the entities' corresponding
nodes. Similarly, the absence of an edge expresses the assumption that the nodes have no relationship
and should not in
uence each other directly.


key: chroma:document
string_value:
nodes. Similarly, the absence of an edge expresses the assumption that the nodes have no relationship
and should not in
uence each other directly.
Second, graphs represent entities and their relations as sets, which are invariant to permutations.
This means GNs are invariant to the order of these elements6, which is often desirable. For example,
the objects in a scene do not have a natural ordering (see Sec. 2.2).
Third, a GN's per-edge and per-node functions are reused across all edges and nodes, respectively.
This means GNs automatically support a form of combinatorial generalization (see Section 5.1):
because graphs are composed of edges, nodes, and global features, a single GN can operate on
graphs of dierent sizes (numbers of edges and nodes) and shapes (edge connectivity).
6Note, an ordering can be imposed by encoding the indices in the node or edge attributes, or via the edges
themselves (e.g. by encoding a chain or partial ordering).
13
4 Design principles for graph network architectures
The GN framework can be used to implement a wide variety of architectures, in accordance with
the design principles listed above in Section 3.2, which also correspond to the sub-sections (4.1,


key: chroma:document
string_value:
the design principles listed above in Section 3.2, which also correspond to the sub-sections (4.1,
4.2, and 4.3) below. In general, the framework is agnostic to specic attribute representations and
functional forms. Here, however, we focus mainly on deep learning architectures, which allow GNs
to act as learnable graph-to-graph function approximators.
4.1 Flexible representations
Graph networks support highly 
exible graph representations in two ways: rst, in terms of the
representation of the attributes; and second, in terms of the structure of the graph itself.
4.1.1 Attributes
The global, node, and edge attributes of a GN block can use arbitrary representational formats. In
deep learning implementations, real-valued vectors and tensors are most common. However, other
data structures such as sequences, sets, or even graphs could also be used.
The requirements of the problem will often determine what representations should be used for
the attributes. For example, when the input data is an image, the attributes might be represented
as tensors of image patches; however, when the input data is a text document, the attributes might
be sequences of words corresponding to sentences.
For each GN block within a broader architecture, the edge and node outputs typically correspond


key: chroma:document
string_value:
be sequences of words corresponding to sentences.
For each GN block within a broader architecture, the edge and node outputs typically correspond
to lists of vectors or tensors, one per edge or node, and the global outputs correspond to a single
vector or tensor. This allows a GN's output to be passed to other deep learning building blocks
such as MLPs, CNNs, and RNNs.
The output of a GN block can also be tailored to the demands of the task. In particular,
Anedge-focused GN uses the edges as output, for example to make decisions about interactions
among entities (Kipf et al., 2018; Hamrick et al., 2018).
Anode-focused GN uses the nodes as output, for example to reason about physical systems
(Battaglia et al., 2016; Chang et al., 2017; Wang et al., 2018b; Sanchez-Gonzalez et al., 2018).
Agraph-focused GN uses the globals as output, for example to predict the potential energy of
a physical system (Battaglia et al., 2016), the properties of a molecule (Gilmer et al., 2017),
or answers to questions about a visual scene (Santoro et al., 2017).
The nodes, edges, and global outputs can also be mixed-and-matched depending on the task. For


key: chroma:document
string_value:
or answers to questions about a visual scene (Santoro et al., 2017).
The nodes, edges, and global outputs can also be mixed-and-matched depending on the task. For
example, Hamrick et al. (2018) used both the output edge and global attributes to compute a policy
over actions.
4.1.2 Graph structure
When dening how the input data will be represented as a graph, there are generally two scenarios:
rst, the input explicitly species the relational structure; and second, the relational structure must
be inferred or assumed. These are not hard distinctions, but extremes along a continuum.
Examples of data with more explicitly specied entities and relations include knowledge graphs,
social networks, parse trees, optimization problems, chemical graphs, road networks, and physical
systems with known interactions. Figures 2a-d illustrate how such data can be expressed as graphs.
Examples of data where the relational structure is not made explicit, and must be inferred or
assumed, include visual scenes, text corpora, programming language source code, and multi-agent
14
systems. In these types of settings, the data may be formatted as a set of entities without relations,
or even just a vector or tensor (e.g., an image). If the entities are not specied explicitly, they might


key: chroma:document
string_value:
or even just a vector or tensor (e.g., an image). If the entities are not specied explicitly, they might
be assumed, for instance, by treating each word in a sentence (Vaswani et al., 2017) or each local
feature vector in a CNN's output feature map, as a node (Watters et al., 2017; Santoro et al., 2017;
Wang et al., 2018c) (Figures 2e-f). Or, it might be possible to use a separate learned mechanism to
infer entities from an unstructured signal (Luong et al., 2015; Mnih et al., 2014; Eslami et al., 2016;
van Steenkiste et al., 2018). If relations are not available, the simplest approach is to instantiate all
possible directed edges between entities (Figure 2f). This can be prohibitive for large numbers of
entities, however, because the number of possible edges grows quadratically with the number of
nodes. Thus developing more sophisticated ways of inferring sparse structure from unstructured
data (Kipf et al., 2018) is an important future direction.
4.2 Congurable within-block structure
The structure and functions within a GN block can be congured in dierent ways, which oers


key: chroma:document
string_value:
4.2 Congurable within-block structure
The structure and functions within a GN block can be congured in dierent ways, which oers

exibility in what information is made available as inputs to its functions, as well as how output edge,
node, and global updates are produced. In particular, each in Equation 1 must be implemented
with some function, f, wheref's argument signature determines what information it requires as
input; in Figure 4, the incoming arrows to each depict whether u,V, andEare taken as inputs.
Hamrick et al. (2018) and Sanchez-Gonzalez et al. (2018) used the full GN block shown in Figure 4a.
Theirimplementations used neural networks (denoted NNe,NNv, and NNubelow, to indicate that
they are dierent functions with dierent parameters). Their implementations used elementwise
summation, but averages and max/min could also be used,
e(ek;vrk;vsk;u):=fe(ek;vrk;vsk;u) = NNe([ek;vrk;vsk;u]) (2)
v
 e0
i;vi;u:=fv
 e0
i;vi;u


key: chroma:document
string_value:
v
 e0
i;vi;u:=fv
 e0
i;vi;u
= NNv
[ e0
i;vi;u]
u
 e0; v0;u:=fu
 e0; v0;u
= NNu
[ e0; v0;u]
e!v
E0
i:= =X
fk:rk=ige0
k
v!u
V0:= =X
iv0
i
e!u
E0:= =X
ke0
k
where [ x;y;z] indicates vector/tensor concatenation. For vector attributes, a MLP is often used for
, while for tensors such as image feature maps, CNNs may be more suitable.
Thefunctions can also use RNNs, which requires an additional hidden state as input and
output. Figure 4b shows a very simple version of a GN block with RNNs as functions: there is no
message-passing in this formulation, and this type of block might be used for recurrent smoothing of


key: chroma:document
string_value:
message-passing in this formulation, and this type of block might be used for recurrent smoothing of
some dynamic graph states. Of course, RNNs as functions could also be used in a full GN block
(Figure 4a).
A variety of other architectures can be expressed in the GN framework, often as dierent
function choices and within-block congurations. The remaining sub-sections explore how a GN's
within-block structure can be congured in dierent ways, with examples of published work which
uses such congurations. See the Appendix for details.
15
(a) Full GN block
u0,u0hid


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:
Figure 4: Dierent internal GN block congurations. See Section 3.2 for details on the notation,
and Section 4 for details about each variant. (a) A full GN predicts node, edge, and global output
attributes based on incoming node, edge, and global attributes. (b) An independent, recurrent
update block takes input and hidden graphs, and the functions are RNNs (Sanchez-Gonzalez
et al., 2018). (c) An MPNN (Gilmer et al., 2017) predicts node, edge, and global output attributes
based on incoming node, edge, and global attributes. Note that the global prediction does not
include aggregated edges. (d) A NLNN (Wang et al., 2018c) only predicts node output attributes.
(e) A relation network (Raposo et al., 2017; Santoro et al., 2017) only uses the edge predictions
to predict global attributes. (f) A Deep Set (Zaheer et al., 2017) bypasses the edge update and
predicts updated global attributes.
4.2.1 Message-passing neural network (MPNN)
Gilmer et al. (2017)'s MPNN generalizes a number of previous architectures and can be translated
naturally into the GN formalism. Following the MPNN paper's terminology (see Gilmer et al.


key: chroma:document
string_value:
naturally into the GN formalism. Following the MPNN paper's terminology (see Gilmer et al.
(2017), pages 2-4):
the message function, Mt, plays the role of the GN's e, but does not take uas input,
elementwise summation is used for the GN's e!v,
the update function, Ut, plays the role of the GN's v,
16


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:
by theeande!vunder the GN framework. Typically, NLNNs assume that dierent regions of
an image (or words in a sentence) correspond to nodes in a fully connected graph, and the attention
mechanism denes a weighted sum over nodes during the aggregation step.
the readout function, R, plays the role of the GN's u, but does not take uorE0as input,
and thus an analog to the GN's e!uis not required;
dmaster serves a roughly similar purpose to the GN's u, but is dened as an extra node
connected to all others, and thus does not in
uence the edge and global updates directly. It
can then be represented in the GN's V.
Figure 4c shows how an MPNN is structured, according to the GN framework. For details and
various MPNN architectures, see the Appendix.
4.2.2 Non-local neural networks (NLNN)
Wang et al. (2018c)'s NLNN, which unies various \intra-/self-/vertex-/graph-attention" approaches
(Lin et al., 2017; Vaswani et al., 2017; Hoshen, 2017; Veli ckovi c et al., 2018; Shaw et al., 2018),


key: chroma:document
string_value:
(Lin et al., 2017; Vaswani et al., 2017; Hoshen, 2017; Veli ckovi c et al., 2018; Shaw et al., 2018),
can also be translated into the GN formalism. The label \attention" refers to how the nodes are
updated: each node update is based on a weighted sum of (some function of) the node attributes of
its neighbors, where the weight between a node and one of its neighbors is computed by a scalar
pairwise function between their attributes (and then normalized across neighbors). The published
NLNN formalism does not explicitly include edges, and instead computes pairwise attention weights
between all nodes. But various NLNN-compliant models, such as the vertex attention interaction
network (Hoshen, 2017) and graph attention network (Veli ckovi c et al., 2018), are able to handle
explicit edges by eectively setting to zero the weights between nodes which do not share an edge.
As Figures 4d and 5 illustrate, the eis factored into the scalar pairwise-interaction function
which returns the unnormalized attention term, denoted e(vrk;vsk)=a0
k, and a vector-valued
non-pairwise term, denoted e(vsk)=b0


key: chroma:document
string_value:
k, and a vector-valued
non-pairwise term, denoted e(vsk)=b0
k. In thee!vaggregation, the a0
kterms are normalized
across each receiver's edges, b0
k, and elementwise summed:
e(ek;vrk;vsk;u):=fe(vrk;vsk) = ( e(vrk;vsk); e(vsk)) = (a0
k;b0
k) =e0
k
ve0
i;vi;u:=fv(e0
i)
e!v
E0
i:=1P
fk:rk=iga0
kX
fk:rk=iga0
kb0
k
In the NLNN paper's terminology (see Wang et al. (2018c), pages 2-4):
theirfplays the role of the above ,
17
theirgplays the role of the above .
This formulation may be helpful for focusing only on those interactions which are most relevant for
the downstream task, especially when the input entities were a set, from which a graph was formed
by adding all possible edges between them.


key: chroma:document
string_value:
the downstream task, especially when the input entities were a set, from which a graph was formed
by adding all possible edges between them.
Vaswani et al. (2017)'s multi-headed self-attention mechanism adds an interesting feature, where
theeande!vare implemented by a parallel set of functions, whose results are concatenated
together as the nal step of e!v. This can be interpreted as using typed edges, where the dierent
types index into dierent ecomponent functions, analogous to Li et al. (2016).
For details and various NLNN architectures, see the Appendix.
4.2.3 Other graph network variants
The full GN (Equation 2) can be used to predict a full graph, or any subset of (u0;V0;E0), as
outlined in Section 4.1.1. For example, to predict a global property of a graph, V0andE0can just
be ignored. Similarly, if global, node, or edge attributes are unspecied in the inputs, those vectors
can be zero-length, i.e., not taken as explicit input arguments. The same idea applies for other GN
variants which do not use the full set of mapping ( ) and reduction ( ) functions. For instance,


key: chroma:document
string_value:
variants which do not use the full set of mapping ( ) and reduction ( ) functions. For instance,
Interaction Networks (Battaglia et al., 2016; Watters et al., 2017) and the Neural Physics Engine
(Chang et al., 2017) use a full GN but for the absence of the global to update the edge properties
(see Appendix for details).
Various models, including CommNet (Sukhbaatar et al., 2016), structure2vec (Dai et al., 2016)
(in the version of (Dai et al., 2017)), and Gated Graph Sequence Neural Networks (Li et al., 2016)
have used a ewhich does not directly compute pairwise interactions, but instead ignore the receiver
node, operating only on the sender node and in some cases an edge attribute. This can be expressed
by implementations of ewith the following signatures, such as:
e(ek;vrk;vsk;u):=fe(vsk)
ore(ek;vrk;vsk;u):=vsk+fe(ek)
ore(ek;vrk;vsk;u):=fe(ek;vsk):
See the Appendix for further details.
Relation Networks (Raposo et al., 2017; Santoro et al., 2017) bypass the node update entirely


key: chroma:document
string_value:
See the Appendix for further details.
Relation Networks (Raposo et al., 2017; Santoro et al., 2017) bypass the node update entirely
and predict the global output from pooled edge information directly (see also Figure 4e),
e(ek;vrk;vsk;u):=fe(vrk;vsk) = NNe([vrk;vsk])
u
 e0; v0;u:=fu
 e0
= NNu
 e0
e!u
E0:= =X
ke0
k
Deep Sets (Zaheer et al., 2017) bypass the edges update completely and predict the global output
from pooled nodes information directly (Figure 4f),
v(ei;vi;u):=fv(vi;u) = NNv([vi;u])
u
 e0; v0;u:=fu
 v0
= NNu
 v0
v!u
V0:= =X
iv0
i
PointNet (Qi et al., 2017) use similar update rule, with a max-aggregation for v!uand a two-step
node update.
18
GM


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:
Figure 6: (a) An example composing multiple GN blocks in sequence to form a GN \core". Here,
the GN blocks can use shared weights, or they could be independent. (b) The encode-process-decode
architecture, which is a common choice for composing GN blocks (see Section 4.3). Here, a GN
encodes an input graph, which is then processed by a GN core. The output of the core is decoded
by a third GN block into an output graph, whose nodes, edges, and/or global attributes would be
used for task-specic purposes. (c) The encode-process-decode architecture applied in a sequential
setting in which the core is also unrolled over time (potentially using a GRU or LSTM architecture),
in addition to being repeated within each time step. Here, merged lines indicate concatenation, and
split lines indicate copying.
4.3 Composable multi-block architectures
A key design principle of graph networks is constructing complex architectures by composing GN
blocks. We dened a GN block as always taking a graph comprised of edge, node, and global
elements as input, and returning a graph with the same constituent elements as output (simply
passing through the input elements to the output when those elements are not explicitly updated).


key: chroma:document
string_value:
elements as input, and returning a graph with the same constituent elements as output (simply
passing through the input elements to the output when those elements are not explicitly updated).
This graph-to-graph input/output interface ensures that the output of one GN block can be passed
as input to another, even if their internal congurations are dierent, similar to the tensor-to-tensor
interface of the standard deep learning toolkit. In the most basic form, two GN blocks, GN1and
GN2, can be composed as GN1GN2by passing the output of the rst as input to the second:
G0= GN 2(GN 1(G)).
Arbitrary numbers of GN blocks can be composed, as show in Figure 6a. The blocks can
be unshared (dierent functions and/or parameters, analogous to layers of a CNN), GN16=
GN26=6=GNM, or shared (reused functions and parameters, analogous to an unrolled RNN),
GN1=GN2==GNM. The white box around the GNcorein Figure 6a represents Mrepeated
internal processing sub-steps, with either shared or unshared GN blocks. Shared congurations
are analogous to message-passing (Gilmer et al., 2017), where the same local update procedure is


key: chroma:document
string_value:
are analogous to message-passing (Gilmer et al., 2017), where the same local update procedure is
applied iteratively to propagate information across the structure (Figure 7). If we exclude the global
u(which aggregates information from across the nodes and edges), the information that a node
has access to after msteps of propagation is determined by the set of nodes and edges that are at
mostmhops away. This can be interpreted as breaking down a complex computation into smaller
elementary steps. The steps can also be used to capture sequentiality in time. In our ball-spring
example, if each propagation step predicts the physical dynamics over one time step of duration  t,
then theMpropagation steps result in a total simulation time of, Mt.
A common architecture design is what we call the encode-process-decode conguration (Hamrick
et al. (2018); also see Figure 6ba): an input graph, Ginpis transformed into a latent representation,
G0, by an encoder, GNenc; a shared core block, GNcore, is applied Mtimes to return GM; and
nally an output graph, Gout, is decoded by GNdec. For example, in our running example, the
encoder might compute the initial forces and interaction energies between the balls, the core might
19
m=0


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:
the graph starting from a particular node. In the top row, the node of interest is in the upper
right; in the bottom row, the node of interest is in the bottom right. Shaded nodes indicate how far
information from the original node can travel in msteps of message passing; bolded edges indicate
which edges that information has the potential to travel across. Note that during the full message
passing procedure, this propagation of information happens simultaneously for all nodes and edges
in the graph (not just the two shown here).
apply an elementary dynamics update, and the decoder might read out the nal positions from the
updated graph state.
Similar to the encode-process-decode design, recurrent GN-based architectures can be built by
maintaining a hidden graph, Gt
hid, taking as input an observed graph, Gt
inp, and returning an output
graph,Gt
out, on each step (see Figure 6c). This type of architecture can be particularly useful for
predicting sequences of graphs, such as predicting the trajectory of a dynamical system over time
(e.g. Sanchez-Gonzalez et al., 2018). The encoded graph, output by GNenc, must have the same
structure as Gt
hid, and they can be easily combined by concatenating their corresponding ek,vi,


key: chroma:document
string_value:
structure as Gt
hid, and they can be easily combined by concatenating their corresponding ek,vi,
anduvectors (where the upward arrow merges into the left-hand horizontal arrow in Figure 6c),
before being passed to GNcore. For the output, the Gt
hidis copied (where the right-hand horizontal
arrow splits into the downward arrow in Figure 6c) and decoded by GNdec. This design reuses GN
blocks in several ways: GNenc,GNdec, and GNcoreare shared across each step, t; and within each
step, GN coremay perform multiple shared sub-steps.
Various other techniques for designing GN-based architectures can be useful. Graph skip
connections, for example, would concatenate a GN block's input graph, Gm, with its output graph,
Gm+1, before proceeding to further computations. Merging and smoothing input and hidden graph
information, as in Figure 6c, can use LSTM- or GRU-style gating schemes, instead of simple
concatenation (Li et al., 2016). Or distinct, recurrent GN blocks (e.g. Figure 4b) can be composed
before and/or after other GN blocks, to improve stability in the representations over multiple
propagation steps (Sanchez-Gonzalez et al., 2018).


key: chroma:document
string_value:
before and/or after other GN blocks, to improve stability in the representations over multiple
propagation steps (Sanchez-Gonzalez et al., 2018).
4.4 Implementing graph networks in code
Similar to CNNs (see Figure 1), which are naturally parallelizable (e.g. on GPUs), GNs have a
natural parallel structure: since the eandvfunctions in Equation 1 are shared over the edges
and nodes, respectively, they can be computed in parallel. In practice, this means that with respect
20
Box 4: Graph Nets open-source software library: github.com/deepmind/graph nets
We have released an open-source library for building GNs in Tensor
ow/Sonnet. It includes
demos of how to create, manipulate, and train GNs to reason about graph-structured data, on
a shortest path-nding task, a sorting task, and a physical prediction task. Each demo uses the
same GN architecture, which highlights the 
exibility of the approach.
Shortest path demo: tinyurl.com/gn-shortest-path-demo
This demo creates random graphs, and trains a GN to label the nodes and edges on the shortest
path between any two nodes. Over a sequence of message-passing steps (as depicted by each


key: chroma:document
string_value:
This demo creates random graphs, and trains a GN to label the nodes and edges on the shortest
path between any two nodes. Over a sequence of message-passing steps (as depicted by each
step's plot), the model renes its prediction of the shortest path.
Sort demo: tinyurl.com/gn-sort-demo
This demo creates lists of random numbers, and trains a GN to sort the list. After a sequence
of message-passing steps, the model makes an accurate prediction of which elements (columns
in the gure) come next after each other (rows).
Physics demo: tinyurl.com/gn-physics-demo
This demo creates random mass-spring physical systems, and trains a GN to predict the state of
the system on the next timestep. The model's next-step predictions can be fed back in as input
to create a rollout of a future trajectory. Each subplot below shows the true and predicted
mass-spring system states over 50 timesteps. This is similar to the model and experiments in
(Battaglia et al., 2016)'s \interaction networks".
21
toeandv, the nodes and edges can be treated like the batch dimension in typical mini-batch
training regimes. Moreover, several graphs can be naturally batched together by treating them as


key: chroma:document
string_value:
training regimes. Moreover, several graphs can be naturally batched together by treating them as
disjoint components of a larger graph. With some additional bookkeeping, this allows batching
together the computations made on several independent graphs.
Reusingeandvalso improves GNs' sample eciency. Again, analogous to a convolutional
kernel, the number of samples which are used to optimize a GN's eandvfunctions is the number
of edges and nodes, respectively, across all training graphs. For example, in the balls example
from Sec. 3.2, a scene with four balls which are all connected by springs will provide twelve (4 3)
examples of the contact interaction between them.
We have released an open-source software library for building GNs, which can be found here:
github.com/deepmind/graph nets . See Box 4 for an overview.
4.5 Summary
In this section, we have discussed the design principles behind graph networks: 
exible representations, congurable within-block structure, and composable multi-block architectures. These three
design principles combine in our framework which is extremely 
exible and applicable to a wide range
of domains ranging from perception, language, and symbolic reasoning. And, as we will see in the


key: chroma:document
string_value:
design principles combine in our framework which is extremely 
exible and applicable to a wide range
of domains ranging from perception, language, and symbolic reasoning. And, as we will see in the
remainder of this paper, the strong relational inductive bias possessed by graph networks supports
combinatorial generalization, thus making it a powerful tool both in terms of implementation and
theory.
5 Discussion
In this paper, we analyzed the extent to which relational inductive bias exists in deep learning
architectures like MLPs, CNNs, and RNNs, and concluded that while CNNs and RNNs do contain
relational inductive biases, they cannot naturally handle more structured representations such as
sets or graphs. We advocated for building stronger relational inductive biases into deep learning
architectures by highlighting an underused deep learning building block called a graph network ,
which performs computations over graph-structured data. Our graph network framework unies
existing approaches that also operate over graphs, and provides a straightforward interface for
assembling graph networks into complex, sophisticated architectures.
5.1 Combinatorial generalization in graph networks
The structure of GNs naturally supports combinatorial generalization because they do not perform
computations strictly at the system level, but also apply shared computations across the entities and


key: chroma:document
string_value:
The structure of GNs naturally supports combinatorial generalization because they do not perform
computations strictly at the system level, but also apply shared computations across the entities and
across the relations as well. This allows never-before-seen systems to be reasoned about, because
they are built from familiar components, in a way that re
ects von Humboldt's \innite use of nite
means" (Humboldt, 1836; Chomsky, 1965).
A number of studies have explored GNs' capacity for combinatorial generalization. Battaglia
et al. (2016) found that GNs trained to make one-step physical state predictions could simulate
thousands of future time steps, and also exhibit accurate zero-shot transfer to physical systems
with double, or half, the number of entities experienced during training. Sanchez-Gonzalez et al.
(2018) found similar results in more complex physical control settings, including that GNs trained as
forward models on simulated multi-joint agents could generalize to agents with new numbers of joints.
Hamrick et al. (2018) and Wang et al. (2018b) each found that GN-based decision-making policies
could transfer to novel numbers of entities as well. In combinatorial optimization problems, Bello
22


key: chroma:document
string_value:
could transfer to novel numbers of entities as well. In combinatorial optimization problems, Bello
22
et al. (2016); Nowak et al. (2017); Dai et al. (2017); Kool and Welling (2018) showed that GNs could
generalize well to problems of much dierent sizes than they had been trained on. Similarly, Toyer
et al. (2017) showed generalization to dierent sizes of planning problems, and Hamilton et al. (2017)
showed generalization to producing useful node embeddings for previously unseen data. On boolean
SAT problems, Selsam et al. (2018) demonstrated generalization both to dierent problem sizes and
across problem distributions: their model retained good performance upon strongly modifying the
distribution of the input graphs and its typical local structure.
These striking examples of combinatorial generalization are not entirely surprising, given GNs'
entity- and relation-centric organization, but nonetheless provide important support for the view
that embracing explicit structure and 
exible learning is a viable approach toward realizing better
sample eciency and generalization in modern AI.
5.2 Limitations of graph networks
One limitation of GNs' and MPNNs' form of learned message-passing (Shervashidze et al., 2011)


key: chroma:document
string_value:
5.2 Limitations of graph networks
One limitation of GNs' and MPNNs' form of learned message-passing (Shervashidze et al., 2011)
is that it cannot be guaranteed to solve some classes of problems, such as discriminating between
certain non-isomorphic graphs. Kondor et al. (2018) suggested that covariance7(Cohen and Welling,
2016; Kondor and Trivedi, 2018), rather than invariance to permutations of the nodes and edges
is preferable, and proposed \covariant compositional networks" which can preserve structural
information, and allow it to be ignored only if desired.
More generally, while graphs are a powerful way of representing structure information, they
have limits. For example, notions like recursion, control 
ow, and conditional iteration are not
straightforward to represent with graphs, and, minimally, require additional assumptions (e.g., in
interpreting abstract syntax trees). Programs and more \computer-like" processing can oer greater
representational and computational expressivity with respect to these notions, and some have argued
they are an important component of human cognition (Tenenbaum et al., 2011; Lake et al., 2015;
Goodman et al., 2015).
5.3 Open questions
Although we are excited about the potential impacts that graph networks can have, we caution that


key: chroma:document
string_value:
Goodman et al., 2015).
5.3 Open questions
Although we are excited about the potential impacts that graph networks can have, we caution that
these models are only one step forward. Realizing the full potential of graph networks will likely be
far more challenging than organizing their behavior under one framework, and indeed, there are a
number of unanswered questions regarding the best ways to use graph networks.
One pressing question is: where do the graphs come from that graph networks operate over?
One of the hallmarks of deep learning has been its ability to perform complex computations over
raw sensory data, such as images and text, yet it is unclear the best ways to convert sensory data
into more structured representations like graphs. One approach (which we have already discussed)
assumes a fully connected graph structure between spatial or linguistic entities, such as in the
literature on self-attention (Vaswani et al., 2017; Wang et al., 2018c). However, such representations
may not correspond exactly to the \true" entities (e.g., convolutional features do not directly
correspond to objects in a scene). Moreover, many underlying graph structures are much more
sparse than a fully connected graph, and it is an open question how to induce this sparsity. Several
lines of active research are exploring these issues (Watters et al., 2017; van Steenkiste et al., 2018;


key: chroma:document
string_value:
lines of active research are exploring these issues (Watters et al., 2017; van Steenkiste et al., 2018;
Li et al., 2018; Kipf et al., 2018) but as of yet there is no single method which can reliably extract
discrete entities from sensory data. Developing such a method is an exciting challenge for future
7Covariance means, roughly, that the activations vary in a predictable way as a function of the ordering of the
incoming edges.
23
research, and once solved will likely open the door for much more powerful and 
exible reasoning
algorithms.
A related question is how to adaptively modify graph structures during the course of computation.
For example, if an object fractures into multiple pieces, a node representing that object also ought
to split into multiple nodes. Similarly, it might be useful to only represent edges between objects
that are in contact, thus requiring the ability to add or remove edges depending on context. The
question of how to support this type of adaptivity is also actively being researched, and in particular,
some of the methods used for identifying the underlying structure of a graph may be applicable (e.g.
Li et al., 2018; Kipf et al., 2018).
Human cognition makes the strong assumption that the world is composed of objects and


key: chroma:document
string_value:
Li et al., 2018; Kipf et al., 2018).
Human cognition makes the strong assumption that the world is composed of objects and
relations (Spelke and Kinzler, 2007), and because GNs make a similar assumption, their behavior
tends to be more interpretable. The entities and relations that GNs operate over often correspond
to things that humans understand (such as physical objects), thus supporting more interpretable
analysis and visualization (e.g., as in Selsam et al., 2018). An interesting direction for future work
is to further explore the interpretability of the behavior of graph networks.
5.4 Integrative approaches for learning and structure
While our focus here has been on graphs, one takeaway from this paper is less about graphs
themselves and more about the approach of blending powerful deep learning approaches with
structured representations. We are excited by related approaches which have explored this idea for
other types of structured representations and computations, such as linguistic trees (Socher et al.,
2011a,b, 2012, 2013; Tai et al., 2015; Andreas et al., 2016), partial tree traversals in a state-action
graph (Guez et al., 2018; Farquhar et al., 2018), hierarchical action policies (Andreas et al., 2017),


key: chroma:document
string_value:
graph (Guez et al., 2018; Farquhar et al., 2018), hierarchical action policies (Andreas et al., 2017),
multi-agent communication channels (Foerster et al., 2016), \capsules" (Sabour et al., 2017), and
programs (Parisotto et al., 2017). Other methods have attempted to capture dierent types of
structure by mimicking key hardware and software components in computers and how they transfer
information between each other, such as persistent slotted storage, registers, memory I/O controllers,
stacks, and queues (e.g. Dyer et al., 2015; Grefenstette et al., 2015; Joulin and Mikolov, 2015;
Sukhbaatar et al., 2015; Kurach et al., 2016; Graves et al., 2016).
5.5 Conclusion
Recent advances in AI, propelled by deep learning, have been transformative across many important
domains. Despite this, a vast gap between human and machine intelligence remains, especially with
respect to ecient, generalizable learning. We argue for making combinatorial generalization a top
priority for AI, and advocate for embracing integrative approaches which draw on ideas from human
cognition, traditional computer science, standard engineering practice, and modern deep learning.
Here we explored 
exible learning-based approaches which implement strong relational inductive


key: chroma:document
string_value:
cognition, traditional computer science, standard engineering practice, and modern deep learning.
Here we explored 
exible learning-based approaches which implement strong relational inductive
biases to capitalize on explicitly structured representations and computations, and presented a
framework called graph networks , which generalize and extend various recent approaches for neural
networks applied to graphs. Graph networks are designed to promote building complex architectures
using customizable graph-to-graph building blocks, and their relational inductive biases promote
combinatorial generalization and improved sample eciency over other standard machine learning
building blocks.
Despite their benets and potential, however, learnable models which operate on graphs are
only a stepping stone on the path toward human-like intelligence. We are optimistic about a
number of other relevant, and perhaps underappreciated, research directions, including marrying
24
learning-based approaches with programs (Ritchie et al., 2016; Andreas et al., 2016; Gaunt et al.,
2016; Evans and Grefenstette, 2018; Evans et al., 2018), developing model-based approaches with an
emphasis on abstraction (Kansky et al., 2017; Konidaris et al., 2018; Zhang et al., 2018; Hay et al.,
2018), investing more heavily in meta-learning (Wang et al., 2016, 2018a; Finn et al., 2017), and


key: chroma:document
string_value:
2018), investing more heavily in meta-learning (Wang et al., 2016, 2018a; Finn et al., 2017), and
exploring multi-agent learning and interaction as a key catalyst for advanced intelligence (Nowak,
2006; Ohtsuki et al., 2006). These directions each involve rich notions of entities, relations, and
combinatorial generalization, and can potentially benet, and benet from, greater interaction with
approaches for learning relational reasoning over explicitly structured representations.
Acknowledgements
We thank Tobias Pfa, Danilo Rezende, Nando de Freitas, Murray Shanahan, Thore Graepel, John
Jumper, Demis Hassabis, and the broader DeepMind and Google communities for valuable feedback
and support.
References
Allamanis, M., Brockschmidt, M., and Khademi, M. (2018). Learning to represent programs with
graphs. In Proceedings of the International Conference on Learning Representations (ICLR) .
Allamanis, M., Chanthirasegaran, P., Kohli, P., and Sutton, C. (2017). Learning continuous
semantic representations of symbolic expressions. In Proceedings of the International Conference
on Machine Learning (ICML) .
Anderson, J. R. (1982). Acquisition of cognitive skill. Psychological Review , 89(4):369.


key: chroma:document
string_value:
on Machine Learning (ICML) .
Anderson, J. R. (1982). Acquisition of cognitive skill. Psychological Review , 89(4):369.
Andreas, J., Klein, D., and Levine, S. (2017). Modular multitask reinforcement learning with policy
sketches. In Proceedings of the International Conference on Machine Learning (ICML) .
Andreas, J., Rohrbach, M., Darrell, T., and Klein, D. (2016). Neural module networks. In Proceedings
of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 39{48.
Ba, J. L., Kiros, J. R., and Hinton, G. E. (2016). Layer normalization. arXiv preprint
arXiv:1607.06450 .
Bahdanau, D., Cho, K., and Bengio, Y. (2015). Neural machine translation by jointly learning to
align and translate. In Proceedings of the International Conference on Learning Representations
(ICLR) .
Battaglia, P., Pascanu, R., Lai, M., Rezende, D. J., et al. (2016). Interaction networks for learning
about objects, relations and physics. In Advances in Neural Information Processing Systems ,
pages 4502{4510.


key: chroma:document
string_value:
about objects, relations and physics. In Advances in Neural Information Processing Systems ,
pages 4502{4510.
Battaglia, P. W., Hamrick, J. B., and Tenenbaum, J. B. (2013). Simulation as an engine of physical
scene understanding. Proceedings of the National Academy of Sciences , 110(45):18327{18332.
Bello, I., Pham, H., Le, Q. V., Norouzi, M., and Bengio, S. (2016). Neural combinatorial optimization
with reinforcement learning. arXiv preprint arXiv:1611.09940 .
Bobrow, D. G. and Hinton, G. E., editors (1990). Articial Intelligence , volume 46. Elsevier Science
Publishers Ltd., Essex, UK. Special Issue 1-2: On Connectionist Symbol Processing.
25
Bojchevski, A., Shchur, O., Z ugner, D., and G unnemann, S. (2018). Netgan: Generating graphs via
random walks. arXiv preprint arXiv:1803.00816 .
Bordes, A., Usunier, N., Garcia-Duran, A., Weston, J., and Yakhnenko, O. (2013). Translating


key: chroma:document
string_value:
Bordes, A., Usunier, N., Garcia-Duran, A., Weston, J., and Yakhnenko, O. (2013). Translating
embeddings for modeling multi-relational data. In Advances in Neural Information Processing
Systems , pages 2787{2795.
Botvinick, M. M. (2008). Hierarchical models of behavior and prefrontal function. Trends in
Cognitive Sciences , 12(5):201{208.
Bronstein, M. M., Bruna, J., LeCun, Y., Szlam, A., and Vandergheynst, P. (2017). Geometric deep
learning: going beyond euclidean data. IEEE Signal Processing Magazine , 34(4):18{42.
Bruna, J., Zaremba, W., Szlam, A., and LeCun, Y. (2014). Spectral networks and locally connected
networks on graphs. In Proceedings of the International Conference on Learning Representations
(ICLR) .
Chang, M. B., Ullman, T., Torralba, A., and Tenenbaum, J. B. (2017). A compositional objectbased approach to learning physical dynamics. In Proceedings of the International Conference on
Learning Representations (ICLR) .


key: chroma:document
string_value:
Learning Representations (ICLR) .
Chen, X., Li, L., Fei-Fei, L., and Gupta, A. (2018a). Iterative visual reasoning beyond convolutions.
InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) .
Chen, X., Liu, C., and Song, D. (2018b). Tree-to-tree neural networks for program translation. In
Workshops of the International Conference on Learning Representations (ICLR) .
Cho, K., Van Merri enboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., and Bengio,
Y. (2014). Learning phrase representations using rnn encoder-decoder for statistical machine
translation. In Proceeding of the Conference on Empirical Methods in Natural Language Processing
(EMNLP) .
Chomsky, N. (1957). Syntactic Structures . Mouton & Co.
Chomsky, N. (1965). Aspects of the Theory of Syntax . MIT Press.
Cohen, T. and Welling, M. (2016). Group equivariant convolutional networks. In International
Conference on Machine Learning , pages 2990{2999.


key: chroma:document
string_value:
Cohen, T. and Welling, M. (2016). Group equivariant convolutional networks. In International
Conference on Machine Learning , pages 2990{2999.
Craik, K. J. W. (1943). The Nature of Explanation . Cambridge University Press.
Cui, Z., Henrickson, K., Ke, R., and Wang, Y. (2018). High-order graph convolutional recurrent
neural network: A deep learning framework for network-scale trac learning and forecasting.
arXiv preprint arXiv:1802.07007 .
Dai, H., Dai, B., and Song, L. (2016). Discriminative embeddings of latent variable models for
structured data. In Proceedings of the International Conference on Machine Learning (ICML) .
Dai, H., Khalil, E. B., Zhang, Y., Dilkina, B., and Song, L. (2017). Learning combinatorial
optimization algorithms over graphs. In Advances in Neural Information Processing Systems .
De Cao, N. and Kipf, T. (2018). MolGAN: An implicit generative model for small molecular graphs.
arXiv preprint arXiv:1805.11973 .
26


key: chroma:document
string_value:
arXiv preprint arXiv:1805.11973 .
26
Deerrard, M., Bresson, X., and Vandergheynst, P. (2016). Convolutional neural networks on graphs
with fast localized spectral ltering. In Advances in Neural Information Processing Systems , pages
3844{3852.
Denil, M., Colmenarejo, S. G., Cabi, S., Saxton, D., and de Freitas, N. (2017). Programmable
agents. arXiv preprint arXiv:1706.06383 .
Devlin, J., Uesato, J., Singh, R., and Kohli, P. (2017). Semantic code repair using neuro-symbolic
transformation networks. arXiv preprint arXiv:1710.11054 .
Duvenaud, D. K., Maclaurin, D., Iparraguirre, J., Bombarell, R., Hirzel, T., Aspuru-Guzik, A., and
Adams, R. P. (2015). Convolutional networks on graphs for learning molecular ngerprints. In
Advances in Neural Information Processing Systems , pages 2224{2232.


key: chroma:document
string_value:
Advances in Neural Information Processing Systems , pages 2224{2232.
Dyer, C., Ballesteros, M., Ling, W., Matthews, A., and Smith, N. A. (2015). Transition-based
dependency parsing with stack long short-term memory. In Proceedings of the Annual Meeting of
the Association for Computational Linguistics (ACL) .
D zeroski, S., De Raedt, L., and Driessens, K. (2001). Relational reinforcement learning. Machine
Learning , 43(1-2):7{52.
Edwards, H. and Storkey, A. (2016). Towards a neural statistician. arXiv preprint arXiv:1606.02185 .
Eliasmith, C. (2013). How to build a brain: A neural architecture for biological cognition . Oxford
University Press.
Elman, J. L. (1990). Finding structure in time. Cognitive Science , 14(2):179{211.
Elman, J. L. (1991). Distributed representations, simple recurrent networks, and grammatical
structure. Machine Learning , 7(2-3):195{225.


key: chroma:document
string_value:
Elman, J. L. (1991). Distributed representations, simple recurrent networks, and grammatical
structure. Machine Learning , 7(2-3):195{225.
Eslami, S. A., Heess, N., Weber, T., Tassa, Y., Szepesvari, D., Hinton, G. E., et al. (2016). Attend,
infer, repeat: Fast scene understanding with generative models. In Advances in Neural Information
Processing Systems , pages 3225{3233.
Evans, R. and Grefenstette, E. (2018). Learning explanatory rules from noisy data. Journal of
Articial Intelligence Research , 61:1{64.
Evans, R., Saxton, D., Amos, D., Kohli, P., and Grefenstette, E. (2018). Can neural networks
understand logical entailment? In Proceedings of the International Conference on Learning
Representations (ICLR) .
Farquhar, G., Rockt aschel, T., Igl, M., and Whiteson, S. (2018). TreeQN and ATreeC: Dierentiable
tree planning for deep reinforcement learning. In Proceedings of the International Conference on
Learning Representations (ICLR) .


key: chroma:document
string_value:
tree planning for deep reinforcement learning. In Proceedings of the International Conference on
Learning Representations (ICLR) .
Finn, C., Abbeel, P., and Levine, S. (2017). Model-agnostic meta-learning for fast adaptation of
deep networks. arXiv preprint arXiv:1703.03400 .
Fodor, J. A. (1975). The Language of Thought . Harvard University Press.
Fodor, J. A. and Pylyshyn, Z. W. (1988). Connectionism and cognitive architecture: A critical
analysis. Cognition , 28(1-2):3{71.
27
Foerster, J., Assael, I. A., de Freitas, N., and Whiteson, S. (2016). Learning to communicate with
deep multi-agent reinforcement learning. In Advances in Neural Information Processing Systems ,
pages 2137{2145.
Fukushima, K. (1980). Neocognitron: A self-organizing neural network model for a mechanism of
pattern recognition unaected by shift in position. Biological Cybernetics , 36:193{202.
Garcia, V. and Bruna, J. (2018). Few-shot learning with graph neural networks. In Proceedings of
the International Conference on Learning Representations (ICLR) .


key: chroma:document
string_value:
Garcia, V. and Bruna, J. (2018). Few-shot learning with graph neural networks. In Proceedings of
the International Conference on Learning Representations (ICLR) .
Garc a-Dur an, A. and Niepert, M. (2017). Learning graph representations with embedding propagation. arXiv preprint arXiv:1710.03059 .
Garnelo, M., Arulkumaran, K., and Shanahan, M. (2016). Towards deep symbolic reinforcement
learning. arXiv preprint arXiv:1609.05518 .
Gaunt, A. L., Brockschmidt, M., Kushman, N., and Tarlow, D. (2016). Dierentiable programs
with neural libraries. arXiv preprint arXiv:1611.02109 .
Geman, S., Bienenstock, E., and Doursat, R. (1992). Neural networks and the bias/variance dilemma.
Neural Computation , 4(1):1{58.
Gentner, D. and Markman, A. B. (1997). Structure mapping in analogy and similarity. American
Psychologist , 52(1):45.
Getoor, L. and Taskar, B. (2007). Introduction to Statistical Relational Learning . MIT press.


key: chroma:document
string_value:
Psychologist , 52(1):45.
Getoor, L. and Taskar, B. (2007). Introduction to Statistical Relational Learning . MIT press.
Ghahramani, Z. (2015). Probabilistic machine learning and articial intelligence. Nature ,
521(7553):452.
Gilmer, J., Schoenholz, S. S., Riley, P. F., Vinyals, O., and Dahl, G. E. (2017). Neural message
passing for quantum chemistry. arXiv preprint arXiv:1704.01212 .
Goodfellow, I., Bengio, Y., Courville, A., and Bengio, Y. (2016). Deep Learning . MIT Press.
Goodman, N. (1955). The new riddle of induction. In Fact, Fiction, and Forecast , pages 59{83.
Harvard University Press.
Goodman, N., Mansinghka, V., Roy, D. M., Bonawitz, K., and Tenenbaum, J. B. (2012). Church: a
language for generative models. arXiv preprint arXiv:1206.3255 .
Goodman, N. D., Tenenbaum, J. B., and Gerstenberg, T. (2015). Concepts in a probabilistic


key: chroma:document
string_value:
Goodman, N. D., Tenenbaum, J. B., and Gerstenberg, T. (2015). Concepts in a probabilistic
language of thought. In Margolis, E. and Laurence, S., editors, The Conceptual Mind: New
Directions in the Study of Concepts . MIT Press.
Goodwin, G. P. and Johnson-Laird, P. (2005). Reasoning about relations. Psychological Review ,
112(2):468.
Gori, M., Monfardini, G., and Scarselli, F. (2005). A new model for learning in graph domains. In
Proceedings of the International Joint Conference on Neural Networks (IJCNN) , volume 2, pages
729{734. IEEE.
Graves, A., Wayne, G., Reynolds, M., Harley, T., Danihelka, I., Grabska-Barwi nska, A., Colmenarejo,
S. G., Grefenstette, E., Ramalho, T., Agapiou, J., et al. (2016). Hybrid computing using a neural
network with dynamic external memory. Nature , 538(7626):471.
28
Grefenstette, E., Hermann, K. M., Suleyman, M., and Blunsom, P. (2015). Learning to transduce


key: chroma:document
string_value:
28
Grefenstette, E., Hermann, K. M., Suleyman, M., and Blunsom, P. (2015). Learning to transduce
with unbounded memory. In Advances in Neural Information Processing Systems , pages 1828{1836.
Griths, T. L., Chater, N., Kemp, C., Perfors, A., and Tenenbaum, J. B. (2010). Probabilistic
models of cognition: Exploring representations and inductive biases. Trends in Cognitive Sciences ,
14(8):357{364.
Grover, A. and Leskovec, J. (2016). node2vec: Scalable feature learning for networks. In Proceedings
of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining ,
pages 855{864. ACM.
Guez, A., Weber, T., Antonoglou, I., Simonyan, K., Vinyals, O., Wierstra, D., Munos, R., and
Silver, D. (2018). Learning to search with MCTSnets. arXiv preprint arXiv:1802.04697 .


key: chroma:document
string_value:
Silver, D. (2018). Learning to search with MCTSnets. arXiv preprint arXiv:1802.04697 .
Gulcehre, C., Denil, M., Malinowski, M., Razavi, A., Pascanu, R., Hermann, K. M., Battaglia, P.,
Bapst, V., Raposo, D., Santoro, A., and de Freitas, N. (2018). Hyperbolic attention networks.
arXiv preprint arXiv:1805.09786 .
Hamaguchi, T., Oiwa, H., Shimbo, M., and Matsumoto, Y. (2017). Knowledge transfer for out-ofknowledge-base entities: A graph neural network approach. In Proceedings of the International
Joint Conference on Articial Intelligence (IJCAI) .
Hamilton, W., Ying, Z., and Leskovec, J. (2017). Inductive representation learning on large graphs.
InAdvances in Neural Information Processing Systems , pages 1025{1035.
Hamrick, J., Allen, K., Bapst, V., Zhu, T., McKee, K., Tenenbaum, J., and Battaglia, P. (2018).
Relational inductive bias for physical construction in humans and machines. In Proceedings of the


key: chroma:document
string_value:
Relational inductive bias for physical construction in humans and machines. In Proceedings of the
40th Annual Conference of the Cognitive Science Society .
Hamrick, J. B., Ballard, A. J., Pascanu, R., Vinyals, O., Heess, N., and Battaglia, P. W. (2017).
Metacontrol for adaptive imagination-based optimization. In Proceedings of the International
Conference on Learning Representations (ICLR) .
Hartford, J., Graham, D. R., Leyton-Brown, K., and Ravanbakhsh, S. (2018). Deep models of
interactions across sets. arXiv preprint arXiv:1803.02879 .
Hay, N., Stark, M., Schlegel, A., Wendelken, C., Park, D., Purdy, E., Silver, T., Phoenix, D. S.,
and George, D. (2018). Behavior is everything{towards representing concepts with sensorimotor
contingencies. In Proceedings of the AAAI Conference on Articial Intelligence (AAAI) .
Hena, M., Bruna, J., and LeCun, Y. (2015). Deep convolutional networks on graph-structured
data. arXiv preprint arXiv:1506.05163 .


key: chroma:document
string_value:
data. arXiv preprint arXiv:1506.05163 .
Hinton, G. E. (1990). Mapping part-whole hierarchies into connectionist networks. Articial
Intelligence , 46(1-2):47{75.
Hjort, N. L., Holmes, C., M uller, P., and Walker, S. G. (2010). Bayesian Nonparametrics . Cambridge
University Press.
Hoshen, Y. (2017). Vain: Attentional multi-agent predictive modeling. In Advances in Neural
Information Processing Systems , pages 2698{2708.
Hu, H., Gu, J., Zhang, Z., Dai, J., and Wei, Y. (2017). Relation networks for object detection.
arXiv preprint arXiv:1711.11575 .
29
Hudson, D. A. and Manning, C. D. (2018). Compositional attention networks for machine reasoning.
InProceedings of the International Conference on Learning Representations (ICLR) .
Humboldt, W. (1999/1836). On Language: On the diversity of human language construction and its
in
uence on the mental development of the human species . Cambridge University Press.


key: chroma:document
string_value:
in
uence on the mental development of the human species . Cambridge University Press.
Hummel, J. E. and Holyoak, K. J. (2003). A symbolic-connectionist theory of relational inference
and generalization. Psychological Review , 110(2):220.
Ioe, S. and Szegedy, C. (2015). Batch normalization: Accelerating deep network training by reducing
internal covariate shift. In Proceedings of the 32nd International Conference on International
Conference on Machine Learning (ICML) .
Johnson, D. D. (2017). Learning graphical state transitions. Proceedings of the International
Conference on Learning Representations (ICLR) .
Joulin, A. and Mikolov, T. (2015). Inferring algorithmic patterns with stack-augmented recurrent
nets. In Advances in Neural Information Processing Systems , pages 190{198.
Kansky, K., Silver, T., M ely, D. A., Eldawy, M., L azaro-Gredilla, M., Lou, X., Dorfman, N., Sidor,
S., Phoenix, S., and George, D. (2017). Schema networks: Zero-shot transfer with a generative
causal model of intuitive physics. In Proceedings of the International Conference on Machine
Learning (ICML) .


