# embedding_metadata

key: chroma:document
string_value:
causal model of intuitive physics. In Proceedings of the International Conference on Machine
Learning (ICML) .
Kearnes, S., McCloskey, K., Berndl, M., Pande, V., and Riley, P. (2016). Molecular graph
convolutions: moving beyond ngerprints. Journal of computer-aided molecular design , 30(8):595{
608.
Kemp, C. and Tenenbaum, J. B. (2008). The discovery of structural form. Proceedings of the
National Academy of Sciences , 105(31):10687{10692.
Kipf, T., Fetaya, E., Wang, K.-C., Welling, M., and Zemel, R. (2018). Neural relational inference
for interacting systems. In Proceedings of the International Conference on Machine Learning
(ICML) .
Kipf, T. N. and Welling, M. (2017). Semi-supervised classication with graph convolutional networks.
InProceedings of the International Conference on Learning Representations (ICLR) .
Koller, D. and Friedman, N. (2009). Probabilistic Graphical Models: Principles and Techniques .
MIT press.


key: chroma:document
string_value:
Koller, D. and Friedman, N. (2009). Probabilistic Graphical Models: Principles and Techniques .
MIT press.
Kondor, R., Son, H. T., Pan, H., Anderson, B., and Trivedi, S. (2018). Covariant compositional
networks for learning graphs. arXiv preprint arXiv:1801.02144 .
Kondor, R. and Trivedi, S. (2018). On the generalization of equivariance and convolution in neural
networks to the action of compact groups. arXiv preprint arXiv:1802.03690 .
Konidaris, G., Kaelbling, L. P., and Lozano-Perez, T. (2018). From skills to symbols: Learning
symbolic representations for abstract high-level planning. Journal of Articial Intelligence
Research , 61:215{289.
Kool, W. and Welling, M. (2018). Attention solves your TSP. arXiv preprint arXiv:1803.08475 .
30
Krizhevsky, A., Sutskever, I., and Hinton, G. E. (2012). ImageNet classication with deep convolutional neural networks. In Advances in Neural Information Processing Systems , pages 1097{1105.


key: chroma:document
string_value:
Kurach, K., Andrychowicz, M., and Sutskever, I. (2016). Neural random-access machines. In
Proceedings of the International Conference on Learning Representations (ICLR) .
Lake, B. M. and Baroni, M. (2018). Still not systematic after all these years: On the compositional
skills of sequence-to-sequence recurrent networks. In Proceedings of the International Conference
on Machine Learning (ICML) .
Lake, B. M., Salakhutdinov, R., and Tenenbaum, J. B. (2015). Human-level concept learning
through probabilistic program induction. Science , 350(6266):1332{1338.
Lake, B. M., Ullman, T. D., Tenenbaum, J. B., and Gershman, S. J. (2017). Building machines that
learn and think like people. Behavioral and Brain Sciences , 40.
LeCun, Y., Bengio, Y., and Hinton, G. (2015). Deep learning. Nature , 521(7553):436.
LeCun, Y., Boser, B., Denker, J. S., Henderson, D., Howard, R. E., Hubbard, W., and Jackel,


key: chroma:document
string_value:
LeCun, Y., Boser, B., Denker, J. S., Henderson, D., Howard, R. E., Hubbard, W., and Jackel,
L. D. (1989). Backpropagation applied to handwritten zip code recognition. Neural computation ,
1(4):541{551.
Li, Y., Tarlow, D., Brockschmidt, M., and Zemel, R. (2016). Gated graph sequence neural networks.
InProceedings of the International Conference on Learning Representations (ICLR) .
Li, Y., Vinyals, O., Dyer, C., Pascanu, R., and Battaglia, P. (2018). Learning deep generative
models of graphs. In Workshops at the International Conference on Learning Representations
(ICLR) .
Li, Y., Yu, R., Shahabi, C., and Liu, Y. (2017). Diusion convolutional recurrent neural network:
Data-driven trac forecasting. arXiv preprint arXiv:1707.01926 .
Lin, Z., Feng, M., Santos, C. N. d., Yu, M., Xiang, B., Zhou, B., and Bengio, Y. (2017). A structured
self-attentive sentence embedding. In Proceedings of the International Conference on Learning


key: chroma:document
string_value:
self-attentive sentence embedding. In Proceedings of the International Conference on Learning
Representations (ICLR) .
Liu, H., Simonyan, K., Vinyals, O., Fernando, C., and Kavukcuoglu, K. (2018). Hierarchical
representations for ecient architecture search. In Proceedings of the International Conference on
Learning Representations (ICLR) .
Luong, M.-T., Pham, H., and Manning, C. D. (2015). Eective approaches to attention-based neural
machine translation. arXiv preprint arXiv:1508.04025 .
Marcus, G. (2001). The algebraic mind.
Marcus, G. (2018a). Deep learning: A critical appraisal. arXiv preprint arXiv:1801.00631 .
Marcus, G. (2018b). Innateness, alphazero, and articial intelligence. arXiv preprint
arXiv:1801.05667 .
McClelland, J. L. (1994). The interaction of nature and nurture in development: A parallel
distributed processing perspective. International perspectives on psychological science , 1:57{88.
31
McClelland, J. L. and Rumelhart, D. E. (1981). An interactive activation model of context eects


key: chroma:document
string_value:
31
McClelland, J. L. and Rumelhart, D. E. (1981). An interactive activation model of context eects
in letter perception: I. an account of basic ndings. Psychological Review , 88(5):375.
Mikolov, T., Yih, W.-t., and Zweig, G. (2013). Linguistic regularities in continuous space word
representations. In Proceedings of the 2013 Conference of the North American Chapter of the
Association for Computational Linguistics: Human Language Technologies , pages 746{751.
Mitchell, T. M. (1980). The need for biases in learning generalizations . Department of Computer
Science, Laboratory for Computer Science Research, Rutgers Univ. New Jersey.
Mnih, V., Heess, N., Graves, A., et al. (2014). Recurrent models of visual attention. In Advances in
neural information processing systems , pages 2204{2212.
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A.,
Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015). Human-level control through deep


key: chroma:document
string_value:
Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015). Human-level control through deep
reinforcement learning. Nature , 518(7540):529.
Monti, F., Boscaini, D., Masci, J., Rodola, E., Svoboda, J., and Bronstein, M. M. (2017). Geometric
deep learning on graphs and manifolds using mixture model cnns. In Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition (CVPR) .
Morav c k, M., Schmid, M., Burch, N., Lis y, V., Morrill, D., Bard, N., Davis, T., Waugh, K.,
Johanson, M., and Bowling, M. (2017). Deepstack: Expert-level articial intelligence in heads-up
no-limit poker. Science , 356(6337):508{513.
Narayanan, A., Chandramohan, M., Chen, L., Liu, Y., and Saminathan, S. (2016). subgraph2vec:
Learning distributed representations of rooted sub-graphs from large graphs. In Workshops at the
20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining .


key: chroma:document
string_value:
Learning distributed representations of rooted sub-graphs from large graphs. In Workshops at the
20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining .
Narayanan, A., Chandramohan, M., Venkatesan, R., Chen, L., Liu, Y., and Jaiswal, S. (2017).
graph2vec: Learning distributed representations of graphs. arXiv preprint arXiv:1707.05005 .
Navon, D. (1977). Forest before trees: The precedence of global features in visual perception.
Cognitive Psychology , 9(3):353{383.
Niepert, M., Ahmed, M., and Kutzkov, K. (2016). Learning convolutional neural networks for
graphs. In Proceedings of the International Conference on Machine Learning (ICML) , pages
2014{2023.
Nilsson, N. J. and Fikes, R. E. (1970). Strips: A new approach to the application of theorem proving
to problem solving. Technical report, SRI International, Menlo Park, CA Articial Intelligence
Center.
Nowak, A., Villar, S., Bandeira, A. S., and Bruna, J. (2017). A note on learning algorithms for
quadratic assignment with graph neural networks. In Proceedings of the Principled Approaches to


key: chroma:document
string_value:
quadratic assignment with graph neural networks. In Proceedings of the Principled Approaches to
Deep Learning Workshop (PADL) at the International Conference of Machine Learning (ICML) .
Nowak, M. A. (2006). Five rules for the evolution of cooperation. science , 314(5805):1560{1563.
Ohtsuki, H., Hauert, C., Lieberman, E., and Nowak, M. A. (2006). A simple rule for the evolution
of cooperation on graphs and social networks. Nature , 441(7092):502.
32
O~ noro-Rubio, D., Niepert, M., Garc a-Dur an, A., Gonz alez-S anchez, R., and L opez-Sastre,
R. J. (2017). Representation learning for visual-relational knowledge graphs. arXiv preprint
arXiv:1709.02314 .
Parisotto, E., Mohamed, A.-r., Singh, R., Li, L., Zhou, D., and Kohli, P. (2017). Neuro-symbolic
program synthesis. In Proceedings of the International Conference on Learning Representations
(ICLR) .


key: chroma:document
string_value:
program synthesis. In Proceedings of the International Conference on Learning Representations
(ICLR) .
Pascanu, R., Li, Y., Vinyals, O., Heess, N., Buesing, L., Racani ere, S., Reichert, D., Weber, T.,
Wierstra, D., and Battaglia, P. (2017). Learning model-based planning from scratch. arXiv
preprint arXiv:1707.06170 .
Pearl, J. (1986). Fusion, propagation, and structuring in belief networks. Articial intelligence ,
29(3):241{288.
Pearl, J. (1988). Probabilistic reasoning in intelligent systems: Networks of plausible inference.
Morgan Kaufmann.
Pearl, J. (2009). Causality: Models, Reasoning and Inference . Cambridge University Press, New
York, NY, USA, 2nd edition.
Pearl, J. (2018). Theoretical impediments to machine learning with seven sparks from the causal
revolution. arXiv preprint arXiv:1801.04016 .
Pennington, J., Socher, R., and Manning, C. (2014). Glove: Global vectors for word representation.
InProceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP) ,


key: chroma:document
string_value:
InProceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP) ,
pages 1532{1543.
Perozzi, B., Al-Rfou, R., and Skiena, S. (2014). Deepwalk: Online learning of social representations.
InProceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and
data mining , pages 701{710. ACM.
Pevn y, T. and Somol, P. (2017). Using neural network formalism to solve multiple-instance problems.
InInternational Symposium on Neural Networks , pages 135{142. Springer.
Pinker, S. and Prince, A. (1988). On language and connectionism: Analysis of a parallel distributed
processing model of language acquisition. Cognition , 28(1-2):73{193.
Plate, T. A. (1995). Holographic reduced representations. IEEE Transactions on Neural Networks ,
6(3):623{641.
Plaut, D. C., McClelland, J. L., Seidenberg, M. S., and Patterson, K. (1996). Understanding normal
and impaired word reading: computational principles in quasi-regular domains. Psychological
Review , 103(1):56.


key: chroma:document
string_value:
and impaired word reading: computational principles in quasi-regular domains. Psychological
Review , 103(1):56.
Pollack, J. B. (1990). Recursive distributed representations. Articial Intelligence , 46(1-2):77{105.
Qi, C. R., Su, H., Mo, K., and Guibas, L. J. (2017). Pointnet: Deep learning on point sets for 3d
classication and segmentation. In Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition (CVPR) .
Raposo, D., Santoro, A., Barrett, D., Pascanu, R., Lillicrap, T., and Battaglia, P. (2017). Discovering
objects and their relations from entangled scene representations. In Workshops at the International
Conference on Learning Representations (ICLR) .
33
Reed, S. and De Freitas, N. (2016). Neural programmer-interpreters. In Proceedings of the
International Conference on Learning Representations (ICLR) .
Ritchie, D., Horsfall, P., and Goodman, N. D. (2016). Deep amortized inference for probabilistic
programs. arXiv preprint arXiv:1610.05735 .


key: chroma:document
string_value:
programs. arXiv preprint arXiv:1610.05735 .
Rosenblatt, F. (1961). Principles of neurodynamics. perceptrons and the theory of brain mechanisms.
Technical report, Cornell Aeronautical Lab Inc., Bualo, NY.
Rumelhart, D. E., McClelland, J. L., Group, P. R., et al. (1987). Parallel Distributed Processing ,
volume 1. MIT Press.
Russell, S. J. and Norvig, P. (2009). Articial Intelligence: A Modern Approach (3rd Edition) .
Pearson.
Sabour, S., Frosst, N., and Hinton, G. E. (2017). Dynamic routing between capsules. In Advances
in Neural Information Processing Systems , pages 3859{3869.
Sanchez-Gonzalez, A., Heess, N., Springenberg, J. T., Merel, J., Riedmiller, M., Hadsell, R., and
Battaglia, P. (2018). Graph networks as learnable physics engines for inference and control. In
Proceedings of the 35th International Conference on Machine Learning (ICLR) .


key: chroma:document
string_value:
Proceedings of the 35th International Conference on Machine Learning (ICLR) .
Santoro, A., Raposo, D., Barrett, D. G., Malinowski, M., Pascanu, R., Battaglia, P., and Lillicrap,
T. (2017). A simple neural network module for relational reasoning. In Advances in Neural
Information Processing Systems .
Scarselli, F., Gori, M., Tsoi, A. C., Hagenbuchner, M., and Monfardini, G. (2009a). Computational
capabilities of graph neural networks. IEEE Transactions on Neural Networks , 20(1):81{102.
Scarselli, F., Gori, M., Tsoi, A. C., Hagenbuchner, M., and Monfardini, G. (2009b). The graph
neural network model. IEEE Transactions on Neural Networks , 20(1):61{80.
Scarselli, F., Yong, S. L., Gori, M., Hagenbuchner, M., Tsoi, A. C., and Maggini, M. (2005). Graph
neural networks for ranking web pages. In Proceedings of the 2005 IEEE/WIC/ACM International
Conference on Web Intelligence , pages 666{672. IEEE.


key: chroma:document
string_value:
neural networks for ranking web pages. In Proceedings of the 2005 IEEE/WIC/ACM International
Conference on Web Intelligence , pages 666{672. IEEE.
Schmidhuber, J. (2015). Deep learning in neural networks: An overview. Neural Networks , 61:85{117.
Selsam, D., Lamm, M., Bunz, B., Liang, P., de Moura, L., and Dill, D. L. (2018). Learning a sat
solver from single-bit supervision. arXiv preprint arXiv:1802.03685 .
Shalev-Shwartz, S., Shamir, O., and Shammah, S. (2017). Failures of gradient-based deep learning.
arXiv preprint arXiv:1703.07950 .
Shaw, P., Uszkoreit, J., and Vaswani, A. (2018). Self-attention with relative position representations.
InProceedings of the 16th Annual Conference of the North American Chapter of the Association
for Computational Linguistics: Human Language Technologies .
Shervashidze, N., Schweitzer, P., Leeuwen, E. J. v., Mehlhorn, K., and Borgwardt, K. M. (2011).


key: chroma:document
string_value:
Weisfeiler-lehman graph kernels. Journal of Machine Learning Research , 12(Sep):2539{2561.
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., Van Den Driessche, G., Schrittwieser, J.,
Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. (2016). Mastering the game of go with
deep neural networks and tree search. Nature , 529(7587):484{489.
34
Smolensky, P. (1990). Tensor product variable binding and the representation of symbolic structures
in connectionist systems. Articial Intelligence , 46(1-2):159{216.
Socher, R., Huval, B., Manning, C. D., and Ng, A. Y. (2012). Semantic compositionality through
recursive matrix-vector spaces. In Proceedings of the Joint Conference on Empirical Methods in
Natural Language Processing (EMNLP) and Computational Natural Language Learning (CNLL) ,
pages 1201{1211. Association for Computational Linguistics.
Socher, R., Lin, C. C., Manning, C., and Ng, A. Y. (2011a). Parsing natural scenes and natural


key: chroma:document
string_value:
Socher, R., Lin, C. C., Manning, C., and Ng, A. Y. (2011a). Parsing natural scenes and natural
language with recursive neural networks. In Proceedings of the 28th International Conference on
Machine Learning (ICML) , pages 129{136.
Socher, R., Pennington, J., Huang, E. H., Ng, A. Y., and Manning, C. D. (2011b). Semi-supervised
recursive autoencoders for predicting sentiment distributions. In Proceedings of the Conference
on Empirical Methods in Natural Language Processing (EMNLP) , pages 151{161. Association for
Computational Linguistics.
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., and Potts, C. (2013).
Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of
the Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 1631{1642.
Spelke, E. S., Breinlinger, K., Macomber, J., and Jacobson, K. (1992). Origins of knowledge.
Psychological review , 99(4):605.


key: chroma:document
string_value:
Psychological review , 99(4):605.
Spelke, E. S. and Kinzler, K. D. (2007). Core knowledge. Developmental Science , 10(1):89{96.
Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. (2014). Dropout:
A simple way to prevent neural networks from overtting. The Journal of Machine Learning
Research , 15(1):1929{1958.
Sukhbaatar, S., Fergus, R., et al. (2016). Learning multiagent communication with backpropagation.
InAdvances in Neural Information Processing Systems , pages 2244{2252.
Sukhbaatar, S., Weston, J., Fergus, R., et al. (2015). End-to-end memory networks. In Advances in
Neural Information Processing Systems , pages 2440{2448.
Sutskever, I., Vinyals, O., and Le, Q. V. (2014). Sequence to sequence learning with neural networks.
InAdvances in Neural Information Processing Systems , pages 3104{3112.


key: chroma:document
string_value:
InAdvances in Neural Information Processing Systems , pages 3104{3112.
Szegedy, C., Ioe, S., Vanhoucke, V., and Alemi, A. A. (2017). Inception-v4, inception-resnet
and the impact of residual connections on learning. In Proceedings of the AAAI Conference on
Articial Intelligence (AAAI) , volume 4, page 12.
Tai, K. S., Socher, R., and Manning, C. D. (2015). Improved semantic representations from
tree-structured long short-term memory networks. In Proceedings of the Annual Meeting of the
Association for Computational Linguistics (ACL) .
Tang, J., Qu, M., Wang, M., Zhang, M., Yan, J., and Mei, Q. (2015). Line: Large-scale information
network embedding. In Proceedings of the 24th International Conference on World Wide Web ,
pages 1067{1077. International World Wide Web Conferences Steering Committee.
Tenenbaum, J. B., Griths, T. L., and Kemp, C. (2006). Theory-based bayesian models of inductive
learning and reasoning. Trends in Cognitive Sciences , 10(7):309{318.
35


key: chroma:document
string_value:
learning and reasoning. Trends in Cognitive Sciences , 10(7):309{318.
35
Tenenbaum, J. B., Kemp, C., Griths, T. L., and Goodman, N. D. (2011). How to grow a mind:
Statistics, structure, and abstraction. Science , 331(6022):1279{1285.
Toyer, S., Trevizan, F., Thiebaux, S., and Xie, L. (2017). Action schema networks: Generalised
policies with deep learning. In Proceedings of the AAAI Conference on Articial Intelligence
(AAAI) .
Ullman, T. D., Spelke, E., Battaglia, P., and Tenenbaum, J. B. (2017). Mind games: Game engines
as an architecture for intuitive physics. Trends in Cognitive Sciences , 21(9):649{665.
van Steenkiste, S., Chang, M., Gre, K., and Schmidhuber, J. (2018). Relational neural expectation
maximization: Unsupervised discovery of objects and their interactions. Proceedings of the
International Conference on Learning Representations (ICLR) .


key: chroma:document
string_value:
maximization: Unsupervised discovery of objects and their interactions. Proceedings of the
International Conference on Learning Representations (ICLR) .
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and
Polosukhin, I. (2017). Attention is all you need. In Advances in Neural Information Processing
Systems .
Veli ckovi c, P., Cucurull, G., Casanova, A., Romero, A., Li o, P., and Bengio, Y. (2018). Graph
attention networks. In Proceedings of the International Conference on Learning Representations
(ICLR) .
Wang, J. X., Kurth-Nelson, Z., Kumaran, D., Tirumala, D., Soyer, H., Leibo, J. Z., Hassabis, D.,
and Botvinick, M. (2018a). Prefrontal cortex as a meta-reinforcement learning system. Nature
neuroscience , page 1.
Wang, J. X., Kurth-Nelson, Z., Tirumala, D., Soyer, H., Leibo, J. Z., Munos, R., Blundell, C.,


key: chroma:document
string_value:
Kumaran, D., and Botvinick, M. (2016). Learning to reinforcement learn. arXiv preprint
arXiv:1611.05763 .
Wang, T., Liao, R., Ba, J., and Fidler, S. (2018b). Nervenet: Learning structured policy with graph
neural networks. In Proceedings of the International Conference on Learning Representations
(ICLR) .
Wang, X., Girshick, R., Gupta, A., and He, K. (2018c). Non-local neural networks. In Proceedings
of the Conference on Computer Vision and Pattern Recognition (CVPR) .
Wang, Y., Sun, Y., Liu, Z., Sarma, S. E., Bronstein, M. M., and Solomon, J. M. (2018d). Dynamic
graph cnn for learning on point clouds. arXiv preprint arXiv:1801.07829 .
Watters, N., Zoran, D., Weber, T., Battaglia, P., Pascanu, R., and Tacchetti, A. (2017). Visual
interaction networks: Learning a physics simulator from video. In Advances in Neural Information
Processing Systems , pages 4542{4550.


key: chroma:document
string_value:
interaction networks: Learning a physics simulator from video. In Advances in Neural Information
Processing Systems , pages 4542{4550.
Wu, J., Lu, E., Kohli, P., Freeman, B., and Tenenbaum, J. (2017). Learning to see physics via
visual de-animation. In Advances in Neural Information Processing Systems , pages 152{163.
Yoon, K., Liao, R., Xiong, Y., Zhang, L., Fetaya, E., Urtasun, R., Zemel, R., and Pitkow, X.
(2018). Inference in probabilistic graphical models by graph neural networks. In Workshops at
the International Conference on Learning Representations (ICLR) .
You, J., Ying, R., Ren, X., Hamilton, W. L., and Leskovec, J. (2018). GraphRNN: A deep generative
model for graphs. arXiv preprint arXiv:1802.08773 .
36
Yuille, A. L. and Liu, C. (2018). Deep nets: What have they ever done for vision? arXiv preprint
arXiv:1805.04025 .


key: chroma:document
string_value:
arXiv:1805.04025 .
Zaheer, M., Kottur, S., Ravanbakhsh, S., Poczos, B., Salakhutdinov, R. R., and Smola, A. J. (2017).
Deep sets. In Advances in Neural Information Processing Systems , pages 3394{3404.
Zambaldi, V., Raposo, D., Santoro, A., Bapst, V., Li, Y., Babuschkin, I., Tuyls, K., Reichert, D.,
Lillicrap, T., Lockhart, E., Shanahan, M., Langston, V., Pascanu, R., Botvinick, M., Vinyals, O.,
and Battaglia, P. (2018). Relational deep reinforcement learning. arXiv preprint arXiv .
Zhang, A., Lerer, A., Sukhbaatar, S., Fergus, R., and Szlam, A. (2018). Composable planning with
attributes. arXiv preprint arXiv:1803.00512 .
Z ugner, D., Akbarnejad, A., and G unnemann, S. (2018). Adversarial Attacks on Neural Networks


key: chroma:document
string_value:
Z ugner, D., Akbarnejad, A., and G unnemann, S. (2018). Adversarial Attacks on Neural Networks
for Graph Data. arXiv preprint arXiv:1805.07984 .
37
Appendix: Formulations of additional models
In this appendix we give more examples of how published networks can t in the frame dened by
Equation 1.
Interaction networks
Interaction Networks (Battaglia et al., 2016; Watters et al., 2017) and the Neural Physics Engine
Chang et al. (2017) use a full GN but for the absence of the global to update the edge properties:
e(ek;vrk;vsk;u):=fe(ek;vrk;vsk) = NNe([ek;vrk;vsk])
v
 e0
i;vi;u:=fv
 e0
i;vi;u
= NNv
[ e0
i;vi;u]
e!v
E0
i:= =X
fk:rk=ige0
k
That work also included an extension to the above formulation which output global, rather than
per-node, predictions:


key: chroma:document
string_value:
fk:rk=ige0
k
That work also included an extension to the above formulation which output global, rather than
per-node, predictions:
e(ek;vrk;vsk;u):=fe(ek;vrk;vsk) = NNe([ek;vrk;vsk])
v
 e0
i;vi;u:=fv
 e0
i;vi;u
= NNv
[ e0
i;vi;u]
u
 e0; v0;u:=fu
 v0;u
= NNu
[ v0;u]
v!g
V0:= =X
iv0
i
Non-pairwise interactions
Gated Graph Sequence Neural Networks (GGS-NN) (Li et al., 2016) use a slightly generalized
formulation where each edge has an attached type tk2f1;::;Tg, and the updates are:
e((ek;tk);vrk;vsk;u):=fe(ek;vsk) = NNe;tk(vsk)
v
 e0
i;vi;u:=fv
 e0
i;vi


key: chroma:document
string_value:
v
 e0
i;vi;u:=fv
 e0
i;vi
= NNv
[ e0
i;vi]
e!v
E0
i:= =X
fk:rk=ige0
k
These updates are applied recurrently (the NNvis a GRU (Cho et al., 2014)), followed by a global
decoder which computes a weighted sum of embedded nal node states. Here each NNe;tkis a neural
network with specic parameters.
CommNet (Sukhbaatar et al., 2016) (in the slightly more general form described by (Hoshen,
2017)) uses:
e(ek;vrk;vsk;u):=fe(vsk) = NN e(vsk)
v
 e0
i;vi;u:=fv
 e0
i;vi
= NNv
[ e0
i;NNv0(vi)]
e!v
E0
i:= =1
jE0
ijX
fk:rk=ige0
k
38
Attention-based approaches


key: chroma:document
string_value:
E0
i:= =1
jE0
ijX
fk:rk=ige0
k
38
Attention-based approaches
The various attention-based approaches use a ewhich is factored into a scalar pairwise-interaction
function which returns the unnormalized attention term, denoted e(vrk;vsk)=a0
k, and a vectorvalued non-pairwise term, denoted e(vsk) =b0
k,
e(ek;vrk;vsk;u):=fe(vrk;vsk) = (e(vrk;vsk); e(vsk)) = (a0
k;b0
k) =e0
k
The single-headed self-attention (SA) in the Transformer architecture (Vaswani et al., 2017),
implements the non-local formulation as:
e(vrk;vsk) = exp (NN query(vrk)|NNkey(vsk))
e(vsk) = NN (vsk)
v
 e0
i;vi;u:=fv
 e0
i
= NNv
 e0
i


key: chroma:document
string_value:
v
 e0
i;vi;u:=fv
 e0
i
= NNv
 e0
i
where NNquery,NNkey, and NNare again neural network functions with dierent parameters and
possibly dierent architectures. They also use a multi-headed version which computes Nhparallel
 e0h
iusing dierent NNquery
h,NNkey
h,NNh, wherehindexes the dierent parameters. These are
passed tofvand concatenated:
fv
f e0h
igh=1:::Nh
= NNv
[ e01
i;:::; e0Nh
i]
Vertex Attention Interaction Networks (Hoshen, 2017) are very similar to single-headed SA,
but use Euclidean distance for the attentional similarity metric, with shared parameters across the
attention inputs' embeddings, and also use the input node feature in the node update function,
e(vrk;vsk) = exp
kNN(vrk)NN(vsk)k2
e(vsk) = NN (vsk)
v


key: chroma:document
string_value:
kNN(vrk)NN(vsk)k2
e(vsk) = NN (vsk)
v
 e0
i;vi;u:=fv
 e0
i
= NNv
[ e0
i;vi]
Graph Attention Networks (Veli ckovi c et al., 2018) are also similar to multi-headed SA, but use
a neural network as the attentional similarity metric, with shared parameters across the attention
inputs' embeddings:
e(vrk;vsk) = exp (NN 0([NN(vrk);NN(vsk)))
e(vsk) = NN (vsk)
v
 e0
i;vi;u:=fv
f e0h
igh=1:::Nh
= NNv
[ e01
i;:::; e0Nh
i]
Stretching beyond the specic non-local formulation, Shaw et al. (2018) extended multi-headed
SA with relative position encodings. \Relative" refers to an encoding of the spatial distance between


key: chroma:document
string_value:
SA with relative position encodings. \Relative" refers to an encoding of the spatial distance between
nodes in a sequence or other signal in a metric space. This can be expressed in GN language as an
edge attribute ek, and replacing the e(vsk) from multi-headed SA above with:
e(ek;vsk) = NNe(vsk) +ek
39
Belief Propagation embeddings
Finally, we brie
y summarize how the general \structure2vec" algorithm of Dai et al. (2016) can t
into our framework. In order to do so, we need to slightly modify our main Equation 1, i.e.:
k=
felgsl=rk
rl6=sk
:=X
rl=sk
sl6=rkel
e0
k=e(k) :=f(k) = NN( k)
 e0
i=
fe0
kgrk=i:=X
fk:rk=igek
v0
i=v
 e0
i:=f( e0
i) = NN(  e0
i)


key: chroma:document
string_value:
v0
i=v
 e0
i:=f( e0
i) = NN(  e0
i)
Edges' features now takes the meaning of \message" between their receiver and sender; note that
there is only one set of parameters to learn for both the edges and nodes updates.
40


key: chroma:document
string_value:
Object Level Visual Reasoning in Videos
Fabien Baradel1, Natalia Neverova2, Christian Wolf1;3,
Julien Mille4, and Greg Mori5
1Universit e Lyon, INSA Lyon, CNRS, LIRIS, F-69621, Villeurbanne, France,
firstname.lastname@liris.cnrs.fr
2Facebook AI Research, Paris, France, nneverova@fb.com
3INRIA, CITI Laboratory, Villeurbanne, France
4Laboratoire d'Informatique de l'Univ. de Tours, INSA Centre Val de Loire,
41034, Blois, France, julien.mille@insa-cvl.fr
5Simon Fraser University, Vancouver, Canada, mori@cs.sfu.ca
https://fabienbaradel.github.io/eccv18_object_level_visual_reasoning/
Abstract. Human activity recognition is typically addressed by detecting key concepts like global and local motion, features related to object
classes present in the scene, as well as features related to the global context. The next open challenges in activity recognition require a level of
understanding that pushes beyond this and call for models with capabilities for ne distinction and detailed comprehension of interactions


key: chroma:document
string_value:
understanding that pushes beyond this and call for models with capabilities for ne distinction and detailed comprehension of interactions
between actors and objects in a scene. We propose a model capable of
learning to reason about semantically meaningful spatio-temporal interactions in videos. The key to our approach is a choice of performing
this reasoning at the object level through the integration of state of the
art object detection networks. This allows the model to learn detailed
spatial interactions that exist at a semantic, object-interaction relevant
level. We evaluate our method on three standard datasets (Twenty-BN
Something-Something, VLOG and EPIC Kitchens) and achieve state of
the art results on all of them. Finally, we show visualizations of the interactions learned by the model, which illustrate object classes and their
interactions corresponding to dierent activity classes.
Keywords: Video understanding Human-object interaction
1 Introduction
The eld of video understanding is extremely diverse, ranging from extracting highly detailed information captured by specically designed motion capture systems [33] to making general sense of videos from the Web [1]. As in
the domain of image recognition, there exist a number of large-scale video
datasets [6,26,13,12,23,14], which allow the training of high-capacity deep learning models from massive amounts of data. These models enable detection of key


key: chroma:document
string_value:
datasets [6,26,13,12,23,14], which allow the training of high-capacity deep learning models from massive amounts of data. These models enable detection of key
cues present in videos, such as global and local motion, various object categories
and global scene-level information, and often achieve impressive performance in
recognizing high-level, abstract concepts in the wild.arXiv:1806.06157v3  [cs.CV]  20 Sep 2018
2 Baradel et al.
person 0.86person 0.87knife 0.98carrot 0.99carrot 0.72carrot 0.98
person 0.65knife 0.99carrot 0.97carrot 0.97carrot 0.93carrot 0.86carrot 0.95
Fig. 1. Humans can understand what happened in a video (\the leftmost carrot was
chopped by the person") given only a pair of frames. Along these lines, the goal of this
work is to explore the capabilities of higher-level reasoning in neural models operating
at the semantic level of objects and interactions.
However, recent attention has been directed toward a more thorough understanding of human-focused activity in diverse internet videos. These eorts
range from atomic human actions [14] to ne-grained object interactions [13]


key: chroma:document
string_value:
range from atomic human actions [14] to ne-grained object interactions [13]
to everyday, commonly occurring human-object interactions [12]. This returns
us to a human-centric viewpoint of activity recognition where it is not only the
presence of certain objects / scenes that dictate the activity present, but the
manner, order, and eects of human interaction with these scene elements that
are necessary for understanding. In a sense, this is akin to the problems in current
3D human activity recognition datasets [33], but requires the more challenging
reasoning and understanding of diverse environments common to internet video
collections.
Humans are able to infer what happened in a video given only a few sample frames. In particular, they can infer complex activities happening between
pairs of frames. This faculty is called reasoning and is a key component of human intelligence. As an example we can consider the pair of images in Figure 1,
which shows a complex situation involving articulated objects (human, carrots
and knife), the change of location and composition of objects. For humans it is
straightforward to draw a conclusion on what happened (a carrot was chopped
by the human). Humans have this extraordinary ability of performing visual reasoning on very complicated tasks while it remains unattainable for contemporary
computer vision algorithms [37,11].
The ability to perform visual reasoning in computer vision algorithms is still


key: chroma:document
string_value:
computer vision algorithms [37,11].
The ability to perform visual reasoning in computer vision algorithms is still
an open problem. Attempts have been made for learning interactions between
dierent entities in images with promising results on Visual Question Answering.
There have been a number of attempts to equip neural models with reasoning
abilities by training them to solve Visual Question Answering (VQA) problems.
Among proposed solutions are prior-less data normalization [27], structuring
networks to model relationships [32,43] as well as more complex attention based
Object Level Visual Reasoning in Videos 3
mechanisms [18]. At the same time, it was shown that high performance on existing VQA datasets can be achieved by simply discovering biases in the data [21].
We extend these eorts to object level reasoning in videos . Since a video is
a temporal sequence, we leverage time as an explicit causal signal to identify
causal object relations. Our approach is related to the concept of the \arrow of
the time" [28] involving the \one-way direction" or \asymmetry" of time. Causal
event occurs before the event it aects ( A!B). In Figure 1 the knife was used
before the carrot switched over to the chopped-up state on the right side. For


key: chroma:document
string_value:
event occurs before the event it aects ( A!B). In Figure 1 the knife was used
before the carrot switched over to the chopped-up state on the right side. For
a video classication problem, we want to identify a causal event Ahappening
in a video that aects its label B. But instead of identifying this causal event
directly from pixels we want to identify it from an object level perspective. We
believe that such an approach would be able to learn causal signals.
Following this hypothesis we propose to make a bridge between object detection and activity recognition. Object detection allows us to extract low-level
information from a scene with all the present object instances and their semantic
meanings. However, detailed activity understanding requires reasoning over these
semantic structures, determining which objects were involved in interactions, of
what nature, and what were the results of these. To compound problems, the
semantic structure of a scene may change during a video (e.g. a new object can
appear, a person may make a move from one point to another one of the scene).
We propose an Object Relation Network (ORN), a neural network module for reasoning between detected semantic object instances through space and
time. The ORN has potential to address these issues and conduct relational
reasoning over object interactions for the purpose of activity recognition. A set


key: chroma:document
string_value:
time. The ORN has potential to address these issues and conduct relational
reasoning over object interactions for the purpose of activity recognition. A set
of object detection masks ranging over dierent object categories and temporal
occurrences is input to the ORN. The ORN is able to infer pairwise relationships
between objects detected at varying dierent moments in time.
Code and object masks predictions will be publicly available6.
2 Related work
Action Recognition. Action recognition has a long history in computer vision. Pre-deep learning approaches in action recognition focused on handcrafted
spatio-temporal features including space-time interest points like SIFT-3D, HOG3D,
IDT and aggregated them using bag-of-words techniques. Some hand-crafted representations, like dense trajectories [42], still give competitive performance and
are frequently combined with deep learning.
In the recent past, work has shifted to deep learning. Early attempts adapt
2D convolutional networks to videos through temporal pooling and 3D convolutions [2,40]. 3D convolutions are now widely adopted for activity recognition with
the introduction of feature transfer by in
ating pre-trained 2D convolutional kernels from image classication models trained on ImageNet/ILSVRC [31] through
3D kernels [6]. The downside of 3D kernels is their computational complexity


key: chroma:document
string_value:
3D kernels [6]. The downside of 3D kernels is their computational complexity
6https://github.com/fabienbaradel/object_level_visual_reasoning
4 Baradel et al.
and the large number of learnable parameters, leading to the introduction of
2.5D kernels, i.e. separable lters in the form of a 2D spatial kernel followed by
a temporal kernel [44]. An alternative to temporal convolutions are Recurrent
Neural Networks (RNNs) in their various gated forms (GRUs, LSTMs) [17,9].
Karpathy et al. [20] presented a wide study on dierent ways of connecting
information in spatial and temporal dimensions through convolutions and pooling. On very general datasets with coarse activity classes they have showed that
there was a small margin between classifying individual frames and classifying
videos with more sophisticated temporal aggregation.
Simoyan et al. [35] proposed a widely adopted two-stream architecture for
action recognition which extracts two dierent streams, one processing raw RGB
input and one processing pre-computed optical 
ow images. The method outperformed the state of the art, but relies on rather small scale optical 
ow computations.
In slightly narrower settings, prior information on the video content can allow


key: chroma:document
string_value:
In slightly narrower settings, prior information on the video content can allow
more ne-grained models. Articulated pose is widely used in cases where humans
are guaranteed to be present [33]. Pose estimation and activity recognition as a
joint (multi-task) problem has recently shown to improve both tasks [25]. Somewhat related to our work, Structural RNNs [19] perform activity recognition
by integrating features from semantic objects and their relationships. However,
they handle the temporal evolution of tracked objects in videos with a set of
RNNs, each of which corresponds to cliques in a graph which models the spatiotemporal relationships between these objects. This graph is hand-crafted manually for each application, though related work provides learnable connections
via gating functions [8].
Attention models are a way to structure deep networks in an often generic
way. They are able to iteratively focus attention to specic parts in the data without requiring prior knowledge about part or object positions. In activity recognition, they have gained some traction in recent years, either as soft-attention
on articulated pose (joints) [36], on feature map cells [34,39], on time [45] or on
parts in raw RGB input through dierentiable crops [3].
When raw video data is globally fed into deep neural networks, they focus


key: chroma:document
string_value:
parts in raw RGB input through dierentiable crops [3].
When raw video data is globally fed into deep neural networks, they focus
on extracting spatio-temporal features and perform aggregations. It has been
shown that these techniques fail on challenging ne-grained datasets, which require learning long temporal dependencies and human-object interactions. A
concentrated eort has been made to create large scale datasets to overcome
these issues [13,12,23,14].
Relational Reasoning. Relational reasoning is a well studied eld for many
applications ranging from visual reasoning [32] to reasoning about physical
systems [4]. Battaglia et al. [4] introduce a fully-dierentiable network physics
engine called Interaction Network (IN). IN learns to predict several physical
systems such as gravitational systems, rigid body dynamics, and mass-spring
systems. It shows impressive results; however, it learns from a virtual environment, which provides access to virtually unlimited training examples. Following
the same perspective, Santoro et al. [32] introduced Relation Network (RN),
Object Level Visual Reasoning in Videos 5
a plug-in module for reasoning in deep networks. RN shows human-level performance in Visual Question Answering (VQA) by inferring pairwise \object"


key: chroma:document
string_value:
a plug-in module for reasoning in deep networks. RN shows human-level performance in Visual Question Answering (VQA) by inferring pairwise \object"
relations. However, in contrast to our work, the term \object" in [32] does not
refer to semantically meaningful entities, but to discrete cells in feature maps.
The number of interactions therefore grows with feature map resolutions, which
makes it dicult to scale. Furthermore, a recent study [21] has shown that some
of these results are subject to dataset bias and do not generalize well to small
changes in the settings of the dataset.
In the same line, a recent work [38] has shown promising results on discovering objects and their interactions in an unsupervised manner using training
examples from virtual environments. In [41], attention and relational modules
are combined on a graph structure. From a dierent perspective, [27] show that
relational reasoning can be learned for visual reasoning in a data driven way without any prior using conditional batch normalization with a feature-wise ane
transformation based on conditioning information. In an opposite approach, a
strong structural prior is learned in the form of a complex attention mechanism: in [18], an external memory module combined with attention processes
over input images and text questions, performing iterative reasoning for VQA.


key: chroma:document
string_value:
over input images and text questions, performing iterative reasoning for VQA.
While most of the discussed work has been designed for VQA and for predictions on physical systems and environments, extensions have been proposed
for video understanding. Reasoning in videos on a mask or segmentation level
has been attempted for video prediction [24], where the goal was to leverage semantic information to be able predict further into the future. Zhou et al [5] have
recently shown state-of-the-art performance on challenging datasets by extending Relation Network to video classication. Their chosen entities are frames, on
which they employ RN to reason on a temporal level only through pairwise frame
relations. The approach is promising, but restricted to temporal contextual information without an understanding on a local object level, which is provided
by our approach.
Reasoning over sets of objects is somewhat related to reasoning from unstructured data points, as done in PointNet [29], designed to learn from unordered sets
of points. PointNet shares many properties with DeepSet [46] which is a more
general framework for extracting information from sets of objects. To some extent, our work is related to PointNet, as we handle unordered sets of objects in a
permutation invariant way. However, we have an object relation viewpoint that
directly reasons over relationships between these semantic entities.
3 Object-level Visual Reasoning in Space and Time


key: chroma:document
string_value:
permutation invariant way. However, we have an object relation viewpoint that
directly reasons over relationships between these semantic entities.
3 Object-level Visual Reasoning in Space and Time
Our goal is to extract multiple types of cues from a video sequence: interactions
between predicted objects and their semantic classes, as well as local and global
motion in the scene. We formulate this objective as a neural architecture with
two heads: an activity head and an object head . Figure 2 gives a functional
overview of the model. Both heads share common features up to a certain layer
shown in red in the gure. The activity head , shown in orange in the gure,
6 Baradel et al.
spatio-temporalblock


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:
Fig. 2. A functional overview of the model. A global convolutional model extracts
features and splits into two heads trained to predict, respectively activity classes and
object classes. The latter are predicted by pooling over object instance masks, which
are predicted by an additional convolutional model. The object instances are passed
through a visual reasoning module.
is a CNN-based architecture employing convolutional layers, including spatiotemporal convolutions, able to extract global motion features. However, it is not
able to extract information from an object level perspective. We leverage the
object head to perform reasoning on the relationships between predicted object
instances.
Our main contribution is a new structured module called Object Relation
Network (ORN), which is able to perform spatio-temporal reasoning between
detected object instances in the video. ORN is able to reason by modeling how
objects move, appear and disappear and how they interact between two frames.
In this section, we will rst describe our main contribution, the ORN network.
We then provide details about object instance features, about the activity head,
and nally about the nal recognition task. In what follows, lowercase letters
denote 1D vectors while uppercase letters are used for 2D and 3D matrices or
higher order tensors. We assume that the input of our system is a video of T


key: chroma:document
string_value:
denote 1D vectors while uppercase letters are used for 2D and 3D matrices or
higher order tensors. We assume that the input of our system is a video of T
frames denoted by X1:T= (Xt)T
t=1where Xtis the RGB image at timestep t.
The goal is to learn a mapping from X1:Tto activity classes y.
3.1 Object Relation Network
ORN (Object Relation Network) is a module for reasoning between semantic
objects through space and time. It captures object moves, arrivals and interactions in an ecient manner. We suppose that for each frame t, we have a set
of objects kwith associated features ok
t. Objects and features are detected and
computed by the object head described in Section 3.2.
Reasoning about activities in videos is inherently temporal, as activities follow the arrow of time [28], i.e. the causality of the time dimension imposes that
past actions have consequences in the future but notvice-versa. We handle this
by sampling: running a process over time t, and for each instant t, sampling a
Object Level Visual Reasoning in Videos 7
second frame t0with t0<t. Our network reasons on objects which interact between pairs of frames and their corresponding sets of objects Ot0=
ok
t0	K0


key: chroma:document
string_value:
second frame t0with t0<t. Our network reasons on objects which interact between pairs of frames and their corresponding sets of objects Ot0=
ok
t0	K0
k=1
andOt=
ok
t	K
k=1. The goal is to learn a general function dened on the set of
all input objects from the combined set of both frames:
gt=g(o1
t0; : : : ;oK0
t0;o1
t; : : : ;oK
t): (1)
The objects in this set are unordered, aside for the frame they belong to.
This task is related to a problem raised in the PointNet algorithm [29] discussed in Section 2. PointNet approximates a general function gover an item
setS=fx1; x2; : : : ; xNgas a symmetric function g0on transformed elements of
the set
g(x1; x2; : : : ; xN)g0(h(x1); h(x2); : : : ; h (xN)): (2)
In [29], g0is a max pooling operation. The authors show that it allows universal


key: chroma:document
string_value:
In [29], g0is a max pooling operation. The authors show that it allows universal
approximation of continuous sets of functions given that the hidden representation (the output of the mapping h()) is of suciently high dimension.
We argue that the approximation in (2) can be extended as follows:
g(x1; x2; : : : ; xN)f X
c2Chc([i2cxi)!
(3)
whereCis the set of cliques of a graph dened over the item set S,[is the
concatenation operator and we chose the sum operator as symmetric function.
The input dimension of the non-linearity hc() depends on the size of clique c
but maps to a xed output dimension H. In the case where Cis composed of
unary cliques only, form (3) decomposes like (2) with the exception of a dierent
symmetry operator (sum instead of max pooling). Choosing dierent graphical
structures through Swill lead to dierent terms in the summation and allows
modeling dierent types of interactions between items in the item set.
Note that the interactions between items in (3) are not exclusively modeled


key: chroma:document
string_value:
modeling dierent types of interactions between items in the item set.
Note that the interactions between items in (3) are not exclusively modeled
through hc(). Indeed, it is interesting to note, that the graphical decomposition provided byCleads to interactions which are dierent from the interactions
the same decomposition would provide when used in a probabilistic graphical
model, like for instance a Markov Random Field (MRF). In particular, a decomposition into unary terms only, as given in equation (2), does notlead to
independence between items, whereas an MRF with unary terms only is equivalent to a distribution over independent random variables. This is a consequence
of the global mapping f(), which is dened on the sum over all direct interactions. Higher-order interactions between several items not directly modeled
through a non-linearity hc() can eventually be learned by the model through
the joint output space of all hc(), provided that the dimensionality Hof this
space is high enough to incorporate all interactions. However, whereas the mappinghc() provides a direct model of interactions between pairs of items, learning
interactions between two items ( j; k), which are not directly captured through a
8 Baradel et al.
visual reasoningmodule


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:
Fig. 3. ORN in the object head operating on detected instances of objects.
clique cand its corresponding hc(), requires learning a corresponding subspace
in the common output space spanned by all hc().
This leads to the question of how to dene the trade-o between the complexity of the decomposition Cand the output dimensionality Hof the mapping
hc(), both of which will determine the complexity of the modeled interactions.
Increasing the size of cliques in Cwill increase the input dimension (and therefore the capacity) of the mapping hc() as well as the computational complexity
of the sum operation.
Inspired by relational networks [32], we chose to directly model inter-frame
interactions between pairs of objects ( j; k) and leave modeling of higher-order
interactions to the output space of the mappings hand the global mapping f:
gt=X
j;kh(oj
t0;ok
t) (4)
In order to better directly model long-range interactions, we make the global
mapping f(;) recurrent, which leads to the following form:
rt=f(gt;rt1) (5)
where rtrepresents the recurrent object reasoning state at time tandgtis the


key: chroma:document
string_value:
rt=f(gt;rt1) (5)
where rtrepresents the recurrent object reasoning state at time tandgtis the
global inter-frame interaction inferred at time tsuch as described in Equation 4.
In practice, this is implemented as a GRU, but for simplicity we omitted the
gates in Equation (5). The pairwise mappings h(;) are implemented as an
MLP. Figure 3 provides a visual explanation of the object head's operating
through time.
Our proposed ORN diers from [32] in three main points:
Objects have a semantic denition | we model relationships with respect
to semantically meaningful entities (object instances) instead of feature map
cells which do not have a semantically meaningful spatial extent. We will show
in the experimental section that this is a key dierence.
Object Level Visual Reasoning in Videos 9
Objects are selected from dierent frames | we infer object pairwise
relations only between objects present in two dierent sets. This is a key design
choice which allows our model to reason about changes in object relationships
over time.
Long range reasoning | integration of the object relations over time is recurrent by using a RNN for f(). Since reasoning from a full sequence cannot


key: chroma:document
string_value:
over time.
Long range reasoning | integration of the object relations over time is recurrent by using a RNN for f(). Since reasoning from a full sequence cannot
be done by inferring the relations between two frames, f() allows long range
reasoning on sequences of variable length.
3.2 Object instance features
The object features Ot=
ok
t	K
k=1for each frame tused for the ORN module
described above are computed and collected from local regions predicted by
a mask predictor. Independently for each frame Xtof the input data block,
we predict object instances as binary masks Bk
tand associated object class
predictions ck
t, a distribution over Cclasses. We use Mask-RCNN [15], which
is able to detect objects in a frame using region proposal networks [30] and
produces a high quality segmentation mask for each object instance.
The objective is to collect features for each object instance, which jointly
describe its appearance, the change in its appearance over time, and its shape,
i.e. the shape of the binary mask. In theory, appearance could also be described
by pooling the feature representation learned by the mask predictor (Mask RCNN). However, in practice we choose to pool features from the dedicated object


key: chroma:document
string_value:
by pooling the feature representation learned by the mask predictor (Mask RCNN). However, in practice we choose to pool features from the dedicated object
head such as shown in Figure 2, which also include motion through the spatiotemporal convolutions shared with the activity head:
uk
t= ROI-Pooling( Ut;Bk
t) (6)
where Utis the feature map output by the object head ,uk
tis aD-dimensional
vector of appearance and appearance change of object k.
Shape information from the binary mask Bk
tis extracted through the following mapping function: bk
t=g(Bk
t);where g() is a MLP. Information about
object kin image Xtis given by a concatenation of appearance, shape, and
object class: ok
t= [bk
tuk
tck
t].
3.3 Global Motion and Context
Current approaches in video understanding focus on modeling the video from
a high-level perspective. By a stack of spatio-temporal convolution and pooling
they focus on learning global scene context information. Eective activity recognition requires integration of both of these sources: global information about the
entire video content in addition to relational reasoning for making ne distinctions regarding object interactions and properties.
In our method, local low-level reasoning is provided through object head


key: chroma:document
string_value:
entire video content in addition to relational reasoning for making ne distinctions regarding object interactions and properties.
In our method, local low-level reasoning is provided through object head
and the ORN module such as described above in Section 3.1. We complement
10 Baradel et al.
this representation by high-level context information described by Vtwhich are
feature outputs from the activity head (orange block in Figure 2).
We use spatial global average pooling over Vtto output T D-dimensional
feature vectors denoted by vt, where vtcorresponds to the context information
of the video at timestep t.
We model the dynamics of the context information through time by employing a RNN f
() given by:
st=f
(vt;st1) (7)
where sis the hidden state of f
() and gives cues about the evolution of the
context though time.
3.4 Recognition
Given an input video sequence X1:T, the two dierent streams corresponding to
the activity head and the object head result in the two representations handr,
respectively where h=P
thtandr=P
trt. Each representation is the hidden
state of the respective GRU, which were described in the preceding subsections.
Recall that hprovides the global motion context while rprovides the object


key: chroma:document
string_value:
state of the respective GRU, which were described in the preceding subsections.
Recall that hprovides the global motion context while rprovides the object
reasoning state output by the ORN module. We perform independent linear
classication for each representation:
y1=W h (8)
y2=Z r (9)
where y1;y2correspond to the logits from the activity head and the object head ,
respectively, and WandZare trainable weights (including biases). The nal
prediction is done by averaging logits y1andy2followed by softmax activation.
4 Network Architectures and feature dimensions
The input RGB images Xtare of size R3WHwhere WandHcorrespond to
the width and height and are of size 224 each. The object and activity heads
(orange and green in Figure 2) are a joint convolutional neural network with
Resnet50 architecture pre-trained on ImageNet/ILSVRC [31], with Conv1 and
Conv5 blocks being in
ated to 2.5D convolutions [44] (3D convolutions with a
separable temporal dimension). This choice has been optimized on the validation
set, as explained in Section 6 and shown in Table 5.
The last conv5 layers have been split into two dierent heads (activity head


key: chroma:document
string_value:
set, as explained in Section 6 and shown in Table 5.
The last conv5 layers have been split into two dierent heads (activity head
and object head). The intermediate feature representations UtandVtare of dimensions 2048T1414 and 2048T77, respectively. We provide a higher
spatial resolution for the feature maps Utof the object head to get more precise
local descriptors. This can be done by changing the stride of the initial conv5
layers from 2 to 1. Temporal convolutions have been congured to keep the same
time temporal dimension through the network.
Object Level Visual Reasoning in Videos 11
Global spatial pooling of activity features results in a 2048 dimensional feature vector fed into a GRU with 512 dimensional hidden state st. ROI-Pooling
of object features results in 2048 dimensional feature vectors uk
t. The encoder of
the binary mask is a MLP with one hidden layer of size 100 and outputs a mask
embedding bk
tof dimension 100. The number of object classes is 80, which leads
in total to a 2229 dimensional object feature vector ok
t.
The non-linearity h() is implemented as an MLP with 2 hidden layers each


key: chroma:document
string_value:
in total to a 2229 dimensional object feature vector ok
t.
The non-linearity h() is implemented as an MLP with 2 hidden layers each
with 512 units and produces an 512 dimensional output space. f() is implemented as a GRU with a 256 dimension hidden state rt. We use ReLU as the
activation function after each layer for each network.
5 Training
We train the model with a loss split into two terms:
Ltot=L^y1+^y2
2;y
+X
tX
kL(^ck
t;ck
t): (10)
whereLis the cross-entropy loss. The rst term corresponds to supervised activity class losses comparing two dierent activity class predictions to the class
ground truth: ^y1is the prediction of the activity head, whereas ^y2is the prediction of the object head, as given by Equations (8) and (9), respectively.
The second term is a loss which pushes the features Uof the object towards
representations of the semantic object classes. The goal is to obtain features
related to, both, motion (through the layers shared with the activity head), as
well as as object classes. As ground-truth object classes are not available, we
dene the loss as the cross-entropy between the class label ck


key: chroma:document
string_value:
well as as object classes. As ground-truth object classes are not available, we
dene the loss as the cross-entropy between the class label ck
tpredicted by the
mask predictor and a dedicated linear class prediction ^ck
tbased on features uk
t,
which, as we recall, are RoI-pooled from U:
ck
t=R uk
t (11)
where Rtrainable parameters (biases integrated) learned end-to-end together
with the other parameters of the model.
We found that rst training the object head only and then the full network
was performing better. A ResNet50 network pretrained on ImageNet is modied
by in
ating some of its lters to 2.5 convolutions (3D convolutions with the time
dimension separated), as described in Section 4; then by ne-tuning.
We train the model using the Adam optimizer [22] with an initial learning
rate of 104on 30 epochs and use early-stopping criterion on the validation set
for hyper-parameter optimization. Training takes 50 minutes per epoch on 4
Titan XP GPUs with clips of 8 frames.
12 Baradel et al.
6 Experimental results
We evaluated the method on three standard datasets, which represent dicult


key: chroma:document
string_value:
Titan XP GPUs with clips of 8 frames.
12 Baradel et al.
6 Experimental results
We evaluated the method on three standard datasets, which represent dicult
ne-grained activity recognition tasks: the Something-Something dataset, the
VLOG dataset and the recently released EPIC Kitchens dataset.
Something-Something (SS) is a recent video classication dataset with
108,000 example videos and 157 classes [13]. It shows humans performing dierent actions with dierent objects, actions and objects being combined in dierent
ways. Solving SS requires common sense reasoning and the state-of-the-art methods in activity recognition tend to fail, which makes this dataset challenging.
VLOG is a multi-label binary classication of human-object interactions
recently released with 114,000 videos and 30 classes [12]. Classes correspond
to objects, and labels of a class are 1 if a person has touched a certain object
during the video, otherwise they are 0. It has recently been shown, that stateof-the-art video based methods [6] are outperformed on VLOG by image based
methods like ResNet-50 [16], although these video methods outperform image
based ResNet-50 on large-scale video datasets like the Kinetics dataset [6]. This


key: chroma:document
string_value:
methods like ResNet-50 [16], although these video methods outperform image
based ResNet-50 on large-scale video datasets like the Kinetics dataset [6]. This
suggests a gap between traditional datasets like Kinetics and the ne-grained
dataset VLOG, making it particularly dicult.
EPIC Kitchens (EPIC) is an egocentric video dataset recently released
containing 55 hours recording of daily activities [7]. This is the largest in rstperson vision and the activities performed are non-scripted, which makes the
dataset very challenging and close to real world data. The dataset is densely
annotated and several tasks exist such as object detection, action recognition and
action prediction. We focus on action recognition with 39'594 action segments
in total and 125 actions classes (i.e verbs). Since the test set is not available yet
we conducted our experiments on the training set (28'561 videos). We use the
videos recorded by person 01 to person 25 for training (22'675 videos) and dene
the validation set as the remaining videos (5'886 videos).
For all datasets we rescale the input video resolution to 256 256. While
training, we crop space-time blocks of 224 224 spatial resolution and Lframes,


key: chroma:document
string_value:
For all datasets we rescale the input video resolution to 256 256. While
training, we crop space-time blocks of 224 224 spatial resolution and Lframes,
withL=8 for the SS dataset and L=4 for VLOG and EPIC. We do not perform
any other data augmentation. While training we extract Lframes from the entire
video by splitting the video into Lsub-sequences and randomly sampling one
frame per sub-sequence. The output sequence of size Lis called a clip. A clip
aims to represent the full video with less frames. For testing we aggregate results
of 10 clips. We use lintel [10] for decoding video on the 
y.
The ablation study is done by using the train set as training data and we
report the result on the validation set. We compare against other state-of-theart approaches on the test set. For the ablation studies, we slightly decreased
the computational complexity of the model: the base network (including activity
and object heads) is a ResNet-18 instead of ResNet-50, a single clip of 4 frames
is extracted from a video at test time.
Comparison with other approaches. Table 1 shows the performance of
the proposed approach on the VLOG dataset. We outperform the state of the art
Object Level Visual Reasoning in Videos 13


key: chroma:document
string_value:
Comparison with other approaches. Table 1 shows the performance of
the proposed approach on the VLOG dataset. We outperform the state of the art
Object Level Visual Reasoning in Videos 13
Table 1. Results on Hand/Semantic Object Interaction Classication (Average precision in % on the test set) on VLOG dataset. R50 and I3D implemented by [12].
mAP
bag
bed
bedding
book/papers
bottle/tube
bowl
box
brush
cabinet
cell-phone
clothing
cup
door
drawers
food
fork
knife
laptop
microwave
oven
pen/pencil
pillow
plate
refrigerator
sink
spoon
stued animal
table
toothbrush
towel
R50 [16] 40.5 29.7 68.9 65.8 64.5 58.2 33.1 22.1 19.0 23.9 54.0 45.5 28.6 49.2 28.7 49.6 19.4 37.5 62.9 48.8 23.0 36.9 39.2 12.5 55.9 58.8 31.1 57.4 26.8 39.6 22.9


key: chroma:document
string_value:
I3D [6] 39.7 24.9 71.7 71.4 62.5 57.1 27.1 19.2 33.9 20.7 50.6 45.8 24.7 54.7 19.1 50.8 19.3 41.9 54.0 27.5 21.4 37.4 42.9 12.6 42.5 60.4 33.9 46.0 23.5 59.6 34.7
Ours 44.7 30.2 72.3 70.7 64.9 59.8 38.2 24.6 26.3 22.4 64.5 47.2 35.4 57.9 25.2 48.5 24.5 40.2 72.0 54.1 26.5 39.9 48.6 15.2 53.5 60.7 36.8 52.8 27.9 64.0 37.6
on this challenging dataset by a margin of 4.2 points (44.7% accuracy against
40.5% by [16]). As mentioned above, traditional video approaches tend to fail
on this challenging ne-grained dataset, providing inferior results. Table 3 shows
performance on SS where we outperform the state of the art given by very recent
methods (+2.3 points). On EPIC we re-implement standard baselines and report


key: chroma:document
string_value:
performance on SS where we outperform the state of the art given by very recent
methods (+2.3 points). On EPIC we re-implement standard baselines and report
results on the validation set (Table 4) since the test set is not available. Our full
method reports an accuracy of 40.89 and outperforms baselines by a large margin
(+6.4 and+7.9 points respectively for against CNN-2D and I3D based on a
ResNet-18).
Eect of object-level reasoning. Table 2 shows the importance of reasoning on the performance of the method. The baseline corresponds to the performance obtained by the activity head trained alone (in
ated ResNet, in the
ResNet-18 version for this table). No object level reasoning is present in this
baseline. The proposed approach (third line) including an object head and the
ORN module gains 0.8, 2.5 and 2.4 points compared to our baseline respectively
on SS, on EPIC and on VLOG. This indicates that the reasoning module is able
to extract complementary features compared to the activity head.
Using semantically dened objects proved to be important and led to a gain of
2 points on EPIC and 2.3 points on VLOG for the full model (6/12.7 points using


key: chroma:document
string_value:
2 points on EPIC and 2.3 points on VLOG for the full model (6/12.7 points using
the object head only) compared to an extension of Santoro et al. [32] operating
on pixel level. This indicates importance of object level reasoning. The gain on
SS is smaller (0.7 point with the full model and 7.8 points with the object head
only) and can be explained by the dierence in spatial resolution of the videos.
Object detections and predictions of the binary masks are done using the initial
video resolution. The mean video resolution for VLOG is 660 1183 and for EPIC
is 640480 against 100157 for SS. Mask-RCNN has been trained on images of
resolution 800800 and thus performs best on higher resolutions. The quality of
the object detector is important for leveraging object level understanding then
for the rest of the ablation study we focus on EPIC and VLOG datasets.
The function fin Equation (5) is an important design choice in our model.
In our proposed model, fis recurrent over time to ensure that the ORN module
captures long range reasoning over time, as shown in Equation (5). Removing
the recurrence in this equation leads to an MLP instead of a (gated) RNN, as


key: chroma:document
string_value:
the recurrence in this equation leads to an MLP instead of a (gated) RNN, as
evaluated in row 4 of Table 2. Performance decreases by 1.1 point on VLOG
and 1.4 points on EPIC. The larger gap for EPIC compared to VLOG and
can arguably be explained by the fact that in SS actions cover the whole video,
14 Baradel et al.
Table 2. Ablation study with ResNet-18 backbone. Results in %: Top-1 accuracy for
EPIC and SS datasets, and mAP for VLOG dataset.
Method Object typeEPIC VLOG SS
obj. head 2 heads obj. head 2 heads obj. head 2 heads
Baseline - - 38.33 - 35.03 - 31.31
ORN pixel 23.71 38.83 14.40 35.18 2.51 31.43
ORN COCO 29.94 40.89 27.14 37.49 10.26 32.12


key: chroma:document
string_value:
ORN COCO 29.94 40.89 27.14 37.49 10.26 32.12
ORN-mlp COCO 28.15 39.41 25.40 36.35 - ORN COCO-visual 28.45 38.92 22.92 35.49 - ORN COCO-shape 21.92 37.16 7.18 35.39 - ORN COCO-class 21.96 37.75 13.40 35.94 - ORN COCO-intra 29.25 38.10 26.78 36.28 - ORN clique-1 COCO 28.25 40.18 26.48 36.71 - ORN clique-3 COCO 22.61 37.67 27.05 36.04 - Table 3. Experimental results on the
Something-Something dataset (classication accuracy in % on the test set).
Methods Top1
C3D + Avg [13] 21.50
I3D [13] 27.63
MultiScale TRN [5] 33.60
Ours 35.97Table 4. Experimental results on the
EPIC Kitchens dataset (accuracy in %
on the validation set { methods with
have been re-implemented).
Methods Top1
R18 [16]32.05
I3D-18 [6]34.20


key: chroma:document
string_value:
have been re-implemented).
Methods Top1
R18 [16]32.05
I3D-18 [6]34.20
Ours 40.89
while solving VLOG requires detecting the right moment when the human-object
interaction occurs and thus long range reasoning plays a less important role.
Visual features extracted from object regions are the most discriminative,
however object shapes and labels also provide complementary information. Finally, the last part of Table 2 evaluates the eect of the cliques size for modeling the interactions between objects and show that pairwise cliques outperform
cliques of size 1 and 3. We would like to recall, that even with unary cliques
only, interactions between objects are still modeled. However, the model needs
to nd subspaces in the hidden representations associated to each interaction.
CNN architecture and kernel in
ations. The convolutional architecture
of the model was optimized over the validation set of the SS dataset, as shown
in Table 5. The architecture itself (in terms of numbers of layers, lters etc.)
is determined by pre-training on image classication. We optimized the choice
of lter in
ations from 2D to 2.5D or 3D for several convolutional blocks. This
has been optimized for the single head model and using a ResNet-18 variant


key: chroma:document
string_value:
has been optimized for the single head model and using a ResNet-18 variant
to speed up computation. Adding temporal convolutions increases performance
up to 100% w.r.t. to pure 2D baselines. This indicates, without surprise, that
motion is a strong cue. In
ating kernels to 2.5D on the input side and on the
Object Level Visual Reasoning in Videos 15
Table 5. Eect of the CNN architecture (choice of kernel in
ations) on a single head
ResNet-18 network. Accuracy in % on the validation set of Something-Something is
shown. 2.5D kernels are separable kernels: 2D followed by a 1D temporal.
Conv1 Conv2 Conv3 Conv4 Conv5 AggregSS2D 3D 2.5D 2D 3D 2.5D 2D 3D 2.5D 2D 3D 2.5D 2D 3D 2.5D GAP RNN
X- - X- - X- - X- - X- - X -15.73
X- - X- - X- - X- - X- - -X 15.88
-X - -X - -X - -X - -X -X -31.42
- - X - - X - - X - - X - - XX -27.58


key: chroma:document
string_value:
-X - -X - -X - -X - -X -X -31.42
- - X - - X - - X - - X - - XX -27.58
X- - X- - X- - X- - -X -X -31.28
X- - X- - X- - -X - -X -X -32.06
X- - X- - -X - -X - -X -X -32.25
X- - X- - X- - X- - - - XX -31.31
X- - X- - X- - - - X - - XX -32.79
X- - X- - - - X - - X - - XX -33.77
-X -X- - X- - X- - X- - X -28.71
-X - -X -X- - X- - X- - X -31.42
- - XX- - X- - X- - X- - X -20.05
- - X - - XX- - X- - X- - X -22.52
output side provided best performances, suggesting that temporal integration
is required at a very low level (motion estimation) as well as on a very high
level, close to reasoning. Our study also corroborates recent research in activity


key: chroma:document
string_value:
is required at a very low level (motion estimation) as well as on a very high
level, close to reasoning. Our study also corroborates recent research in activity
recognition, indicating that 2.5D kernels provide a good trade-o between highcapacity and learnable numbers of parameters. Finally temporal integration via
RNN outperforms global average pooling over space and time. The choice of
a (gated) RNN for temporal integration of the activity head features proved
important (see Table 5) compared to global average pooling (GAP) over space
and time.
Visualizing the learned object interactions. Figure 4 shows visualizations of the pairwise object relationships the model learned from data, in particular from the VLOG dataset. Each graph is computed for a given activity class,
and strong arcs between two nodes in the graph indicate strong relationships
between the object classes, i.e. the model detects a high correlation between
these relationships and the activity. The graphs were obtained by thresholding
the summed activations of each pairwise relationship ( j; k) in equation (4). Each
pair ( j; k) can be assigned a pair of object classes cj
tandck
tthrough the predictions of the object instance mask predictor. Integrating over all samples of the
dataset for a given class leads to the visualizations in Figure 4. We can see that


key: chroma:document
string_value:
tthrough the predictions of the object instance mask predictor. Integrating over all samples of the
dataset for a given class leads to the visualizations in Figure 4. We can see that
the object interactions are highly relevant to the detected activities: the persontouches-bed activity is correlated to interactions between relevant object classes
person and bed. Similarly, activities human-bowl interaction and human-cup interaction show interactions with the respective objects bowl and cup. Moreover,
other recovered relationships are highly correlated to the scene (for example,
dining-table and bowl for activity human-bowl interaction ).
16 Baradel et al.
Fig. 4. Example of object pairwise interactions learned by our model on VLOG for
four dierent classes. Objects co-occurrences are at the top and learned pairwise objects
interactions are at the bottom. Line thickness indicates learned importance of a given
relation. Interactions have been normalized by the object co-occurrences.
teddy bearteddy bearteddy bearbedcell phonepersonpersonbedteddy bear
bottlewine glasspersonpersonpersonpersonbottle
Fig. 5. Examples of failure cases { a) small sized objects (on the left). Our model
detects a cell phone and a person but fails to detect hand-cell-phone contact ; b) confusion between semantically similar objects (on the right). The model falsly predicts


key: chroma:document
string_value:
detects a cell phone and a person but fails to detect hand-cell-phone contact ; b) confusion between semantically similar objects (on the right). The model falsly predicts
hand-cup contact instead of hand-glass-contact even though the wine glass is detected.
Finally, Figure 5 shows some failure cases, which are either due to errors
done by the object mask prediction (Mask R-CNN) or by the ORN itself.
7 Conclusion
We presented a method for activity recognition in videos which leverages object instance detections for visual reasoning on object interactions over time.
The choice of reasoning over semantically well-dened objects is key to our approach and outperforms state of the art methods which reason on grid-levels,
such as cells of convolutional feature maps. Temporal dependencies and causal
relationships are dealt with by integrating relationships between dierent time
instants. We evaluated the method on three dicult datasets, on which standard
approaches do not perform well, and report state-of-the-art results.
Acknowledgements. This work was funded by grant Deepvision (ANR-15CE23-0029, STPGP-479356-15), a joint French/Canadian call by ANR & NSERC.
Object Level Visual Reasoning in Videos 17
References


key: chroma:document
string_value:
Object Level Visual Reasoning in Videos 17
References
1. Abu-El-Haija, S., Kothari, N., Lee, J., Natsev, P., Toderici, G., Varadarajan, B.,
Vijayanarasimhan, S.: Youtube-8m: A large-scale video classication benchmark.
arXiv preprint arxiv:1609.08675 (2016)
2. Baccouche, M., Mamalet, F., Wolf, C., Garcia, C., Baskurt, A.: Sequential deep
learning for human action recognition. In: HBU (2011)
3. Baradel, F., Wolf, C., Mille, J., Taylor, G.: Glimpse clouds: Human activity recognition from unstructured feature points. In: CVPR (2018)
4. Battaglia, P.W., Pascanu, R., Lai, M., Rezende, D.J., Kavukcuoglu, K.: Interaction
networks for learning about objects, relations and physics. In: NIPS (2016)
5. Bolei, Z., Zhang, A.A., Torralba, A.: Temporal relational reasoning in videos. In:
ECCV (2018)


key: chroma:document
string_value:
5. Bolei, Z., Zhang, A.A., Torralba, A.: Temporal relational reasoning in videos. In:
ECCV (2018)
6. Carreira, J., Zisserman, A.: Quo vadis, action recognition? a new model and the
kinetics dataset. In: CVPR (2017)
7. Damen, D., Doughty, H., Farinella, G.M., Fidler, S., Furnari, A., Kazakos, E.,
Moltisanti, D., Munro, J., Perrett, T., Price, W., Wray, M.: Scaling egocentric
vision: The epic-kitchens dataset. In: ECCV (2018)
8. Deng, Z., Vahdat, A., Hu, H., Mori, G.: Structure inference machines: Recurrent
neural networks for analyzing relations in group activity recognition. In: CVPR
(2016)
9. Donahue, J., Hendricks, L.A., Guadarrama, S., Rohrbach, M., Venugopalan, S.,
Saenko, K., Darrell, T.: Long-term recurrent convolutional networks for visual
recognition and description. In: CVPR (2015)


key: chroma:document
string_value:
Saenko, K., Darrell, T.: Long-term recurrent convolutional networks for visual
recognition and description. In: CVPR (2015)
10. Duke, B.: Lintel: Python video decoding. https://github.com/dukebw/lintel
(2018)
11. Fleuret, F., Li, T., Dubout, C., Wampler, E.K., Yantis, S., Geman, D.: Comparing
machines and humans on a visual categorization test. Proceedings of the National
Academy of Sciences of the United States of America 108 43 , 17621{5 (2011)
12. Fouhey, D.F., Kuo, W., Efros, A.A., Malik, J.: From lifestyle vlogs to everyday
interactions. In: CVPR (2018)
13. Goyal, R., Ebrahimi Kahou, S., Michalski, V., Materzynska, J., Westphal, S., Kim,
H., Haenel, V., Fruend, I., Yianilos, P., Mueller-Freitag, M., Hoppe, F., Thurau,
C., Bax, I., Memisevic, R.: The "something something" video database for learning
and evaluating visual common sense. In: ICCV (2017)


key: chroma:document
string_value:
C., Bax, I., Memisevic, R.: The "something something" video database for learning
and evaluating visual common sense. In: ICCV (2017)
14. Gu, C., Sun, C., Ross, D.A., Vondrick, C., Pantofaru, C., Li, Y., Vijayanarasimhan,
S., Toderici, G., Ricco, S., Sukthankar, R., Schmid, C., Malik, J.: Ava: A
video dataset of spatio-temporally localized atomic visual actions. arXiv preprint
arXiv:1705.08421 (2017)
15. He, K., Gkioxari, G., Dollar, P., Girshick, R.: Mask r-cnn. In: ICCV (2017)
16. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition.
In: CVPR (2016)
17. Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Computation
9(8), 1735{1780 (1997)
18. Hudson, D., Manning, C.: Compositional attention networks for machine reasoning.
In: ICLR (2018)


key: chroma:document
string_value:
9(8), 1735{1780 (1997)
18. Hudson, D., Manning, C.: Compositional attention networks for machine reasoning.
In: ICLR (2018)
19. Jain, A., Zamir, A.R., Savarese, S., Saxena, A.: Structural-RNN: Deep Learning
on Spatio-Temporal Graphs. In: CVPR (2016)
20. Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., Fei-Fei, L.: Largescale video classication with convolutional neural networks. In: CVPR (2014)
18 Baradel et al.
21. Kim, J., Ricci, M., Serre, T.: Not-so-CLEVR: Visual relations strain feedforward
neural networks. arXiv preprint arXiv:1802.03390 (2018)
22. Kingma, D., Ba, J.: Adam: A method for stochastic optimization. In: ICML (2015)


key: chroma:document
string_value:
22. Kingma, D., Ba, J.: Adam: A method for stochastic optimization. In: ICML (2015)
23. Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalanditis, Y., Li, L.J., Shamma, D.A., Bernstein, M., Fei-Fei, L.: Visual genome:
Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision (IJCV) 123, 32{73 (2017)
24. Luc, P., , Neverova, N., Couprie, C., Verbeek, J., LeCun, Y.: Predicting deeper
into the future of semantic segmentation. In: ICCV (2017)
25. Luvizon, D., Picard, D., Tabia, H.: 2d/3d pose estimation and action recognition
using multitask deep learning. In: CVPR (2018)
26. Monfort, M., Zhou, B., Bargal, S.A., Andonian, A., Yan, T., Ramakrishnan,
K., Brown, L., Fan, Q., Gutfruend, D., Vondrick, C., Oliva, A.: Moments
in time dataset: one million videos for event understanding. arXiv preprint


key: chroma:document
string_value:
in time dataset: one million videos for event understanding. arXiv preprint
arXiv:1801.03150 (2018)
27. Perez, E., Vries, H.D., Strub, F., Dumoulin, V., Courville, A.: Learning visual reasoning without strong priors. In: ICML Machine Learning in Speech and Language
Processing Workshop (2017)
28. Pickup, L.C., Pan, Z., Wei, D., Shih, Y., Zhang, C., Zisserman, A., Scholkopf, B.,
Freeman, W.T.: Seeing the arrow of time. In: CVPR (2014)
29. Qi, C.R., Su, H., Mo, K., Guibas, L.J.: Pointnet: Deep learning on point sets for
3d classication and segmentation. In: CVPR (2017)
30. Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: Towards real-time object
detection with region proposal networks. In: NIPS (2015)
31. Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z.,


key: chroma:document
string_value:
31. Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z.,
Karpathy, A., Khosla, A., Bernstein, M., Berg, A., Fei-Fei, L.: Imagenet large scale
visual recognition challenge. IJCV 115(3), 211{252 (2015)
32. Santoro, A., Raposo, D., Barrett, D.G., Malinowski, M., Pascanu, R., Battaglia,
P., Lillicrap, T.: A simple neural network module for relational reasoning. In: NIPS
(2017)
33. Shahroudy, A., Liu, J., Ng, T.T., Wang, G.: NTU RGB+D: A Large Scale Dataset
for 3D Human Activity Analysis. In: CVPR (2016)
34. Sharma, S., Kiros, R., Salakhutdinov, R.: Action recognition using visual attention.
In: ICLR Workshop (2016)
35. Simonyan, K., Zisserman, A.: Two-stream convolutional networks for action recognition in videos. In: NIPS (2014)


key: chroma:document
string_value:
35. Simonyan, K., Zisserman, A.: Two-stream convolutional networks for action recognition in videos. In: NIPS (2014)
36. Song, S., Lan, C., Xing, J., Zeng, W., Liu, J.: An End-to-End Spatio-Temporal
Attention Model for Human Action Recognition from Skeleton Data. In: AAAI
(2016)
37. Stabinger, S., Rodr guez-S anchez, A., Piater, J.: 25 years of CNNs: Can we compare
to human abstraction capabilities? In: ICANN (2016)
38. van Steenkiste, S., Chang, M., Gre, K., Schmidhuber, J.: Relational neural expectation maximization: Unsupervised discovery of objects and their interactions.
In: ICLR (2018)
39. Sun, L., Jia, K., Chen, K., Yeung, D., Shi, B.E., Savarese, S.: Lattice long shortterm memory for human action recognition. In: ICCV (2017)
40. Tran, D., Bourdev, L., Fergus, R., Torresani, L., Paluri, M.: Learning spatiotemporal features with 3d convolutional networks. In: ICCV (2015)


key: chroma:document
string_value:
41. Velikovi, P., Cucurull, G., Casanova, A., Romero, A., Li, P., Bengio, Y.: Graph
attention networks. In: ICLR (2018)
Object Level Visual Reasoning in Videos 19
42. Wang, H., Kl aser, A., Schmid, C., Liu, C.L.: Action Recognition by Dense Trajectories. In: CVPR (2011)
43. Watters, N., Zoran, D., Weber, T., Battaglia, P., Pascanu, R., Tacchetti, A.: Visual
interaction networks: Learning a physics simulator from video. In: NIPS (2017)
44. Xie, S., Sun, C., Huang, J., Tu, Z., Murphy, K.: Rethinking spatiotemporal feature
learning for video understanding. arXiv prepring arxiv:1712.04851 (2017)
45. Yeung, S., Russakovsky, O., Jin, N., Andriluka, M., Mori, G., Fei-Fei, L.: Every moment counts: Dense detailed labeling of actions in complex videos. arXiv preprint
arXiv:1507.05738 (2015)


key: chroma:document
string_value:
arXiv:1507.05738 (2015)
46. Zaheer, M., Kottur, S., Ravanbakhsh, S., Poczos, B., Salakhutdinov, R.R., Smola,
A.J.: Deep sets. In: NIPS. pp. 3391{3401 (2017), http://papers.nips.cc/paper/
6931-deep-sets.pdf


key: chroma:document
string_value:
Massively Multilingual Neural Machine Translation
Roee Aharoni
Bar Ilan University
Ramat-Gan
Israel
roee.aharoni@gmail.comMelvin Johnson andOrhan Firat
Google AI
Mountain View
California
melvinp,orhanf@google.com
Abstract
Multilingual neural machine translation
(NMT) enables training a single model that
supports translation from multiple source languages into multiple target languages. In this
paper, we push the limits of multilingual NMT
in terms of the number of languages being
used. We perform extensive experiments in
training massively multilingual NMT models,
translating up to 102 languages to and from
English within a single model. We explore
different setups for training such models and
analyze the trade-offs between translation
quality and various modeling decisions. We
report results on the publicly available TED
talks multilingual corpus where we show
that massively multilingual many-to-many
models are effective in low resource settings,
outperforming the previous state-of-the-art
while supporting up to 59 languages. Our
experiments on a large-scale dataset with
102 languages to and from English and up to
one million examples per direction also show
promising results, surpassing strong bilingual
baselines and encouraging future work on
massively multilingual NMT.
1 Introduction


key: chroma:document
string_value:
one million examples per direction also show
promising results, surpassing strong bilingual
baselines and encouraging future work on
massively multilingual NMT.
1 Introduction
Neural machine translation (NMT) (Kalchbrenner and Blunsom, 2013; Bahdanau et al., 2014;
Sutskever et al., 2014) is the current state-ofthe-art approach for machine translation in both
academia (Bojar et al., 2016, 2017, 2018) and industry (Wu et al., 2016; Hassan et al., 2018). Recent works (Dong et al., 2015; Firat et al., 2016a;
Ha et al., 2016; Johnson et al., 2017) extended the
approach to support multilingual translation, i.e.
training a single model that is capable of translating between multiple language pairs.
Multilingual models are appealing for several
reasons. First, they are more efﬁcient in terms
Work carried out during an internship at Google AI.of the number of required models and model parameters, enabling simpler deployment. Another
beneﬁt is transfer learning; when low-resource
language pairs are trained together with highresource ones, the translation quality may improve
(Zoph et al., 2016; Nguyen and Chiang, 2017). An
extreme case of such transfer learning is zero-shot


key: chroma:document
string_value:
(Zoph et al., 2016; Nguyen and Chiang, 2017). An
extreme case of such transfer learning is zero-shot
translation (Johnson et al., 2017), where multilingual models are able to translate between language
pairs that were never seen during training.
While very promising, it is still unclear how far
one can scale multilingual NMT in terms of the
number of languages involved. Previous works
on multilingual NMT typically trained models
with up to 7 languages (Dong et al., 2015; Firat et al., 2016b; Ha et al., 2016; Johnson et al.,
2017; Gu et al., 2018) and up to 20 trained directions (Cettolo et al., 2017) simultaneously. One
recent exception is Neubig and Hu (2018) who
trained many-to-one models from 58 languages
into English. While utilizing signiﬁcantly more
languages than previous works, their experiments
were restricted to many-to-one models in a lowresource setting with up to 214k examples per
language-pair and were evaluated only on four
translation directions.
In this work, we take a step towards practical
“universal” NMT – training massively multilingual models which support up to 102 languages
and with up to one million examples per languagepair simultaneously. Speciﬁcally, we focus on


key: chroma:document
string_value:
and with up to one million examples per languagepair simultaneously. Speciﬁcally, we focus on
training “English-centric” many-to-many models,
in which the training data is composed of many
language pairs that contain English either on the
source side or the target side. This is a realistic
setting since English parallel data is widely available for many language pairs. We restrict our experiments to Transformer models (Vaswani et al.,
2017) as they were shown to be very effective
in recent benchmarks (Ott et al., 2018), also inarXiv:1903.00089v3  [cs.CL]  2 Jul 2019
the context of multilingual models (Lakew et al.,
2018; Sachan and Neubig, 2018).
We evaluate the performance of such massively
multilingual models while varying factors like
model capacity, the number of trained directions
(tasks) and low-resource vs. high-resource settings. Our experiments on the publicly available
TED talks dataset (Qi et al., 2018) show that massively multilingual many-to-many models with up
to 58 languages to-and-from English are very effective in low resource settings, allowing to use
high-capacity models while avoiding overﬁtting


key: chroma:document
string_value:
to 58 languages to-and-from English are very effective in low resource settings, allowing to use
high-capacity models while avoiding overﬁtting
and achieving superior results to the current stateof-the-art on this dataset (Neubig and Hu, 2018;
Wang et al., 2019) when translating into English.
We then turn to experiment with models trained
on 103 languages in a high-resource setting. For
this purpose we compile an English-centric inhouse dataset, including 102 languages aligned
to-and-from English with up to one million examples per language pair. We then train a single model on the resulting 204 translation directions and ﬁnd that such models outperform strong
bilingual baselines by more than 2 BLEU averaged across 10 diverse language pairs, both toand-from English. Finally, we analyze the tradeoffs between the number of involved languages
and translation accuracy in such settings, showing
that massively multilingual models generalize better to zero-shot scenarios. We hope these results
will encourage future research on massively multilingual NMT.
2 Low-Resource Setting: 59 Languages
2.1 Experimental Setup
The main question we wish to answer in this work
is how well a single NMT model can scale to
support a very large number of language pairs.
The answer is not trivial: on the one hand, training multiple language pairs together may result


key: chroma:document
string_value:
is how well a single NMT model can scale to
support a very large number of language pairs.
The answer is not trivial: on the one hand, training multiple language pairs together may result
in transfer learning (Zoph et al., 2016; Nguyen
and Chiang, 2017). This may improve performance as we increase the number of language
pairs, since more information can be shared between the different translation tasks, allowing the
model to learn which information to share. On the
other hand, adding many language pairs may result in a bottleneck; the model has a limited capacity while it needs to handle this large number
of translation tasks, and sharing all parameters between the different languages can be sub-optimal(Wang et al., 2018) especially if they are not from
the same typological language family (Sachan and
Neubig, 2018).
We begin tackling this question by experimenting with the TED Talks parallel corpus compiled
by Qi et al. (2018)1, which is unique in that it includes parallel data from 59 languages. For comparison, this is signiﬁcantly “more multilingual”
than the data available from all previous WMT
news translation shared task evaluations throughout the years – the latest being Bojar et al. (2016,
2017, 2018), which included 14 languages so far.2
We focus on the setting where we train


key: chroma:document
string_value:
2017, 2018), which included 14 languages so far.2
We focus on the setting where we train
“English-centric” models, i.e. training on all
language pairs that contain English in either the
source or the target, resulting in 116 translation
directions. This dataset is also highly imbalanced, with language pairs including between 3.3k
to 214k sentence pairs for training. Table 9 in
the supplementary material details the languages
and training set sizes for this dataset. Since the
dataset is already tokenized we did not apply additional preprocessing other than applying joint
subword segmentation (Sennrich et al., 2016) with
32k symbols.
Regarding the languages we evaluate on, we begin with the same four languages as Neubig and
Hu (2018) – Azerbeijani (Az), Belarusian (Be),
Galician (Gl) and Slovak (Sk). These languages
present an extreme low-resource case, with as few
as 4.5k training examples for Belarusian-English.
In order to better understand the effect of training
set size in these settings, we evaluate on four additional languages that have more than 167k training examples each – Arabic (Ar), German (De),
Hebrew (He) and Italian (It).
2.2 Model Details
Using the same data, we trained three massively


key: chroma:document
string_value:
Hebrew (He) and Italian (It).
2.2 Model Details
Using the same data, we trained three massively
multilingual models: a many-to-many model
which we train using all 116 translation directions
with 58 languages to-and-from English, a one-tomany model from English into 58 languages, and
a many-to-one model from 58 languages into English. We follow the method of Ha et al. (2016);
Johnson et al. (2017) and add a target-language
1github.com/neulab/
word-embeddings-for-nmt
2Chinese, Czech, English, Estonian, Finnish, French,
German, Hindi, Hungarian, Latvian, Romanian, Russian,
Spanish, Turkish. According to http://www.statmt.
org/wmtXX
preﬁx token to each source sentence to enable
many-to-many translation. These different setups
enable us to examine the effect of the number of
translation tasks on the translation quality as measured in BLEU (Papineni et al., 2002). We also
compare our massively multilingual models to
bilingual baselines and to two recently published
results on this dataset (Neubig and Hu (2018);
Wang et al. (2019)).


key: chroma:document
string_value:
bilingual baselines and to two recently published
results on this dataset (Neubig and Hu (2018);
Wang et al. (2019)).
Regarding the models, we focused on the Transformer in the “Base” conﬁguration. We refer the
reader to Vaswani et al. (2017) for more details
on the model architecture. Speciﬁcally, we use 6
layers in both the encoder and the decoder, with
model dimension set at 512, hidden dimension
size of 2048 and 8 attention heads. We also applied dropout at a rate of 0.2 in the following components: on the sum of the input embeddings and
the positional embeddings, on the output of each
sub-layer before added to the previous layer input
(residual connection), on the inner layer output after the ReLU activation in each feed-forward sublayer, and to the attention weight in each attention
sub-layer. This results in a model with approximately 93M trainable parameters. For all models we used the inverse square root learning rate
schedule from Vaswani et al. (2017) with learningrate set at 3 and 40k warmup steps. All models are
implemented in Tensorﬂow-Lingvo (Shen et al.,
2019).


key: chroma:document
string_value:
implemented in Tensorﬂow-Lingvo (Shen et al.,
2019).
In all cases we report test results for the checkpoint that performed best on the development set
in terms of BLEU. For the multilingual models we
create a development set that includes examples
we uniformly sample from a concatenation of all
the individual language pair development sets, resulting in 13k development examples per model.
Another important detail regarding multilingual
training is the batching scheme. In all of our multilingual models we use heterogeneous batching,
where each batch contains examples which are
uniformly sampled from a concatenation of all the
language pairs the model is trained on. Speciﬁcally, we use batches of 64 examples for sequences
shorter than 69 tokens and batches of 16 examples for longer sequences. We did not use oversampling as the dataset is relatively small.
2.3 Results
We use tokenized BLEU in order to be comparable with Neubig and Hu (2018). Table 1 showsAz-En Be-En Gl-En Sk-En Avg.
# of examples 5.9k 4.5k 10k 61k 20.3k
Neubig & Hu 18
baselines 2.7 2.8 16.2 24 11.42


key: chroma:document
string_value:
Neubig & Hu 18
baselines 2.7 2.8 16.2 24 11.42
many-to-one 11.7 18.3 29.1 28.3 21.85
Wang et al. 18 11.82 18.71 30.3 28.77 22.4
Ours
many-to-one 11.24 18.28 28.63 26.78 21.23
many-to-many 12.78 21.73 30.65 29.54 23.67
Table 1: X !En test BLEU on the TED Talks corpus,
for the language pairs from Neubig and Hu (2018)
Ar-En De-En He-En It-En Avg.
# of examples 213k 167k 211k 203k 198.5k
baselines 27.84 30.5 34.37 33.64 31.59
many-to-one 25.93 28.87 30.19 32.42 29.35
many-to-many 28.32 32.97 33.18 35.14 32.4
Table 2: X !En test BLEU on the TED Talks corpus,
for language pairs with more than 167k examples
updates ar_many2one ro_many2one nl_many2one de_many2one it_many2one ar de it nl
0 0


key: chroma:document
string_value:
updates ar_many2one ro_many2one nl_many2one de_many2one it_many2one ar de it nl
0 0
49148 0.2839321792
51751 0.2832087576
53492 0.2823074162
56111 0.2822623849
57843 0.2835516334
59595 0.2890901268
61344 0.2845613658
63938 0.2924402654
65676 0.2828268111
67401 0.2909407914
70019 0.2906394601
71770 0.283375144
73479 0.2862167656
75201 0.2850508094
77780 0.2772497535
79488 0.286601305
81201 0.2964281738
82895 0.2811564505
84594 0.2892799973
86301 0.2811136246
88849 0.2873187661
90551 0.287561506
93101 0.2806210518
95667 0.2852652967
97386 0.286067158
99069 0.285282582
101647 0.2873030901


key: chroma:document
string_value:
95667 0.2852652967
97386 0.286067158
99069 0.285282582
101647 0.2873030901
103358 0.2812287509
105932 0.2841886282
107656 0.2752591372
109350 0.2852451503
111069 0.2802457809
113631 0.2839408815
115325 0.2698247731
117038 0.2862476707
119599 0.2838998437
121302 0.2805997431
123015 0.2826767564
Figure 1: Development BLEU on
fIt,Ro,Nl,De,Ar g!En vs. training BLEU for the
many-to-one and many-to-many models. Best viewed
in color.
the results of our experiments when evaluating
on the same language pairs as they did. The results under “Neubig & Hu 18” are their bilingual baselines and their best many-to-one models.
Their many-to-one models use similar-languageregularization, i.e. ﬁne-tuning a pre-trained manyto-one model with data from the language pair of
interest together with data from a language pair


key: chroma:document
string_value:
interest together with data from a language pair
that has a typologically-similar source language
and more training data (i.e. Russian and Belarusian, Turkish and Azerbaijani). The results under
“Ours” are our many-to-one and many-to-many
models we trained identically in terms of model
architecture and hyper-parameters.
We ﬁrst note that our many-to-many model outperforms all other models when translating into
English, with 1.82 BLEU improvement (when averaged across the four language pairs) over the
best ﬁne-tuned many-to-one models of Neubig
and Hu (2018) and 2.44 BLEU improvement over
our many-to-one model when averaged across the
four low-resource language pairs (Table 1). This
is surprising as it uses the same X !En data,
model architecture and capacity as our many-toone model, while handling a heavier burden since
it also supports 58 additional translation tasks
(from English into58 languages). Our models also
outperform the more complex models of Wang
et al. (2019) which use ”Soft Decoupled Encoding” for the input tokens, while our models use a
simple subword segmentation.


key: chroma:document
string_value:
et al. (2019) which use ”Soft Decoupled Encoding” for the input tokens, while our models use a
simple subword segmentation.
One possible explanation is that the many-toone model overﬁts the English side of the corpus
as it is multi-way-parallel: in such setting the English sentences are overlapping across the different language pairs, making it much easier for the
model to memorize the training set instead of generalizing (when enough capacity is available). On
the other hand, the many-to-many model is trained
on additional target languages other than English,
which can act as regularizers for the X !En tasks,
reducing such overﬁtting.
To further illustrate this, Figure 1 tracks the
BLEU scores on the individual development sets
during training for Italian (It), Romanian (Ro),
Dutch (Nl), German (De) and Arabic (Ar) into English (left), together with BLEU scores on a subset of the training set for each model. We can
see that while the many-to-one model degrades in
performance on the development set, the manyto-many model still improves. Note the large
gap in the many-to-one model between the training set BLEU and the development set BLEU,
which points on the generalization issue that is


key: chroma:document
string_value:
gap in the many-to-one model between the training set BLEU and the development set BLEU,
which points on the generalization issue that is
not present in the many-to-many setting. We also
note that our many-to-one model is on average
0.75 BLEU behind the best many-to-one models
in Neubig and Hu (2018). We attribute this to the
fact that their models are ﬁne-tuned using similarlanguage-regularization while our model is not.
We ﬁnd an additional difference between the
results on the resource-scarce languages (Table 1) and the higher-resource languages (Table
2). Speciﬁcally, the bilingual baselines outperform the many-to-one models only in the higherresource setting. This makes sense as in the low-En-Az En-Be En-Gl En-Sk Avg.
# of examples 5.9k 4.5k 10k 61k 20.3k
baselines 2.16 2.47 3.26 5.8 3.42
one-to-many 5.06 10.72 26.59 24.52 16.72
many-to-many 3.9 7.24 23.78 21.83 14.19
En-Ar En-De En-He En-It Avg.


key: chroma:document
string_value:
many-to-many 3.9 7.24 23.78 21.83 14.19
En-Ar En-De En-He En-It Avg.
# of examples 213k 167k 211k 203k 198.5k
baselines 12.95 23.31 23.66 30.33 22.56
one-to-many 16.67 30.54 27.62 35.89 27.68
many-to-many 14.25 27.95 24.16 33.26 24.9
Table 3: En !X test BLEU on the TED Talks corpus
resource setting the baselines have very few training examples to outperform the many-to-one models, while in the higher resource setting they have
access to more training data. This corroborates the
results of Gu et al. (2018) that showed the sensitivity of such models to similar low resource conditions and the improvements gained from using
many-to-one models (however with much fewer
language pairs).
Table 3 shows the results of our massively
multilingual models and bilingual baselines when
evaluated out-of-English. In this case we see an
opposite trend: the many-to-many model performs
worse than the one-to-many model by 2.53 BLEU
on average. While previous works (Wang et al.,


key: chroma:document
string_value:
worse than the one-to-many model by 2.53 BLEU
on average. While previous works (Wang et al.,
2018; Sachan and Neubig, 2018) discuss the phenomena of quality degradation in English-to-many
settings, this shows that increasing the number of
source languages also causes additional degradation in a many-to-many model. This degradation
may be due to the English-centric setting: since
most of the translation directions the model is
trained on are into English, this leaves less capacity for the other target languages (while still performing better than the bilingual baselines on all 8
language pairs). We also note that in this case the
results are consistent among the higher and lower
resource pairs – the one-to-many model is better
than the many-to-many model, which outperforms
the bilingual baselines in all cases. This is unlike
the difference we saw in the X !En experiments
since here we do not have the multi-way-parallel
overﬁtting issue.
2.4 Discussion
From the above experiments we learn that NMT
models can scale to 59 languages in a lowresource, imbalanced, English-centric setting,
with the following observations: (1) massively
multilingual many-to-many models outperform


key: chroma:document
string_value:
models can scale to 59 languages in a lowresource, imbalanced, English-centric setting,
with the following observations: (1) massively
multilingual many-to-many models outperform
many-to-one and bilingual models with similar capacity and identical training conditions when averaged over 8 language pairs into English. We
attribute this improvement over the many-to-one
models to the multiple target language pairs which
may act as regularizers, especially in this lowresource multi-way-parallel setting that is prone to
memorization. (2) many-to-many models are inferior in performance when going out-of-English
in comparison to a one-to-many model. We attribute this to English being over-represented in
the English-centric many-to-many setting, where
it appears as a target language in 58 out of 116
trained directions, which may harm the performance on the rest of the target languages as the
model capacity is limited.3
It is important to stress the fact that we compared the different models under identical training
conditions and did not perform extensive hyperparameter tuning for each setting separately. However, we believe that such tuning may improve
performance even further, as the diversity in each
training batch is very different between the different settings. For example, while the baseline
model batches include only one language in the


key: chroma:document
string_value:
performance even further, as the diversity in each
training batch is very different between the different settings. For example, while the baseline
model batches include only one language in the
source and one language in the target, the manyto-many model includes 59 languages in each
side with a strong bias towards English. These
differences may require tailored hyper-parameter
choices for each settings (i.e. different batch sizes,
learning rate schedules, dropout rates etc.) which
would be interesting to explore in future work.
In the following experiments we investigate
whether these observations hold using (1) an even
larger set of languages, and (2) a much larger,
balanced training corpus that is not multi-wayparallel.
3 High-Resource Setting: 103 Languages
3.1 Experimental Setup
In this setting we scale the number of languages
and examples per language pair further when
training a single massively multilingual model.
Since we are not aware of a publicly available resource for this purpose, we construct an in-house
dataset. This dataset includes 102 language pairs
which we “mirror” to-and-from English, with up
to one million examples per language pair. This
results in 103 languages in total, and 204 translation directions which we train simultaneously.


key: chroma:document
string_value:
to one million examples per language pair. This
results in 103 languages in total, and 204 translation directions which we train simultaneously.
3This issue may be alleviated by over-sampling the nonEnglish-target pairs, but we leave this for future work.More details about this dataset are available in Table 4, and Table 10 in the supplementary material
details all the languages in the dataset.4
Similarly to our previous experiments, we compare the massively multilingual models to bilingual baselines trained on the same data. We tokenize the data using an in-house tokenizer and then
apply joint subword segmentation to achieve an
open-vocabulary. In this setting we used a vocabulary of 64k subwords rather than 32k. Since the
dataset contains 24k unique characters, a 32k symbol vocabulary will consist of mostly characters,
thereby increasing the average sequence length.
Regarding the model, for these experiments we
use a larger Transformer model with 6 layers in
both the encoder and the decoder, model dimension set to 1024, hidden dimension size of 8192,
and 16 attention heads. This results in a model
with approximately 473.7M parameters.5Since
the model and data are much larger in this case,
we used a dropout rate of 0.1 for our multilingual
models and tuned it to 0.3 for our baseline models


key: chroma:document
string_value:
the model and data are much larger in this case,
we used a dropout rate of 0.1 for our multilingual
models and tuned it to 0.3 for our baseline models
as it improved the translation quality on the development set.
We evaluate our models on 10 languages from
different typological families: Semitic – Arabic
(Ar), Hebrew (He), Romance – Galician (Gl),
Italian (It), Romanian (Ro), Germanic – German
(De), Dutch (Nl), Slavic – Belarusian (Be), Slovak (Sk) and Turkic – Azerbaijani (Az) and Turkish (Tr). We evaluate both to-and-from English,
where each language pair is trained on up to one
million examples. As in the previous experiment,
we report test results from the model that performed best in terms of BLEU on the development
set.
4The average number of examples per language pair is
940k, as for 13 out of the 102 pairs we had less than one
million examples available.
5This is larger than the Transformer “Big” conﬁguration,
which includes approximately 213M trained parameters.
# of language pairs 102
examples per pair
min 63,879
max 1,000,000
average 940,087
std. deviation 188,194


key: chroma:document
string_value:
# of language pairs 102
examples per pair
min 63,879
max 1,000,000
average 940,087
std. deviation 188,194
total # of examples 95,888,938
Table 4: Training set details for the 103 langauges corpus, X!En data.
Ar Az Be De He It Nl Ro Sk Tr Avg.
baselines 23.34 16.3 21.93 30.18 31.83 36.47 36.12 34.59 25.39 27.13 28.33
many-to-one 26.04 23.68 25.36 35.05 33.61 35.69 36.28 36.33 28.35 29.75 31.01
many-to-many 22.17 21.45 23.03 37.06 30.71 35.0 36.18 36.57 29.87 27.64 29.97
Table 5: X !En test BLEU on the 103-language corpus
Ar Az Be De He It Nl Ro Sk Tr Avg.
baselines 10.57 8.07 15.3 23.24 19.47 31.42 28.68 27.92 11.08 15.54 19.13
one-to-many 12.08 9.92 15.6 31.39 20.01 33 31.06 28.43 17.67 17.68 21.68


key: chroma:document
string_value:
one-to-many 12.08 9.92 15.6 31.39 20.01 33 31.06 28.43 17.67 17.68 21.68
many-to-many 10.57 9.84 14.3 28.48 17.91 30.39 29.67 26.23 18.15 15.58 20.11
Table 6: En !X test BLEU on the 103-language corpus
3.2 Results
Table 5 describes the results when translating into
English. First, we can see that both multilingual
models perform better than the baselines in terms
of average BLEU. This shows that massively multilingual many-to-many models can work well in
realistic settings with millions of training examples, 102 languages and 204 jointly trained directions to-and-from English. Looking more closely,
we note several different behaviors in comparison
to the low-resource experiments on the TED Talks
corpus. First, the many-to-one model here performs better than the many-to-many model. This
shows that the previous result was indeed due to
the pathologies of the low-resource dataset; when
the training data is large enough and not multiway-parallel there is no overﬁtting in the many-toone model, and it outperforms the many-to-many
model in most cases while they are trained identically.


key: chroma:document
string_value:
model in most cases while they are trained identically.
One particular outlier in this case is German-toEnglish, where the many-to-one model is 2 BLEU
points below the many-to-many model. We examine the BLEU score of this language pair on its
dedicated German-English development set during training in the many-to-one model and ﬁnd
that it highly ﬂuctuates. We then measure the
performance on the test set for this language pair
by choosing the best checkpoint on the dedicated
German-English development set (instead of on
the mixed multilingual development set) and ﬁnd
it to be 38.07, which is actually higher in 1 BLEU
than the best result of the many-to-many model.
This shows that while training many languages together, there is no “silver bullet”: some languages
may suffer from severe interference during training (i.e. a reduction of 3 BLEU in this case, from38.07 to 35.05) while other languages continue to
improve with more updates.
Table 6 describes the results when translating
out-of-English. Again, both of the massively multilingual models perform better than the baselines when averaged across the 10 evaluated language pairs, while handling up to 102 languages


key: chroma:document
string_value:
out-of-English. Again, both of the massively multilingual models perform better than the baselines when averaged across the 10 evaluated language pairs, while handling up to 102 languages
to-and-from English and 204 translation tasks simultaneously. In this case the results are similar to those we observed on the TED talks corpus, where the one-to-many model performs better
than the many-to-many model. Again, this advantage may be due to the one-to-many model handling a smaller number of tasks while not being
biased towards English in the target side like the
many-to-many model.
4 Analysis
The above results show that massively multilingual NMT is indeed possible in large scale settings
and can improve performance over strong bilingual baselines. However, it was shown in a somewhat extreme case with more than 100 languages
trained jointly, where we saw that in some cases
the joint training may harm the performance for
some language pairs (i.e. German-English above).
In the following analysis we would like to better understand the trade-off between the number
of languages involved and the translation accuracy while keeping the model capacity and training conﬁguration ﬁxed.
4.1 Multilinguality & Supervised
Performance
We ﬁrst study the effect of varying the number of languages on the translation accuracy in


key: chroma:document
string_value:
4.1 Multilinguality & Supervised
Performance
We ﬁrst study the effect of varying the number of languages on the translation accuracy in
a supervised setting, where we focus on manyAr-En En-Ar Fr-En En-Fr Ru-En En-Ru Uk-En En-Uk Avg.
5-to-5 23.87 12.42 38.99 37.3 29.07 24.86 26.17 16.48 26.14
25-to-25 23.43 11.77 38.87 36.79 29.36 23.24 25.81 17.17 25.8
50-to-50 23.7 11.65 37.81 35.83 29.22 21.95 26.02 15.32 25.18
75-to-75 22.23 10.69 37.97 34.35 28.55 20.7 25.89 14.59 24.37
103-to-103 21.16 10.25 35.91 34.42 27.25 19.9 24.53 13.89 23.41
Table 7: Supervised performance while varying the number of languages involved
Ar-Fr Fr-Ar Ru-Uk Uk-Ru Avg.
5-to-5 1.66 4.49 3.7 3.02 3.21


key: chroma:document
string_value:
Ar-Fr Fr-Ar Ru-Uk Uk-Ru Avg.
5-to-5 1.66 4.49 3.7 3.02 3.21
25-to-25 1.83 5.52 16.67 4.31 7.08
50-to-50 4.34 4.72 15.14 20.23 11.1
75-to-75 1.85 4.26 11.2 15.88 8.3
103-to-103 2.87 3.05 12.3 18.49 9.17
Table 8: Zero-Shot performance while varying the
number of languages involved
to-many models. We create four subsets of the
in-house dataset by sub-sampling it to a different number of languages in each subset. In
this way we create four additional English-centric
datasets, containing 5, 25, 50 and 75 languages
each to-and-from English. We make sure that
each subset contains all the languages from the
next smaller subsets – i.e. the 25 language subset contains the 5 language subset, the 50 language subset contains the 25 language subset and
so on. We train a similar-capacity large Transformer model (with 473.7M parameters) on each
of these subsets and measure the performance for
each model on the 8 supervised language pairs


key: chroma:document
string_value:
of these subsets and measure the performance for
each model on the 8 supervised language pairs
from the smallest subset – fArabic, French, Russian, Ukrainian g$English. In this way we can
analyze to what extent adding more languages improves or harms translation quality while keeping
the model capacity ﬁxed, testing the capacity vs.
accuracy “saturation point”.
Table 7 shows the results of this experiment,
reporting the test results for the models that performed best on the multilingual development set.
We can see that in most cases the best results
are obtained using the 5-to-5 model, showing that
there is indeed a trade off between the number
of languages and translation accuracy when using a ﬁxed model capacity and the same training setup. One may expect that the gaps between
the different models should become smaller and
even close with more updates, as the models with
more languages see less examples per language
in each batch, thus requiring more updates to improve in terms of BLEU. However, in our setting
these gaps did not close even after the models converged, leaving 2.73 average BLEU difference be-tween the 5-to-5 and the 103-to-103 model.
4.2 Multilinguality & Zero-Shot
Performance
We then study the effect of the number of languages on zero-shot translation accuracy. Since


key: chroma:document
string_value:
4.2 Multilinguality & Zero-Shot
Performance
We then study the effect of the number of languages on zero-shot translation accuracy. Since
we ﬁnd zero-shot accuracy as an interesting measure for model generalization, we hypothesize that
by adding more languages, the model is forced to
create a more generalized representation to better utilize its capacity, which may improve zeroshot performance. We choose four language pairs
for this purpose: Arabic $French which are distant languages, and Ukrainian $Russian which are
similar. Table 8 shows the results of our models
on these language pairs. For Arabic $French the
BLEU scores are very low in all cases, with the
50-to-50 and 25-to-25 models being slightly better than rest on Ar-Fr and Fr-Ar respectively. On
Russian $Ukrainian we see clear improvements
when increasing the number of languages to more
than ﬁve.
Figure 2 further illustrates this, showing the better generalization performance of the massively
multilingual models under this zero-shot setting.
While the zero-shot performance in this case is
low and unstable for the 5-to-5 and 25-to-25 modupdate 5-to-5 25-to-25 50-to-50 75-to-75 103-to-103
0 0 0 0 0 0


key: chroma:document
string_value:
0 0 0 0 0 0
25000 3.360397369 7.345648855 7.028211653 6.687645614 6.629930437
50000 2.003555186 10.08476391 11.66040972 11.46485135 11.34905592
100000 2.383616194 9.54657495 15.44517726 14.72926438 13.17522973
150000 2.588021383 7.121089101 14.5408541 15.54533243 15.89359492
200000 2.589718811 10.99432111 15.82096368 17.38970876 16.26121253
250000 2.854427323 14.00393397 16.53215885 15.97282737 15.3215304
300000 2.823847346 6.934611499 16.78672731 16.52613729 16.16107672
350000 2.950227261 4.779103771 16.73545986 16.12752229 17.03165472
400000 4.301280901 6.631205231 16.16190225 17.38892049 17.55904406


key: chroma:document
string_value:
400000 4.301280901 6.631205231 16.16190225 17.38892049 17.55904406
450000 3.882381693 6.804813445 18.20554733 17.48778224 17.9339543
500000 3.45445089 6.358428299 17.32598543 16.71108902 15.97367823
550000 3.18 6.38 18.65 18.28 17.4
600000 2.86 9.5 18.46 14.92 17.12
650000 2.55 12.2 18.98 15.68 16.19
700000 2.98 8.44 20.16 15.4 18.52
Figure 2: Zero-shot BLEU during training for Ukranian to Russian
els, it is much better for the 50-to-50, 75-to-75 and
103-to-103 models. Given these results we can
say that the balance between capacity and generalization here favors the mid range 50-to-50 model,
even when using models with more than 473M
trained parameters. This may hint at the necessity of even larger models for such settings, which
is a challenging avenue for future work. We also
note that our 103 language corpus includes up to


key: chroma:document
string_value:
trained parameters. This may hint at the necessity of even larger models for such settings, which
is a challenging avenue for future work. We also
note that our 103 language corpus includes up to
one million examples per language pair – while in
real-world MT deployments, systems are trained
on much more examples per pair. This again emphasizes the need for better techniques for training
such massively multilingual models as we may already be hitting the capacity barrier in our setting.
5 Related Work
Dong et al. (2015) extended the NMT model of
Bahdanau et al. (2014) to one-to-many translation
(from English into 4 languages) by adding a dedicated decoder per target language, showing improvements over strong single-pair baselines. Firat et al. (2016a,b) proposed many-to-many models (with up to 6 languages) by using separate encoders and decoders per language while sharing
the attention mechanism. They also introduced
the notion of zero-resource translation, where they
use synthetic training data generated through pivoting to train translation directions without available training data. Ha et al. (2016) and Johnson
et al. (2017) proposed to use a shared encoderdecoder-attention model for many-to-many translation (with up to 7 languages in the latter). In
order to determine the target language in such


key: chroma:document
string_value:
order to determine the target language in such
scenarios they proposed adding dedicated targetlanguage symbols to the source. This method enabled zero-shot translation, showing the ability of
the model to generalize to unseen pairs.
Recent works propose different methods for parameter sharing between language pairs in multilingual NMT. Blackwood et al. (2018) propose
sharing all parameters but the attention mechanism
and show improvements over sharing all parameters. Sachan and Neubig (2018) explore sharing various components in self-attentional (Transformer) models. Lu et al. (2018) add a shared “interlingua” layer while using separate encoders and
decoders. Zaremoodi et al. (2018) utilize recurrent
units with multiple blocks together with a trainable
routing network. Platanios et al. (2018) propose
to share the entire network, while using a contex-tual parameter generator that learns to generate the
parameters of the system given the desired source
and target languages. Gu et al. (2018) propose
a “Universal Language Representation” layer together with a Mixture-of-Language-Experts component to improve a many-to-one model from 5
languages into English.
While the mentioned studies provide valuable
contributions to improving multilingual models,
they apply their models on only up to 7 languages


key: chroma:document
string_value:
languages into English.
While the mentioned studies provide valuable
contributions to improving multilingual models,
they apply their models on only up to 7 languages
(Johnson et al., 2017) and 20 trained directions
(Cettolo et al., 2017) in a single model, whereas
we focus on scaling NMT to much larger numbers of languages and trained directions. Regarding massively multilingual models, Neubig and
Hu (2018) explored methods for rapid adaptation
of NMT to new languages by training multilingual models on the 59-language TED Talks corpus and ﬁne-tuning them using data from the new
languages. While modeling signiﬁcantly more
languages than previous studies, they only train
many-to-one models, which we show are inferior
in comparison to our proposed massively multilingual many-to-many models when evaluated into
English on this dataset.
Tiedemann (2018) trained an English-centric
many-to-many model on translations of the bible
including 927 languages. While this work pointed
to an interesting phenomena in the latent space
learned by the model where it clusters representations of typologically-similar languages together, it did not include any evaluation of the
produced translations. Similarly, Malaviya et al.
(2017) trained a many-to-English system including 1017 languages from bible translations, and


key: chroma:document
string_value:
produced translations. Similarly, Malaviya et al.
(2017) trained a many-to-English system including 1017 languages from bible translations, and
used it to infer typological features for the different languages (without evaluating the translation quality). In another relevant work, Artetxe
and Schwenk (2018) trained an NMT model on
93 languages and used the learned representations
to perform cross-lingual transfer learning. Again,
they did not report the performance of the translation model learned in that massively multilingual
setting.
6 Conclusions and Future Work
We showed that NMT models can successfully
scale to 102 languages to-and-from English with
204 trained directions and up to one million examples per direction. Such models improve the
translation quality over similar single-pair baselines when evaluated to and from English by more
than 2 BLEU when averaged over 10 diverse language pairs in each case. We show a similar result on the low-resource TED Talks corpus with 59
languages and 116 trained directions. We analyze
the trade-offs between translation quality and the
number of languages involved, pointing on capacity bottlenecks even with very large models and
showing that massively multilingual models can
generalize better to zero-shot settings.
We hope this work will encourage future research on massively multilingual NMT, enabling


key: chroma:document
string_value:
showing that massively multilingual models can
generalize better to zero-shot settings.
We hope this work will encourage future research on massively multilingual NMT, enabling
easier support for systems that can serve more people around the globe. There are many possible avenues for future work, including semi-supervised
learning in such settings, exploring ways to reduce the performance degradation when increasing the number of languages, or using such models
for multilingual transfer learning (McCann et al.,
2017; Eriguchi et al., 2018; Artetxe and Schwenk,
2018). Understanding and improving zero-shot
performance in such scenarios is also a promising
direction for future work.
Acknowledgments
We would like to thank the Google Brain and
Google Translate teams for their useful inputs and
discussions. We would also like to thank the entire
Lingvo development team for their foundational
contributions to this project.
References
Mikel Artetxe and Holger Schwenk. 2018. Massively multilingual sentence embeddings for zeroshot cross-lingual transfer and beyond. arXiv
preprint arXiv:1812.10464 .
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly
learning to align and translate. arXiv preprint


key: chroma:document
string_value:
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly
learning to align and translate. arXiv preprint
arXiv:1409.0473 .
Graeme Blackwood, Miguel Ballesteros, and Todd
Ward. 2018. Multilingual neural machine translation with task-speciﬁc attention. In Proceedings of
the 27th International Conference on Computational
Linguistics , Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Ondˇrej Bojar, Rajen Chatterjee, Christian Federmann,
et al. 2017. Findings of the 2017 conference on machine translation (wmt17). In Proceedings of the
Second Conference on Machine Translation .Ondˇrej Bojar, Rajen Chatterjee, Christian Federmann,
et al. 2016. Findings of the 2016 conference on machine translation. In ACL 2016 FIRST CONFERENCE ON MACHINE TRANSLATION (WMT16) ,
pages 131–198.
Ondˇrej Bojar, Christian Federmann, Mark Fishel,
Yvette Graham, Barry Haddow, Philipp Koehn, and


key: chroma:document
string_value:
Ondˇrej Bojar, Christian Federmann, Mark Fishel,
Yvette Graham, Barry Haddow, Philipp Koehn, and
Christof Monz. 2018. Findings of the 2018 conference on machine translation (wmt18). In Proceedings of the Third Conference on Machine Translation: Shared Task Papers .
Mauro Cettolo, Federico Marcello, Bentivogli Luisa,
Niehues Jan, St ¨uker Sebastian, Sudoh Katsuitho,
Yoshino Koichiro, and Federmann Christian. 2017.
Overview of the iwslt 2017 evaluation campaign. In
International Workshop on Spoken Language Translation .
Daxiang Dong, Hua Wu, Wei He, Dianhai Yu, and
Haifeng Wang. 2015. Multi-task learning for multiple language translation. In Proceedings of the
53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint
Conference on Natural Language Processing .
Akiko Eriguchi, Melvin Johnson, Orhan Firat, Hideto
Kazawa, and Wolfgang Macherey. 2018. Zeroshot cross-lingual classiﬁcation using multilingual neural machine translation. arXiv preprint
arXiv:1809.04686 .


key: chroma:document
string_value:
arXiv:1809.04686 .
Orhan Firat, Kyunghyun Cho, and Yoshua Bengio.
2016a. Multi-way, multilingual neural machine
translation with a shared attention mechanism. In
Proceedings of the 2016 Conference of the North
American Chapter of the Association for Computational Linguistics: Human Language Technologies .
Orhan Firat, Baskaran Sankaran, Yaser Al-Onaizan,
Fatos T Yarman Vural, and Kyunghyun Cho. 2016b.
Zero-resource translation with multi-lingual neural
machine translation. In Proceedings of the 2016
Conference on Empirical Methods in Natural Language Processing .
Jiatao Gu, Hany Hassan, Jacob Devlin, and Victor O.K.
Li. 2018. Universal neural machine translation for
extremely low resource languages. In Proceedings of the 2018 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language Technologies .
Thanh-Le Ha, Jan Niehues, and Alexander Waibel.
2016. Toward multilingual neural machine translation with universal encoder and decoder. arXiv
preprint arXiv:1611.04798 .
Hany Hassan, Anthony Aue, Chang Chen, et al.


key: chroma:document
string_value:
preprint arXiv:1611.04798 .
Hany Hassan, Anthony Aue, Chang Chen, et al.
2018. Achieving human parity on automatic chinese to english news translation. arXiv preprint
arXiv:1803.05567 .
Melvin Johnson, Mike Schuster, Quoc V Le, et al.
2017. Google’s multilingual neural machine translation system: Enabling zero-shot translation. Transactions of the Association of Computational Linguistics.
Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent continuous translation models. In Proceedings
of the 2013 Conference on Empirical Methods in
Natural Language Processing , Seattle, Washington,
USA.
Surafel Melaku Lakew, Mauro Cettolo, and Marcello
Federico. 2018. A comparison of transformer and
recurrent neural networks on multilingual neural
machine translation. In Proceedings of the 27th International Conference on Computational Linguistics, Santa Fe, New Mexico, USA.
Yichao Lu, Phillip Keung, Faisal Ladhak, Vikas Bhardwaj, Shaonan Zhang, and Jason Sun. 2018. A neural interlingua for multilingual machine translation.
InProceedings of the Third Conference on Machine
Translation: Research Papers , Belgium, Brussels.


key: chroma:document
string_value:
InProceedings of the Third Conference on Machine
Translation: Research Papers , Belgium, Brussels.
Chaitanya Malaviya, Graham Neubig, and Patrick Littell. 2017. Learning language representations for typology prediction. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language
Processing .
Bryan McCann, James Bradbury, Caiming Xiong, and
Richard Socher. 2017. Learned in translation: Contextualized word vectors. In Advances in Neural Information Processing Systems .
Graham Neubig and Junjie Hu. 2018. Rapid adaptation
of neural machine translation to new languages. In
Proceedings of the 2018 Conference on Empirical
Methods in Natural Language Processing .
Toan Q. Nguyen and David Chiang. 2017. Transfer
learning across low-resource, related languages for
neural machine translation. In Proc. IJCNLP .
Myle Ott, Sergey Edunov, David Grangier, and
Michael Auli. 2018. Scaling neural machine translation. In Proceedings of the Third Conference on
Machine Translation: Research Papers .
Kishore Papineni, Salim Roukos, Todd Ward, and WeiJing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of
40th Annual Meeting of the Association for Computational Linguistics , Philadelphia, Pennsylvania,
USA.


key: chroma:document
string_value:
40th Annual Meeting of the Association for Computational Linguistics , Philadelphia, Pennsylvania,
USA.
Emmanouil Antonios Platanios, Mrinmaya Sachan,
Graham Neubig, and Tom Mitchell. 2018. Contextual parameter generation for universal neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language
Processing .Ye Qi, Sachan Devendra, Felix Matthieu, Padmanabhan Sarguna, and Neubig Graham. 2018. When
and why are pre-trained word embeddings useful for
neural machine translation. In HLT-NAACL .
Devendra Sachan and Graham Neubig. 2018. Parameter sharing methods for multilingual self-attentional
translation models. In Proceedings of the Third
Conference on Machine Translation: Research Papers, Belgium, Brussels.
Rico Sennrich, Barry Haddow, and Alexandra Birch.
2016. Neural machine translation of rare words with
subword units. In Proceedings of the 54th Annual
Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , volume 1, pages
1715–1725.
Jonathan Shen, Patrick Nguyen, Yonghui Wu, Zhifeng
Chen, et al. 2019. Lingvo: a modular and scalable
framework for sequence-to-sequence modeling.


key: chroma:document
string_value:
Jonathan Shen, Patrick Nguyen, Yonghui Wu, Zhifeng
Chen, et al. 2019. Lingvo: a modular and scalable
framework for sequence-to-sequence modeling.
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014.
Sequence to sequence learning with neural networks. In Advances in neural information processing systems , pages 3104–3112.
J¨org Tiedemann. 2018. Emerging language spaces
learned from massively multilingual corpora. arXiv
preprint arXiv:1802.00273 .
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob
Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz
Kaiser, and Illia Polosukhin. 2017. Attention is all
you need. In Advances in Neural Information Processing Systems , pages 5998–6008.
Xinyi Wang, Hieu Pham, Philip Arthur, and Graham
Neubig. 2019. Multilingual neural machine translation with soft decoupled encoding. In International
Conference on Learning Representations .
Yining Wang, Jiajun Zhang, Feifei Zhai, Jingfang Xu,


key: chroma:document
string_value:
Conference on Learning Representations .
Yining Wang, Jiajun Zhang, Feifei Zhai, Jingfang Xu,
and Chengqing Zong. 2018. Three strategies to improve one-to-many multilingual translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , Brussels, Belgium.
Yonghui Wu, Mike Schuster, Zhifeng Chen, et al. 2016.
Google’s neural machine translation system: Bridging the gap between human and machine translation.
arXiv preprint arXiv:1609.08144 .
Poorya Zaremoodi, Wray Buntine, and Gholamreza
Haffari. 2018. Adaptive knowledge sharing in
multi-task learning: Improving low-resource neural
machine translation. In Proceedings of the 56th Annual Meeting of the Association for Computational
Linguistics , Melbourne, Australia.
Barret Zoph, Deniz Yuret, Jonathan May, and Kevin
Knight. 2016. Transfer learning for low-resource
neural machine translation. In Proceedings of the
2016 Conference on Empirical Methods in Natural
Language Processing .
A Supplementary Material
Language Train set size
Arabic 214111
Hebrew 211819
Russian 208458
Korean 205640
Italian 204503
Japanese 204090
Chinese-Taiwan 202646


key: chroma:document
string_value:
Arabic 214111
Hebrew 211819
Russian 208458
Korean 205640
Italian 204503
Japanese 204090
Chinese-Taiwan 202646
Chinese-China 199855
Spanish 196026
French 192304
Portuguese-Brazil 184755
Dutch 183767
Turkish 182470
Romanian 180484
Polish 176169
Bulgarian 174444
Vietnamese 171995
German 167888
Persian 150965
Hungarian 147219
Serbian 136898
Greek 134327
Croatian 122091
Ukrainian 108495
Czech 103093
Thai 98064
Indonesian 87406
Slovak 61470
Swedish 56647
Portuguese 51785
Danish 44940
Albanian 44525
Lithuanian 41919
Macedonian 25335
Finnish 24222
Burmese 21497
Armenian 21360
French-Canadian 19870
Slovenian 19831
Hindi 18798
Norwegian 15825
Georgian 13193
Estonian 10738
Kurdish 10371
Galician 10017
Marathi 9840
Mongolian 7607
Esperanto 6535
Tamil 6224
Urdu 5977
Azerbaijani 5946


key: chroma:document
string_value:
Marathi 9840
Mongolian 7607
Esperanto 6535
Tamil 6224
Urdu 5977
Azerbaijani 5946
Bosnian 5664
Chinese 5534
Malay 5220
Basque 5182
Bengali 4649
Belarusian 4509
Kazakh 3317
Table 9: Language pairs in the TED talks dataset (58
languages, paired with English) with the train-set size
for each pair.Languages
Afrikaans Laothian
Albanian Latin
Amharic Latvian
Arabic Lithuanian
Armenian Luxembourgish*
Azerbaijani Macedonian
Basque Malagasy
Belarusian Malay
Bengali Malayalam
Bosnian Maltese
Bulgarian Maori
Burmese Marathi
Catalan Mongolian
Cebuano Nepali
Chichewa* Norwegian
Chinese Pashto
Corsican* Persian
Croatian Polish
Czech Portuguese
Danish Punjabi
Dutch Romanian
Esperanto Russian
Estonian Samoan*
Finnish Scots Gaelic*
French Serbian
Frisian Sesotho
Galician Shona*
Georgian Sindhi*
German Sinhalese
Greek Slovak
Gujarati Slovenian
Haitian Creole Somali


key: chroma:document
string_value:
Galician Shona*
Georgian Sindhi*
German Sinhalese
Greek Slovak
Gujarati Slovenian
Haitian Creole Somali
Hausa* Spanish
Hawaiian* Sundanese
Hebrew Swahili
Hindi Swedish
Hmong* Tagalog
Hungarian Tajik*
Icelandic Tamil
Igbo Telugu
Indonesian Thai
Irish Turkish
Italian Ukrainian
Japanese Urdu
Javanese Uzbek
Kannada Vietnamese
Kazakh Welsh
Khmer Xhosa
Korean Yiddish
Kurdish Yoruba*
Kyrgyz Zulu
Table 10: Language pairs in the in-house dataset (102
languages, paired with English). For languages marked
with * we had less than 1M examples, while for the rest
we used exactly 1M.


key: chroma:document
string_value:
Attention Augmented Convolutional Networks
Irwan Bello Barret Zoph Ashish Vaswani Jonathon Shlens
Quoc V . Le
Google Brain
fibello,barretzoph,avaswani,shlens,qvl g@google.com
Abstract
Convolutional networks have been the paradigm of
choice in many computer vision applications. The convolution operation however has a signiﬁcant weakness in that it
only operates on a local neighborhood, thus missing global
information. Self-attention, on the other hand, has emerged
as a recent advance to capture long range interactions, but
has mostly been applied to sequence modeling and generative modeling tasks. In this paper, we consider the use of
self-attention for discriminative visual tasks as an alternative to convolutions. We introduce a novel two-dimensional
relative self-attention mechanism that proves competitive
in replacing convolutions as a stand-alone computational
primitive for image classiﬁcation. We ﬁnd in control experiments that the best results are obtained when combining
both convolutions and self-attention. We therefore propose
to augment convolutional operators with this self-attention
mechanism by concatenating convolutional feature maps
with a set of feature maps produced via self-attention. Extensive experiments show that Attention Augmentation leads


key: chroma:document
string_value:
mechanism by concatenating convolutional feature maps
with a set of feature maps produced via self-attention. Extensive experiments show that Attention Augmentation leads
to consistent improvements in image classiﬁcation on ImageNet and object detection on COCO across many different models and scales, including ResNets and a stateof-the art mobile constrained network, while keeping the
number of parameters similar. In particular, our method
achieves a 1:3%top-1 accuracy improvement on ImageNet
classiﬁcation over a ResNet50 baseline and outperforms
other attention mechanisms for images such as Squeezeand-Excitation [17]. It also achieves an improvement of
1.4 mAP in COCO Object Detection on top of a RetinaNet
baseline.
1. Introduction
Convolutional Neural Networks have enjoyed tremendous success in many computer vision applications, especially in image classiﬁcation [24, 23]. The design of the
convolutional layer imposes 1) locality via a limited receptive ﬁeld and 2) translation equivariance via weight sharing.
20 30 40 50 60 70
number of parameters (millions)72737475767778798081top-1 accuracy
73.6%76.4%77.9%78.4%


key: chroma:document
string_value:
number of parameters (millions)72737475767778798081top-1 accuracy
73.6%76.4%77.9%78.4%
74.3%77.5%78.4%78.9%
74.7%77.7%78.7%79.1%
ResNet
SE-ResNet
AA-ResNet (ours)Figure 1. Attention Augmentation systematically improves image classiﬁcation across a large variety of networks of different
scales. ImageNet classiﬁcation accuracy [9] versus the number of
parameters for baseline models (ResNet) [14], models augmented
with channel-wise attention (SE-ResNet) [17] and our proposed
architecture (AA-ResNet).
Both these properties prove to be crucial inductive biases
when designing models that operate over images. However,
the local nature of the convolutional kernel prevents it from
capturing global contexts in an image, often necessary for
better recognition of objects in images [33].
Self-attention [43], on the other hand, has emerged as a
recent advance to capture long range interactions, but has
mostly been applied to sequence modeling and generative
modeling tasks. The key idea behind self-attention is to


key: chroma:document
string_value:
recent advance to capture long range interactions, but has
mostly been applied to sequence modeling and generative
modeling tasks. The key idea behind self-attention is to
produce a weighted average of values computed from hidden units. Unlike the pooling or the convolutional operator,
the weights used in the weighted average operation are produced dynamically via a similarity function between hidden units. As a result, the interaction between input signals
depends on the signals themselves rather than being predetermined by their relative location like in convolutions. In
particular, this allows self-attention to capture long rangearXiv:1904.09925v5  [cs.CV]  9 Sep 2020
Input Attention maps Weighted average of the values
Standard convolutionOutput
W
H
Nh=2
Values²Head Head¹
Figure 2. Attention-augmented convolution : For each spatial location (h, w), Nhattention maps over the image are computed from
queries and keys. These attention maps are used to compute Nhweighted averages of the values V . The results are then concatenated,
reshaped to match the original volume’s spatial dimensions and mixed with a pointwise convolution. Multi-head attention is applied in
parallel to a standard convolution operation and the outputs are concatenated.
interactions without increasing the number of parameters.


key: chroma:document
string_value:
parallel to a standard convolution operation and the outputs are concatenated.
interactions without increasing the number of parameters.
In this paper, we consider the use of self-attention for
discriminative visual tasks as an alternative to convolutions. We develop a novel two-dimensional relative selfattention mechanism [37] that maintains translation equivariance while being infused with relative position information, making it well suited for images. Our self-attention
formulation proves competitive for replacing convolutions
entirely, however we ﬁnd in control experiments that the
best results are obtained when combining both. We therefore do not completely abandon the idea of convolutions,
but instead propose to augment convolutions with this selfattention mechanism. This is achieved by concatenating
convolutional feature maps, which enforce locality, to selfattentional feature maps capable of modeling longer range
dependencies (see Figure 2).
We test our method on the CIFAR-100 and ImageNet
classiﬁcation [22, 9] and the COCO object detection [27]
tasks, across a wide range of architectures at different computational budgets, including a state-of-the art resource
constrained architecture [42]. Attention Augmentation
yields systematic improvements with minimal additional


key: chroma:document
string_value:
constrained architecture [42]. Attention Augmentation
yields systematic improvements with minimal additional
computational burden and notably outperforms the popular Squeeze-and-Excitation [17] channelwise attention approach in all experiments. In particular, Attention Augmentation achieves a 1.3% top-1 accuracy ImageNet on top of
a ResNet50 baseline and 1.4 mAP increase in COCO object detection on top of a RetinaNet baseline. Suprisingly,
experiments also reveal that fully self-attentional models,
a special case of Attention Augmentation, only perform
slightly worse than their fully convolutional counterparts on
ImageNet, indicating that self-attention is a powerful standalone computational primitive for image classiﬁcation.2. Related Work
2.1. Convolutional networks
Modern computer vision has been built on powerful image featurizers learned on image classiﬁcation tasks such
as CIFAR-10 [22] and ImageNet [9]. These datasets have
been used as benchmarks for delineating better image featurizations and network architectures across a broad range
of tasks [21]. For example, improving the “backbone” network typically leads to improvements in object detection
[19] and image segmentation [6]. These observations have
inspired the research and design of new architectures, which
are typically derived from the composition of convolution


key: chroma:document
string_value:
[19] and image segmentation [6]. These observations have
inspired the research and design of new architectures, which
are typically derived from the composition of convolution
operations across an array of spatial scales and skip connections [23, 41, 39, 40, 14, 47, 13]. Indeed, automated
search strategies for designing architectures based on convolutional primitives result in state-of-the-art accuracy on
large-scale image classiﬁcation tasks that translate across a
range of tasks [55, 21].
2.2. Attention mechanisms in networks
Attention has enjoyed widespread adoption as a computational module for modeling sequences because of its
ability to capture long distance interactions [2, 44, 4, 3].
Most notably, Bahdanau et al. [2] ﬁrst proposed to combine attention with a Recurrent Neural Network [15] for
alignment in Machine Translation. Attention was further
extended by Vaswani et al. [43], where the self-attentional
Transformer architecture achieved state-of-the-art results in
Machine Translation. Using self-attention in cooperation
with convolutions is a theme shared by recent work in Natural Language Processing [49] and Reinforcement Learning [52]. For example, the QANet [50] and Evolved Transformer [38] architectures alternate between self-attention


key: chroma:document
string_value:
layers and convolution layers for Question Answering applications and Machine Translation respectively. Additionally, multiple attention mechanisms have been proposed for visual tasks to address the weaknesses of convolutions [17, 16, 7, 46, 45, 53]. For instance, Squeezeand-Excitation [17] and Gather-Excite [16] reweigh feature
channels using signals aggregated from entire feature maps,
while BAM [31] and CBAM [46] reﬁne convolutional features independently in the channel and spatial dimensions.
In non-local neural networks [45], improvements are shown
in video classiﬁcation and object detection via the additive use of a few non-local residual blocks that employ
self-attention in convolutional architectures. However, nonlocal blocks are only added to the architecture after ImageNet pretraining and are initialized in such a way that they
do not break pretraining.
In contrast, our attention augmented networks do not rely
on pretraining of their fully convolutional counterparts and
employ self-attention along the entire architecture. The use
of multi-head attention allows the model to attend jointly


key: chroma:document
string_value:
on pretraining of their fully convolutional counterparts and
employ self-attention along the entire architecture. The use
of multi-head attention allows the model to attend jointly
to both spatial and feature subspaces. Additionally, we enhance the representational power of self-attention over images by extending relative self-attention [37, 18] to two dimensional inputs allowing us to model translation equivariance in a principled way. Finally our method produces additional feature maps, rather than recalibrating convolutional
features via addition [45, 53] or gating [17, 16, 31, 46]. This
property allows us to ﬂexibly adjust the fraction of attentional channels and consider a spectrum of architectures,
ranging from fully convolutional to fully attentional models.
3. Methods
We now formally describe our proposed Attention Augmentation method. We use the following naming conventions:H,WandFinrefer to the height, width and number
of input ﬁlters of an activation map. Nh,dvanddkrespectively refer the number of heads, the depth of values and the
depth of queries and keys in multihead-attention (MHA).
We further assume that Nhdividesdvanddkevenly and
denotedh
vanddh
kthe depth of values and queries/keys per
attention head.


key: chroma:document
string_value:
We further assume that Nhdividesdvanddkevenly and
denotedh
vanddh
kthe depth of values and queries/keys per
attention head.
3.1. Self-attention over images
Given an input tensor of shape (H;W;F in),1we ﬂatten
it to a matrix X2RHWFinand perform multihead attention as proposed in the Transformer architecture [43]. The
output of the self-attention mechanism for a single head h
1We omit the batch dimension for simplicity.can be formulated as:
Oh=Softmax0
@(XW q)(XW k)T
q
dh
k1
A(XW v)(1)
whereWq,Wk2RFindh
kandWv2RFindh
vare learned
linear transformations that map the input Xto queriesQ=
XW q, keysK=XW kand valuesV=XW v. The outputs
of all heads are then concatenated and projected again as
follows:
MHA (X) =Concath
O1;:::;O Nhi
WO(2)
whereWO2Rdvdvis a learned linear transformation.


key: chroma:document
string_value:
O1;:::;O Nhi
WO(2)
whereWO2Rdvdvis a learned linear transformation.
MHA (X)is then reshaped into a tensor of shape (H;W;d v)
to match the original spatial dimensions. We note that
multi-head attention incurs a complexity of O((HW)2dk)
and a memory cost of O((HW)2Nh)as it requires to store
attention maps for each head.
3.1.1 Two-dimensional Positional Embeddings
Without explicit information about positions, self-attention
ispermutation equivariant :
MHA ((X)) =(MHA (X))
for any permutation of the pixel locations, making it ineffective for modeling highly structured data such as images. Multiple positional encodings that augment activation
maps with explicit spatial information have been proposed
to alleviate related issues. In particular, the Image Transformer [32] extends the sinusoidal waves ﬁrst introduced in
the original Transformer [43] to 2 dimensional inputs and
CoordConv [29] concatenates positional channels to an activation map.


key: chroma:document
string_value:
the original Transformer [43] to 2 dimensional inputs and
CoordConv [29] concatenates positional channels to an activation map.
However these encodings did not help in our experiments on image classiﬁcation and object detection (see Section 4.5). We hypothesize that this is because such positional encodings, while not permutation equivariant, do not
satisfy translation equivariance , which is a desirable property when dealing with images. As a solution, we propose
to extend the use of relative position encodings [37] to two
dimensions and present a memory efﬁcient implementation
based on the Music Transformer [18].
Relative positional embeddings: Introduced in [37] for
the purpose of language modeling, relative self-attention
augments self-attention with relative position embeddings
and enables translation equivariance while preventing permutation equivariance. We implement two-dimensional relative self-attention by independently adding relative height
information and relative width information. The attention
logit for how much pixel i= (ix;iy)attends to pixel
j= (jx;jy)is computed as:
li;j=qT
iq
dh
k(kj+rW
jxix+rH
jyiy) (3)


key: chroma:document
string_value:
li;j=qT
iq
dh
k(kj+rW
jxix+rH
jyiy) (3)
whereqiis the query vector for pixel i(the i-th row of Q),
kjis the key vector for pixel j(the j-th row of K) andrW
jxix
andrH
jyiyare learned embeddings for relative width jxix
and relative height jyiy, respectively. The output of head
hnow becomes:
Oh=Softmax0
@QKT+Srel
H+Srel
Wq
dh
k1
AV (4)
whereSrel
H;Srel
W2RHWHWare matrices of relative position logits along height and width dimensions that satisfy
Srel
H[i;j] =qT
irH
jyiyandSrel
W[i;j] =qT
irW
jxix.
The relative attention algorithm in [37] explicitly
stores all relative embeddings rijin a tensor of shape
(HW;HW;dh
k), thus incurring an additional memory cost
ofO((HW)2dh
k). This compares to O((HW)2Nh)for the


key: chroma:document
string_value:
k), thus incurring an additional memory cost
ofO((HW)2dh
k). This compares to O((HW)2Nh)for the
position-unaware version self-attention that does not use
position encodings. As we typically have Nh<dh
k, such an
implementation can prove extremely prohibitive and restrict
the number of images that can ﬁt in a minibatch. Instead, we
extend the memory efﬁcient relative masked attention algorithm presented in [18] to unmasked relative self-attention
over 2 dimensional inputs. Our implementation has a memory cost ofO(HWdh
k). We leave the Tensorﬂow code of
the algorithm in the Appendix.
The relative positional embeeddings rHandrWare
learned and shared across heads but not layers. For each
layer, we add (2(H+W)2)dh
kparameters to model relative distances along height and width.
3.2. Attention Augmented Convolution
Multiple previously proposed attention mechanisms over
images [17, 16, 31, 46] suggest that the convolution operator is limited by its locality and lack of understanding
of global contexts. These methods capture long-range dependencies by recalibrating convolutional feature maps. In


key: chroma:document
string_value:
of global contexts. These methods capture long-range dependencies by recalibrating convolutional feature maps. In
particular, Squeeze-and-Excitation (SE) [17] and GatherExcite (GE) [16] perform channelwise reweighing while
BAM [31] and CBAM [46] reweigh both channels and
spatial positions independently . In contrast to these approaches, we 1) use an attention mechanism that can attend
jointly to spatial and feature subspaces (each head corresponding to a feature subspace) and 2) introduce additional
feature maps rather than reﬁning them. Figure 2 summarizes our proposed augmented convolution.Concatenating convolutional and attentional feature
maps: Formally, consider an original convolution operator with kernel size k,Fininput ﬁlters and Foutoutput
ﬁlters. The corresponding attention augmented convolution
can be written as
AAConv (X) =Concath
Conv (X);MHA (X)i
:
We denote=dv
Foutthe ratio of attentional channels to
number of original output ﬁlters and =dk
Foutthe ratio of
key depth to number of original output ﬁlters. Similarly to


key: chroma:document
string_value:
number of original output ﬁlters and =dk
Foutthe ratio of
key depth to number of original output ﬁlters. Similarly to
the convolution, the proposed attention augmented convolution 1) is equivariant to translation and 2) can readily operate on inputs of different spatial dimensions. We include
Tensorﬂow code for the proposed attention augmented convolution in the Appendix A.3.
Effect on number of parameters: Multihead attention
introduces a 1x1 convolution with Fininput ﬁlters and
(2dk+dv) =Fout(2+)output ﬁlters to compute queries,
keys and values and an additional 1x1 convolution with
dv=Foutinput and output ﬁlters to mix the contribution of different heads. Considering the decrease in ﬁlters
in the convolutional part, this leads to the following change
in parameters:
paramsFinFout(2+ (1k2)+Fout
Fin2);(5)
where we ignore the parameters introduced by relative position embeddings for simplicity as these are negligible. In
practice, this causes a slight decrease in parameters when
replacing 3x3 convolutions and a slight increase in parameters when replacing 1x1 convolutions. Interestingly, we ﬁnd


key: chroma:document
string_value:
practice, this causes a slight decrease in parameters when
replacing 3x3 convolutions and a slight increase in parameters when replacing 1x1 convolutions. Interestingly, we ﬁnd
in experiments that attention augmented networks still signiﬁcantly outperform their fully convolutional counterparts
while using less parameters.
Attention Augmented Convolutional Architectures: In
all our experiments, the augmented convolution is followed
by a batch normalization [20] layer which can learn to scale
the contribution of the convolution feature maps and the attention feature maps. We apply our augmented convolution
once per residual block similarly to other visual attention
mechanisms [17, 16, 31, 46] and along the entire architecture as memory permits (see Section 4 for more details).
Since the memory cost O((Nh(HW)2)can be prohibitive for large spatial dimensions, we augment convolutions with attention starting from the last layer (with smallest spatial dimension) until we hit memory constraints. To
reduce the memory footprint of augmented networks, we
typically resort to a smaller batch size and sometimes additionally downsample the inputs to self-attention in the layers with the largest spatial dimensions where it is applied.
Downsampling is performed by applying 3x3 average pooling with stride 2 while the following upsampling (required
for the concatenation) is obtained via bilinear interpolation.


key: chroma:document
string_value:
Downsampling is performed by applying 3x3 average pooling with stride 2 while the following upsampling (required
for the concatenation) is obtained via bilinear interpolation.
4. Experiments
In the subsequent experiments, we test Attention Augmentation on standard computer vision architectures such
as ResNets [14, 47, 13], and MnasNet [42] on the CIFAR100 [22], ImageNet [9] and COCO [25] datasets. Our experiments show that Attention Augmentation leads to systematic improvements on both image classiﬁcation and object detection tasks across a broad array of architectures and
computational demands. We validate the utility of the proposed two-dimensional relative attention mechanism in ablation experiments. In all experiments, we substitute convolutional feature maps with self-attention feature maps as
it makes for an easier comparison against the baseline models. Unless speciﬁed otherwise, all results correspond to our
two-dimensional relative self-attention mechanism. Experimental details can be found in the Appendix.
4.1. CIFAR-100 image classiﬁcation
We ﬁrst investigate how Attention Augmentation performs on CIFAR-100 [22], a standard benchmark for lowresolution imagery, using a Wide ResNet architecture [51].
The Wide-ResNet-28-10 architecture is comprised of 3


key: chroma:document
string_value:
The Wide-ResNet-28-10 architecture is comprised of 3
stages of 4 residual blocks each using two 33convolutions. We augment the Wide-ResNet-28-10 by augmenting
the ﬁrst convolution of all residual blocks with relative attention using Nh=8 heads and =2=0.2 and a minimum of
20 dimensions per head for the keys. We compare Attention
Augmentation (AA) against other forms of attention including Squeeze-and-Excitation (SE) [17] and the parameterfree formulation of Gather-Excite (GE) [16]. Table 1 shows
that Attention Augmentation improves performance both
over the baseline network and Squeeze-and-Excitation at a
similar parameter and complexity cost.
Architecture Params GFlops top-1 top-5
Wide-ResNet [51] 36.3M 10.4 80.3 95.0
GE-Wide-ResNet [16] 36.3M 10.4 79.8 95.0
SE-Wide-ResNet [17] 36.5M 10.4 81.0 95.3
AA-Wide-ResNet (ours) 36.2M 10.9 81.6 95.2
Table 1. Image classiﬁcation on the CIFAR-100 dataset [22] using


key: chroma:document
string_value:
Table 1. Image classiﬁcation on the CIFAR-100 dataset [22] using
the Wide-ResNet 28-10 architecture [51].
4.2. ImageNet image classiﬁcation with ResNet
We next examine how Attention Augmentation performs
on ImageNet [9, 21], a standard large-scale dataset for high
resolution imagery, across an array of architectures. We
start with the ResNet architecture [14, 47, 13] because of itswidespread use and its ability to easily scale across several
computational budgets. The building block in ResNet-34
comprises two 3x3 convolutions with the same number of
output ﬁlters. ResNet-50 and its larger counterparts use a
bottleneck block comprising of 1x1, 3x3, 1x1 convolutions
where the last pointwise convolution expands the number
of ﬁlters and the ﬁrst one contracts the number of ﬁlters.
We modify all ResNets by augmenting the 3x3 convolutions as this decreases number of parameters.2We apply
Attention Augmentation in each residual block of the last 3
stages of the architecture – when the spatial dimensions of


key: chroma:document
string_value:
Attention Augmentation in each residual block of the last 3
stages of the architecture – when the spatial dimensions of
the activation maps are 28x28, 14x14 and 7x7 – and downsample only during the ﬁrst stage. All attention augmented
networks use =2=0.2, except for ResNet-34 which uses
==0.25. The number of attention heads is ﬁxed to Nh=8.
Architecture Params (M) Infer Train top-1
ResNet-50 25.6 - - 76.4
SE [17] 28.1 +12% +92% 77.5 (77.0)
BAM [31] 25.9 +19% +43% 77.3
CBAM [46] 28.1 +56% +132% 77.4 (77.4)
GALA [28] 29.4 +86% +133% 77.5 (77.3)
AA (= 0:25) 24.3 +29% +25% 77.7
Table 2. Image classiﬁcation performance of different attention
mechanisms on the ImageNet dataset. refers to the increase
in latency times compared to the ResNet50 on a single Tesla V100


key: chroma:document
string_value:
mechanisms on the ImageNet dataset. refers to the increase
in latency times compared to the ResNet50 on a single Tesla V100
GPU with Tensorﬂow using a batch size of 128. For fair comparison, we also include top-1 results (in parentheses) when scaling
networks in width to match 25:6M parameters as the ResNet50
baseline.
Architecture GFlops Params top-1 top-5
ResNet-34 [14] 7.4 21.8M 73.6 91.5
SE-ResNet-34 [17] 7.4 22.0M 74.3 91.8
AA-ResNet-34 (ours) 7.1 20.7M 74.7 92.0
ResNet-50 [14] 8.2 25.6M 76.4 93.1
SE-ResNet-50 [17] 8.2 28.1M 77.5 93.7
AA-ResNet-50 (ours) 8.3 25.8M 77.7 93.8
ResNet-101 [14] 15.6 44.5M 77.9 94.0
SE-ResNet-101 [17] 15.6 49.3M 78.4 94.2


key: chroma:document
string_value:
SE-ResNet-101 [17] 15.6 49.3M 78.4 94.2
AA-ResNet-101 (ours) 16.1 45.4M 78.7 94.4
ResNet-152 [14] 23.0 60.2M 78.4 94.2
SE-ResNet-152 [17] 23.1 66.8M 78.9 94.5
AA-ResNet-152 (ours) 23.8 61.6M 79.1 94.6
Table 3. Image classiﬁcation on the ImageNet dataset [9] across
a range of ResNet architectures: ResNet-34, ResNet-50, Resnet101, and ResNet-152 [14, 47, 13].
Table 2 benchmarks Attention Augmentation against
channel and spatial attention mechanisms BAM [31],
CBAM [46] and GALA [28] with channel reduction ratio= 16 on the ResNet50 architecture. Despite the
2We found that augmenting the pointwise expansions works just as well
but does not save parameters or computations.
Architecture GFlops Params top-1 top-5
MnasNet-0.75 0.45 2.91M 73.3 91.3
AA-MnasNet-0.75 0.51 3.02M 73.9 91.6


key: chroma:document
string_value:
AA-MnasNet-0.75 0.51 3.02M 73.9 91.6
MnasNet-1.0 0.63 3.89M 75.2 92.4
AA-MnasNet-1.0 0.70 4.06M 75.7 92.6
MnasNet-1.25 1.01 5.26M 76.7 93.2
AA-MnasNet-1.25 1.11 5.53M 77.2 93.6
MnasNet-1.4 1.17 6.10M 77.2 93.5
AA-MnasNet-1.4 1.29 6.44M 77.7 93.8
Table 4. Baseline and attention augmented MnasNet [42] accuracies with width multipliers 0.75, 1.0, 1.25 and 1.4.
lack of specialized kernels (See Appendix A.3), Attention
Augmentation offers a competitive accuracy/computational
trade-off compared to previously proposed attention mechanisms. Table 3 compares the non-augmented networks and
Squeeze-and-Excitation (SE) [17] across different network
scales. In all experiments, Attention Augmentation signiﬁcantly increases performance over the non-augmented
baseline and notably outperforms Squeeze-and-Excitation


key: chroma:document
string_value:
baseline and notably outperforms Squeeze-and-Excitation
(SE) [17] while being more parameter efﬁcient (Figure 1).
Remarkably, our AA-ResNet-50 performs comparably to
the baseline ResNet-101 and our AA-ResNet-101 outperforms the baseline ResNet-152. These results suggest that
attention augmentation is preferable to simply making networks deeper. We include and discuss attention maps visualizations from different pixel positions in the appendix.
4.3. ImageNet classiﬁcation with MnasNet
In this section, we inspect the use of Attention Augmentation in a resource constrained setting by conducting
ImageNet experiments with the MnasNet architecture [42],
which is an extremely parameter-efﬁcient architecture. In
particular, the MnasNet was found by neural architecture search [54], using only the highly optimized mobile inverted bottleneck block [36] and the Squeeze-andExcitation operation [17] as the primitives in its search
space. We apply Attention Augmentation to the mobile
inverted bottleneck by replacing convolutional channels in
the expansion pointwise convolution using =2=0.1 and
Nh=4 heads. Our augmented MnasNets use augmented inverted bottlenecks in the last 13 blocks out of 18 in the


key: chroma:document
string_value:
Nh=4 heads. Our augmented MnasNets use augmented inverted bottlenecks in the last 13 blocks out of 18 in the
MnasNet architecture, starting when the spatial dimension
is 28x28. We downsample only in the ﬁrst stage where Attention Augmentation is applied. We leave the ﬁnal pointwise convolution, also referred to as the “head”, unchanged.
In Table 4, we report ImageNet accuracies for the baseline MnasNet and its attention augmented variants at different width multipliers. Our experiments show that Attention Augmentation yields accuracy improvements across
all width multipliers. Augmenting MnasNets with relative
self-attention incurs a slight parameter increase, however
2.5 3.0 3.5 4.0 4.5 5.0 5.5 6.0 6.5 7.0
number of parameters (millions)737475767778top-1 accuracy
73.3%75.2%76.7%77.2%
73.9%75.7%77.2%77.7%
MnasNet
AA-MnasNet (ours)Figure 3. ImageNet top-1 accuracy as a function of number of parameters for MnasNet (black) and Attention-Augmented-MnasNet


key: chroma:document
string_value:
AA-MnasNet (ours)Figure 3. ImageNet top-1 accuracy as a function of number of parameters for MnasNet (black) and Attention-Augmented-MnasNet
(red) with width multipliers 0:75,1:0,1:25and1:4.
we verify in Figure 3 that the accuracy improvements are
not just explained by the parameter increase. Additionally,
we note that the MnasNet architecture employs Squeezeand-Excitation at multiple locations that were optimally selected via architecture search, further suggesting the beneﬁts of our method.
4.4. Object Detection with COCO dataset
We next investigate the use of Attention Augmentation
on the task of object detection on the COCO dataset [27].
We employ the RetinaNet architecture with a ResNet-50
and ResNet-101 backbone as done in [26], using the opensourced RetinaNet codebase.3We apply Attention Augmentation uniquely on the ResNet backbone, modifying
them similarly as in our ImageNet classiﬁcation experiments.
Our relative self-attention mechanism improves the performance of the RetinaNet on both ResNet-50 and ResNet101 as shown in Table 5. Most notably, Attention Augmentation yields a 1.4% mAP improvement over a strong
RetinaNet baseline from [26]. In contrast to the success


key: chroma:document
string_value:
RetinaNet baseline from [26]. In contrast to the success
of Squeeze-and-Excitation in image classiﬁcation with ImageNet, our experiments show that adding Squeeze-andExcitation operators in the backbone network of the RetinaNet signiﬁcantly hurts performance, in spite of grid
searching over the squeeze ratio 2f4;8;16g. We hypothesize that localization requires precise spatial information which SE discards during the spatial pooling operation,
thereby negatively affecting performance. Self-attention on
the other hand maintains spatial information and is likely to
3https://github.com/tensorflow/tpu/tree/master/
models/official/retinanet
Backbone architecture GFlops Params mAP COCO mAP 50 mAP 75
ResNet-50 [26] 182 33.4M 36.8 54.5 39.5
SE-ResNet-50 [17] 183 35.9M 36.5 54.0 39.1
AA-ResNet-50 (ours) 182 33.1M 38.2 56.5 40.7
ResNet-101 [26] 243 52.4M 38.5 56.4 41.2
SE-ResNet-101 [17] 243 57.2M 37.4 55.0 39.9


key: chroma:document
string_value:
SE-ResNet-101 [17] 243 57.2M 37.4 55.0 39.9
AA-ResNet-101 (ours) 245 51.7M 39.2 57.8 41.9
Table 5. Object detection on the COCO dataset [27] using the RetinaNet architecture [26] with different backbone architectures. We report
mean Average Precision at three different IoU values.
be able to identify object boundaries successfully. Visualizations of attention maps (See Figures 9 and 10 in the Appendix) reveal that some heads are indeed delineating objects from their background which might be important for
localization.
4.5. Ablation Study
Fully-attentional vision models: In this section, we investigate the performance of Attention Augmentation as a
function of the fraction of attentional channels. As we increase this fraction to 100%, we begin to replace a ConvNet with a fully attentional model, only leaving pointwise
convolutions and the stem unchanged. Table 6 presents the
performance of Attention Augmentation on the ResNet-50
architecture for varying ratios =2f0:25;0:5;0:75;1:0g.
Performance slightly degrades as the ratio of attentional
channels increases, which we hypothesize is partly explained by the average pooling operation for downsampling


key: chroma:document
string_value:
Performance slightly degrades as the ratio of attentional
channels increases, which we hypothesize is partly explained by the average pooling operation for downsampling
at the ﬁrst stage where Attention Augmentation is applied.
Attention Augmentation proves however quite robust to the
fraction of attentional channels. For instance, AA-ResNet50 with==0.75 outperforms its ResNet-50 counterpart,
while being more parameter and ﬂops efﬁcient, indicating
that mostly employing attentional channels is readily competitive.
Perhaps surprisingly, these experiments also reveal that
our proposed self-attention mechanism is a powerful standalone computational primitive for image classiﬁcation and
that fully attentional models are viable for discriminative visual tasks. In particular, AA-ResNet-50 with ==1, which
uses exclusively attentional channels, is only 2.5% worse
in accuracy than its fully convolutional counterpart, in spite
of downsampling with average pooling and having 25% less
parameters. Notably, this fully attentional architecture4also
outperforms ResNet-34 while being more parameter and
ﬂops efﬁcient (see Table 6).
4We consider pointwise convolutions as dense layers. This architecture


key: chroma:document
string_value:
ﬂops efﬁcient (see Table 6).
4We consider pointwise convolutions as dense layers. This architecture
employs 4 non-pointwise convolutions in the stem and the ﬁrst stage of the
architecture, but we believe such operations can be replaced by attention
too.Architecture GFlops Params top-1 top-5
ResNet-34 [14] 7.4 21.8M 73.6 91.5
ResNet-50 [14] 8.2 25.6M 76.4 93.1
== 0:25 7.9 24.3M 77.7 93.8
== 0:5 7.3 22.3M 77.3 93.6
== 0:75 6.8 20.7M 76.7 93.2
== 1:0 6.3 19.4M 73.9 91.5
Table 6. Attention Augmented ResNet-50 with varying ratios of
attentional channels.
0 25 50 75 100
fraction of attention versus convolutional channels707274767880top-1 accuracy76.4%77.5%
76.6%
75.4%
71.1%77.7%
77.3%
76.7%
73.9%


key: chroma:document
string_value:
76.6%
75.4%
71.1%77.7%
77.3%
76.7%
73.9%
no position
with position
Figure 4. Effect of relative position embeddings as the ratio
of attentional channels increases on our Attention-Augmented
ResNet50.
Importance of position encodings: In Figure 4, we show
the effect of our proposed two-dimensional relative position encodings as a function of the fraction of attentional
channels. As expected, experiments demonstrate that our
relative position encodings become increasingly more important as the architecture employs more attentional channels. In particular, the fully self-attentional ResNet-50 gains
2.8% top-1 ImageNet accuracy when using relative position
encodings, which indicates the necessity of maintaining position information for fully self-attentional vision models.
We additionally compare our proposed two-dimensional
Architecture Position Encodings top-1 top-5
AA-ResNet-34 None 74.4 91.9
AA-ResNet-34 2d Sine 74.4 92.0
AA-ResNet-34 CoordConv 74.4 92.0
AA-ResNet-34 Relative (ours) 74.7 92.0
AA-ResNet-50 None 77.5 93.7


key: chroma:document
string_value:
AA-ResNet-34 Relative (ours) 74.7 92.0
AA-ResNet-50 None 77.5 93.7
AA-ResNet-50 2d Sine 77.5 93.7
AA-ResNet-50 CoordConv 77.5 93.8
AA-ResNet-50 Relative (ours) 77.7 93.8
Table 7. Effects of different position encodings in Attention Augmentation on ImageNet classiﬁcation.
Position Encodings mAP COCO mAP 50 mAP 75
None 37.7 56.0 40.2
CoordConv [29] 37.4 55.5 40.1
Relative (ours) 38.2 56.5 40.7
Table 8. Effects of different position encodings in Attention Augmentation on the COCO object detection task using a RetinaNet
AA-ResNet-50 backbone.
relative position encodings to other position encoding
schemes. We apply Attention Augmentation using the same
hyperparameters as 4.2 with the following different position encoding schemes: 1) The position-unaware version of
self-attention (referred to as None ), 2) a two-dimensional
implementation of the sinusoidal positional waves (referred


key: chroma:document
string_value:
self-attention (referred to as None ), 2) a two-dimensional
implementation of the sinusoidal positional waves (referred
to as 2d Sine ) as used in [32], 3) CoordConv [29] for which
we concatenate (x,y,r) coordinate channels to the inputs of
the attention function, and 4) our proposed two-dimensional
relative position encodings (referred to as Relative ).
In Table 7 and 8, we present the results on ImageNet
classiﬁcation and the COCO object detection task respectively. On both tasks, Attention Augmentation without position encodings already yields improvements over the fully
convolutional non-augmented variants. Our experiments
also reveal that the sinusoidal encodings and the coordinate
convolution do not provide improvements over the positionunaware version of Attention Augmentation. We obtain additional improvements when using our two-dimensional relative attention, demonstrating the utility of preserving translation equivariance while preventing permutation equivariance.
5. Discussion and future work
In this work, we consider the use of self-attention for vision models as an alternative to convolutions. We introduce
a novel two-dimensional relative self-attention mechanism
for images that enables training of competitive fully selfattentional vision models on image classiﬁcation for the ﬁrst


key: chroma:document
string_value:
for images that enables training of competitive fully selfattentional vision models on image classiﬁcation for the ﬁrst
time. We propose to augment convolutional operators with
this self-attention mechanism and validate the superiority ofthis approach over other attention schemes. Extensive experiments show that Attention Augmentation leads to systematic improvements on both image classiﬁcation and object detection tasks across a wide range of architectures and
computational settings.
Several open questions from this work remain. In future work, we will focus on the fully attentional regime
and explore how different attention mechanisms trade off
computational efﬁciency versus representational power. For
instance, identifying a local attention mechanism may result in an efﬁcient and scalable computational mechanism
that could prevent the need for downsampling with average
pooling [34]. Additionally, it is plausible that architectural
design choices that are well suited when exclusively relying
on convolutions are suboptimal when using self-attention
mechanisms. As such, it would be interesting to see if using Attention Augmentation as a primitive in automated architecture search procedures proves useful to ﬁnd even better models than those previously found in image classiﬁcation [55], object detection [12], image segmentation [6] and
other domains [5, 1, 35, 8]. Finally, one can ask to which


key: chroma:document
string_value:
other domains [5, 1, 35, 8]. Finally, one can ask to which
degree fully attentional models can replace convolutional
networks for visual tasks.
Acknowledgements
The authors would like to thank Tsung-Yi Lin, Prajit Ramachandran, Mingxing Tan, Yanping Huang and the
Google Brain team for insightful comments and discussions.
References
[1] Maximilian Alber, Irwan Bello, Barret Zoph, Pieter-Jan Kindermans, Prajit Ramachandran, and Quoc V . Le. Backprop
evolution. CoRR , abs/1808.02822, 2018. 8
[2] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio.
Neural machine translation by jointly learning to align and
translate. In International Conference on Learning Representations , 2015. 2
[3] Irwan Bello, Sayali Kulkarni, Sagar Jain, Craig Boutilier,
Ed Huai-hsin Chi, Elad Eban, Xiyang Luo, Alan Mackey,
and Ofer Meshi. Seq2slate: Re-ranking and slate optimization with rnns. CoRR , abs/1810.02019, 2018. 2


key: chroma:document
string_value:
and Ofer Meshi. Seq2slate: Re-ranking and slate optimization with rnns. CoRR , abs/1810.02019, 2018. 2
[4] Irwan Bello, Hieu Pham, Quoc V . Le, Mohammad Norouzi,
and Samy Bengio. Neural combinatorial optimization with
reinforcement learning. 2016. 2
[5] Irwan Bello, Barret Zoph, Vijay Vasudevan, and Quoc V . Le.
Neural optimizer search with reinforcement learning. In Proceedings of the 34th International Conference on Machine
Learning - Volume 70 , ICML’17, pages 459–468. JMLR.org,
2017. 8
[6] Liang-Chieh Chen, Maxwell Collins, Yukun Zhu, George
Papandreou, Barret Zoph, Florian Schroff, Hartwig Adam,
and Jon Shlens. Searching for efﬁcient multi-scale architectures for dense image prediction. In Advances in Neural
Information Processing Systems , pages 8713–8724, 2018. 2,
8
[7] Yunpeng Chen, Yannis Kalantidis, Jianshu Li, Shuicheng
Yan, and Jiashi Feng. A2-nets: Double attention networks.


key: chroma:document
string_value:
[7] Yunpeng Chen, Yannis Kalantidis, Jianshu Li, Shuicheng
Yan, and Jiashi Feng. A2-nets: Double attention networks.
CoRR , abs/1810.11579, 2018. 3
[8] Ekin Dogus Cubuk, Barret Zoph, Dandelion Man ´e, Vijay Vasudevan, and Quoc V . Le. Autoaugment: Learning augmentation policies from data. CoRR , abs/1805.09501, 2018. 8
[9] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li,
and Li Fei-Fei. Imagenet: A large-scale hierarchical image
database. In IEEE Conference on Computer Vision and Pattern Recognition . IEEE, 2009. 1, 2, 5
[10] Xavier Gastaldi. Shake-shake regularization. arXiv preprint
arXiv:1705.07485 , 2017. 11
[11] Golnaz Ghiasi, Tsung-Yi Lin, and Quoc V Le. Dropblock:
A regularization method for convolutional networks. In
Advances in Neural Information Processing Systems , pages
10750–10760, 2018. 11


key: chroma:document
string_value:
A regularization method for convolutional networks. In
Advances in Neural Information Processing Systems , pages
10750–10760, 2018. 11
[12] Golnaz Ghiasi, Tsung-Yi Lin, Ruoming Pang, and Quoc V
Le. NAS-FPN: Learning scalable feature pyramid architecture for object detection. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , June 2019. 8
[13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition , 2016.
2, 5
[14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
Identity mappings in deep residual networks. In European
Conference on Computer Vision , 2016. 1, 2, 5, 7, 11
[15] Sepp Hochreiter and Juergen Schmidhuber. Long short-term
memory. Neural Computation , 1997. 2
[16] Jie Hu, Li Shen, Samuel Albanie, Gang Sun, and Andrea
Vedaldi. Gather-excite: Exploiting feature context in convolutional neural networks. In Advances in Neural Information
Processing Systems , pages 9423–9433, 2018. 3, 4, 5


key: chroma:document
string_value:
Processing Systems , pages 9423–9433, 2018. 3, 4, 5
[17] Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitation networks. In Proceedings of the IEEE Conference on Computer
Vision and Pattern Recognition , 2018. 1, 2, 3, 4, 5, 6, 7
[18] Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Noam Shazeer, Curtis Hawthorne, Andrew M Dai,
Matthew D Hoffman, and Douglas Eck. Music transformer.
InAdvances in Neural Processing Systems , 2018. 3, 4
[19] Jonathan Huang, Vivek Rathod, Chen Sun, Menglong Zhu,
Anoop Korattikara, Alireza Fathi, Ian Fischer, Zbigniew Wojna, Yang Song, Sergio Guadarrama, et al. Speed/accuracy trade-offs for modern convolutional object detectors. In
IEEE Conference on Computer Vision and Pattern Recognition, 2017. 2
[20] Sergey Ioffe and Christian Szegedy. Batch normalization:
Accelerating deep network training by reducing internal covariate shift. In International Conference on Learning Representations , 2015. 4


key: chroma:document
string_value:
Accelerating deep network training by reducing internal covariate shift. In International Conference on Learning Representations , 2015. 4
[21] Simon Kornblith, Jonathon Shlens, and Quoc V Le. Do better imagenet models transfer better? In Proceedings of the
IEEE Conference on Computer Vision and Pattern Recognition, 2019. 2, 5[22] Alex Krizhevsky. Learning multiple layers of features from
tiny images. Technical report, University of Toronto, 2009.
2, 5
[23] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton.
Imagenet classiﬁcation with deep convolutional neural networks. In Advances in Neural Information Processing System, 2012. 1, 2
[24] Yann LeCun, L ´eon Bottou, Yoshua Bengio, and Patrick
Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 1998. 1
[25] Tsung-Yi Lin, Piotr Doll ´ar, Ross Girshick, Kaiming He,
Bharath Hariharan, and Serge Belongie. Feature pyramid
networks for object detection. In Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition ,
2017. 5


key: chroma:document
string_value:
networks for object detection. In Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition ,
2017. 5
[26] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and
Piotr Doll ´ar. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer
vision , pages 2980–2988, 2017. 6, 7, 11
[27] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays,
Pietro Perona, Deva Ramanan, Piotr Doll ´ar, and C Lawrence
Zitnick. Microsoft coco: Common objects in context. In
European Conference on Computer Vision , pages 740–755.
Springer, 2014. 2, 6, 7
[28] Drew Linsley, Dan Scheibler, Sven Eberhardt, and Thomas
Serre. Global-and-local attention networks for visual recognition. CoRR , abs/1805.08819, 2018. 5
[29] Rosanne Liu, Joel Lehman, Piero Molino, Felipe Petroski
Such, Eric Frank, Alex Sergeev, and Jason Yosinski. An
intriguing failing of convolutional neural networks and the


key: chroma:document
string_value:
Such, Eric Frank, Alex Sergeev, and Jason Yosinski. An
intriguing failing of convolutional neural networks and the
coordconv solution. In Advances in Neural Information Processing Systems , pages 9628–9639, 2018. 3, 8
[30] Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. arXiv preprint
arXiv:1608.03983 , 2016. 11
[31] Jongchan Park, Sanghyun Woo, Joon-Young Lee, and In So
Kweon. Bam: bottleneck attention module. In British Machine Vision Conference , 2018. 3, 4, 5
[32] Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Łukasz
Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In International Conference on Machine
Learning , 2018. 3, 8
[33] Andrew Rabinovich, Andrea Vedaldi, Carolina Galleguillos,
Eric Wiewiora, and Serge Belongie. Objects in context.
2007. 1
[34] Prajit Ramachandran, Niki Parmar, Ashish Vaswani, Irwan
Bello, Anselm Levskaya, and Jonathon Shlens. Stand-alone


key: chroma:document
string_value:
Bello, Anselm Levskaya, and Jonathon Shlens. Stand-alone
self-attention in vision models. CoRR , abs/1906.05909,
2019. 8
[35] Prajit Ramachandran, Barret Zoph, and Quoc V . Le. Searching for activation functions. CoRR , abs/1710.05941, 2017.
8
[36] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted
residuals and linear bottlenecks. In Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition ,
pages 4510–4520, 2018. 6
[37] Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Selfattention with relative position representations. arXiv
preprint arXiv:1803.02155 , 2018. 2, 3, 4
[38] David R. So, Chen Liang, and Quoc V . Le. The evolved
transformer. CoRR , abs/1901.11117, 2019. 3
[39] Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and
Alex Alemi. Inception-v4, Inception-Resnet and the impact


key: chroma:document
string_value:
[39] Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and
Alex Alemi. Inception-v4, Inception-Resnet and the impact
of residual connections on learning. In International Conference on Learning Representations Workshop Track , 2016.
2
[40] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet,
Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent
Vanhoucke, and Andrew Rabinovich. Going deeper with
convolutions. In IEEE Conference on Computer Vision and
Pattern Recognition , 2015. 2
[41] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon
Shlens, and Zbigniew Wojna. Rethinking the Inception architecture for computer vision. In IEEE Conference on Computer Vision and Pattern Recognition , 2016. 2, 11
[42] Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan,
and Quoc V Le. Mnasnet: Platform-aware neural architecture search for mobile. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2018. 2,
5, 6, 11


key: chroma:document
string_value:
5, 6, 11
[43] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia
Polosukhin. Attention is all you need. In Advances in Neural
Information Processing Systems , pages 5998–6008, 2017. 1,
2, 3
[44] Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. Pointer
networks. In NIPS , pages 2692–2700, 2015. 2
[45] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proceedings of the
IEEE Conference on Computer Vision and Pattern Recognition, pages 7794–7803, 2018. 3
[46] Sanghyun Woo, Jongchan Park, Joon-Young Lee, and In
So Kweon. Cbam: Convolutional block attention module.
InProceedings of the European Conference on Computer Vision (ECCV) , pages 3–19, 2018. 3, 4, 5
[47] Saining Xie, Ross Girshick, Piotr Doll ´ar, Zhuowen Tu, and
Kaiming He. Aggregated residual transformations for deep


key: chroma:document
string_value:
[47] Saining Xie, Ross Girshick, Piotr Doll ´ar, Zhuowen Tu, and
Kaiming He. Aggregated residual transformations for deep
neural networks. In Proceedings of the IEEE Conference on
Computer Vision and Pattern Recognition , 2017. 2, 5
[48] Yoshihiro Yamada, Masakazu Iwamura, Takuya Akiba, and
Koichi Kise. Shakedrop regularization for deep residual
learning. arXiv preprint arXiv:1802.02375 , 2018. 11
[49] Baosong Yang, Longyue Wang, Derek F. Wong, Lidia S.
Chao, and Zhaopeng Tu. Convolutional self-attention network. In CoRR , volume abs/1810.13320, 2018. 2
[50] Adams Wei Yu, David Dohan, Minh-Thang Luong, Rui
Zhao, Kai Chen, Mohammad Norouzi, and Quoc V Le.
QAnet: Combining local convolution with global selfattention for reading comprehension. In International Conference on Learning Representations , 2018. 2


key: chroma:document
string_value:
QAnet: Combining local convolution with global selfattention for reading comprehension. In International Conference on Learning Representations , 2018. 2
[51] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In British Machine Vision Conference , 2016. 5[52] Vinicius Zambaldi, David Raposo, Adam Santoro, Victor Bapst, Yujia Li, Igor Babuschkin, Karl Tuyls, David
Reichert, Timothy Lillicrap, Edward Lockhart, Murray
Shanahan, Victoria Langston, Razvan Pascanu, Matthew
Botvinick, Oriol Vinyals, and Peter Battaglia. Deep reinforcement learning with relational inductive biases. In ICLR ,
2019. 2
[53] Han Zhang, Ian J. Goodfellow, Dimitris N. Metaxas, and
Augustus Odena. Self-attention generative adversarial networks. arXiv:1805.08318 , 2018. 3
[54] Barret Zoph and Quoc V . Le. Neural architecture search
with reinforcement learning. In International Conference on
Learning Representations , 2017. 6
[55] Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V
Le. Learning transferable architectures for scalable image


key: chroma:document
string_value:
[55] Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V
Le. Learning transferable architectures for scalable image
recognition. In Proceedings of the IEEE conference on
computer vision and pattern recognition , pages 8697–8710,
2018. 2, 8, 11
A. Appendix
A.1. Experimental details
Tuning Unless speciﬁed otherwise, we use the default hyperparameters found in reference baseline codebases without tuning.was searched inf0.1, 0.2, 0.5g,inf0.0, 0.1,
0.25, 0.5, 0.75, 1.0 gand the number of heads was chosen
based on memory constraints (starting from 8 and decreasing when necessary). We report the ﬁnal accuracy for each
run without performing early stopping.
CIFAR-100 Given the low resolution of CIFAR-100 images, we do not downsample feature maps before the attention operation and instead resort to a smaller batch size.
We train all networks for 500 epochs using synchronous
SGD with momentum 0.9 distributed across 8 TESLA V100
GPUs. The learning rate is linearly scaled from 0 to
0:2B=256, whereBis the total batch size, for the ﬁrst 5%


key: chroma:document
string_value:
GPUs. The learning rate is linearly scaled from 0 to
0:2B=256, whereBis the total batch size, for the ﬁrst 5%
training epochs and then annealed with cosine decay [30].
We use standard CIFAR preprocessing: mean normalizing,
random ﬂipping and cropping [55, 10, 48]. Non-augmented
architectures are trained with a batch size of 1024 and a
weight decay of 2e-4. Augmented architectures are trained
with batch size of 256 and a weight decay of 5e-4.
ImageNet classiﬁcation with ResNet We train all
ResNet architectures for 100 epochs using synchronous
SGD with momentum 0.9 across 8 TESLA V100 GPUs and
weight decay of 1e-4. We use the largest batch size per
workerB2f32;64;128;256gthat ﬁts in a minibatch. The
initial learning rate is scaled linearly according to the total
batch size using a base learning rate of 0.128 for total batch
size of 256. During training, we linearly scale the learning
rate from 0 to this value for the ﬁrst 5% of training epochs
and divide it by 10 at epochs 30, 60, 80 and 90. We use


key: chroma:document
string_value:
rate from 0 to this value for the ﬁrst 5% of training epochs
and divide it by 10 at epochs 30, 60, 80 and 90. We use
standard Inception data augmentation as described in [41].
ImageNet classiﬁcation with MnasNet We follow the
training setup described in [42] and train all networks for
350 epochs with the RMSProp optimizer using exponential
learning rate decay. When training our augmented MnasNets, we divide the learning rate by 2 and adjusted the learning rate decay so that the ﬁnal learning rate stays the same.
Object Detection with COCO dataset We follow the
setup described in [26, 11] and train the RetinaNet from
scratch for 150 epochs without using ImageNet pretraining
for the ResNet backbone.
A.2. Computational & Memory costs
Table 9 provides the breakdown of self-attention related computational costs per image. Storing attention
maps in each layer induces a memory cost of Nh(HW)2bfloat16 . At inference, the memory cost for storing attention maps is only 1.2% of the memory required to store
model parameters (49MB).
Layer Memory Params FLOPS
fStage 2 - H=W=14g* 4 600KB 43k 22M


key: chroma:document
string_value:
model parameters (49MB).
Layer Memory Params FLOPS
fStage 2 - H=W=14g* 4 600KB 43k 22M
fStage 3 - H=W=14g* 6 600KB 90k 40M
fStage 4 - H=W=7g* 3 37.5KB 190k 19M
Training 6MB (total) 1.3M 390M
Inference 600KB (max) 1.3M 390M
Table 9. Computational costs associated with self-attention in the
forward pass of the ResNet50. During inference, we only consider
the largest memory cost since activations are not stored.
Figures 5 and 6 show the accuracies of our attention
augmented networks across FLOPS counts, which correlate
with running times across hardware platforms.
5 10 15 20 25
computational demand (GFlops).72737475767778798081top-1 accuracy.
73.6%76.4%77.9%78.4%
74.3%77.5%78.4%78.9%
74.7%77.7%78.7%79.1%
ResNet
SE-ResNet
AA-ResNet (ours)
Figure 5. ImageNet top-1 accuracy as a function of computational


key: chroma:document
string_value:
ResNet
SE-ResNet
AA-ResNet (ours)
Figure 5. ImageNet top-1 accuracy as a function of computational
demand for variety of ResNet architectures [14]. From left to right:
ResNet-34, ResNet-50, ResNet-101 and ResNet-152.
0.4 0.6 0.8 1.0 1.2 1.4
computational demand (GFlops).737475767778top-1 accuracy
73.3%75.2%76.7%77.2%
73.9%75.7%77.2%77.7%
MnasNet
AA-MnasNet (ours)
Figure 6. ImageNet top-1 accuracy as a function of computational
demand for MnasNet (black) and Attention-Augmented-MnasNet
(red) with width multipliers 0:75,1:0,1:25and1:4.
A.3. 2D Relative Self-Attention implementation
While our method is simple and only requires matrix
multiplication, addition and the softmax operation (Equations 3 and 4), our implementation relies on non-trivial operations (e.g. tiling, transposing and reshaping) because
no low-level kernels currently exist for hardware platforms.
Future work may develop specialized kernels as previously


key: chroma:document
string_value:
no low-level kernels currently exist for hardware platforms.
Future work may develop specialized kernels as previously
done for convolutions. Therefore, we believe that current
latency times (Table 2) reﬂect the lack of dedicated engineering as opposed to inefﬁciency in the proposed method.
def shape list(x):
"""Return list of dims, statically where possible."""
static = x.get shape().as list()
shape = tf.shape(x)
ret = []
for i, static dim in enumerate(static):
dim = static dim or shape[i]
ret.append(dim)
return ret
def split heads 2d(inputs, Nh):
"""Split channels into multiple heads."""
B, H, W, d = shape list(inputs)
retshape = [B, H, W, Nh, d // Nh]
split = tf.reshape(inputs, ret shape)
return tf.transpose(split, [0, 3, 1, 2, 4])
def combine heads 2d(inputs):
"""Combine heads (inverse of split heads 2d)."""
transposed = tf.transpose(inputs, [0, 2, 3, 1, 4])
Nh, channels = shape list(transposed)[ 2:]


key: chroma:document
string_value:
transposed = tf.transpose(inputs, [0, 2, 3, 1, 4])
Nh, channels = shape list(transposed)[ 2:]
retshape = shape list(transposed)[: 2] + [Nh channels]
return tf.reshape(transposed, ret shape)
def rel toabs(x):
"""Converts tensor from relative to aboslute indexing."""
# [B, Nh, L, 2L 1]
B, Nh, L, = shape list(x)
# Pad to shift from relative to absolute indexing.
colpad = tf.zeros((B, Nh, L, 1))
x = tf.concat([x, col pad], axis=3)
flat x = tf.reshape(x, [B, Nh, L 2L])
flat pad = tf.zeros((B, Nh, L 1))
flat xpadded = tf.concat([flat x, flat pad], axis=2)
# Reshape and slice out the padded elements.
final x = tf.reshape(flat xpadded, [B, Nh, L+1, 2 L1])


key: chroma:document
string_value:
final x = tf.reshape(flat xpadded, [B, Nh, L+1, 2 L1])
final x = final x[:, :, :L, L 1:]
return final x
def relative logits 1d(q, rel k, H, W, Nh, transpose mask):
"""Compute relative logits along one dimenion."""
rellogits = tf.einsum(’bhxyd,md >bhxym’, q, rel k)
# Collapse height and heads
rellogits = tf.reshape(
rellogits, [ 1, Nh H, W, 2 W1])
rellogits = rel toabs(rel logits)
# Shape it and tile height times
rellogits = tf.reshape(rel logits, [ 1, Nh, H, W, W])
rellogits = tf.expand dims(rel logits, axis=3)
rellogits = tf.tile(rel logits, [1, 1, 1, H, 1, 1])
# Reshape for adding to the logits.
rellogits = tf.transpose(rel logits, transpose mask)


key: chroma:document
string_value:
# Reshape for adding to the logits.
rellogits = tf.transpose(rel logits, transpose mask)
rellogits = tf.reshape(rel logits, [ 1, Nh, H W, HW])
return rel logits
Figure 7. Helper functions in Tensorﬂow for 2D relative selfattention.def relative logits(q, H, W, Nh, dkh):
"""Compute relative logits."""
# Relative logits in width dimension first.
relembeddings w = tf.get variable(
’rwidth’, shape=(2 W1, dkh),
initializer=tf.random normal initializer(dkh 0.5))
# [B, Nh, HW, HW]
rellogits w = relative logits 1d(
q, rel embeddings w, H, W, Nh, [0, 1, 2, 4, 3, 5])
# Relative logits in height dimension next.
# For ease, we 1) transpose height and width,
# 2) repeat the above steps and
# 3) transpose to eventually put the logits
# in their right positions.
relembeddings h = tf.get variable(


key: chroma:document
string_value:
# 2) repeat the above steps and
# 3) transpose to eventually put the logits
# in their right positions.
relembeddings h = tf.get variable(
’rheight’, shape=(2 H1, dkh),
initializer=tf.random normal initializer(dkh 0.5))
# [B, Nh, HW, HW]
rellogits h = relative logits 1d(
tf.transpose(q, [0, 1, 3, 2, 4]),
relembeddings h, W, H, Nh, [0, 1, 4, 2, 5, 3])
return rel logits h, rel logits w
def self attention 2d(inputs, dk, dv, Nh, relative=True):
"""2d relative self attention."""
, H, W, = shape list(inputs)
dkh = dk // Nh
dvh = dv // Nh
flatten hw = lambda x, d: tf.reshape(x, [ 1, Nh, H W, d])
# Compute q, k, v
kqv = tf.layers.conv2d(inputs, 2 dk + dv, 1)


key: chroma:document
string_value:
# Compute q, k, v
kqv = tf.layers.conv2d(inputs, 2 dk + dv, 1)
k, q, v = tf.split(kqv, [dk, dk, dv], axis=3)
q= dkh   0.5 # scaled dot product
# After splitting, shape is [B, Nh, H, W, dkh or dvh]
q = split heads 2d(q, Nh)
k = split heads 2d(k, Nh)
v = split heads 2d(v, Nh)
# [B, Nh, HW, HW]
logits = tf.matmul(flatten hw(q, dkh), flatten hw(k, dkh),
transpose b=True)
if relative:
rellogits h, rel logits w = relative logits(q, H, W, Nh,
dkh)
logits += rel logits h
logits += rel logits w
weights = tf.nn.softmax(logits)
attn out = tf.matmul(weights, flatten hw(v, dvh))


key: chroma:document
string_value:
weights = tf.nn.softmax(logits)
attn out = tf.matmul(weights, flatten hw(v, dvh))
attn out = tf.reshape(attn out, [ 1, Nh, H, W, dvh])
attn out = combine heads 2d(attn out)
# Project heads
attn out = tf.layers.conv2d(attn out, dv, 1)
return attn out
def augmented conv2d(X, Fout, k, dk, dv, Nh, relative):
conv out = tf.layers.conv2d(inputs=X, filters=Fout dv,
kernel size=k, padding=’same’)
attn out = self attention 2d(X, dk, dv, Nh, relative=
relative)
return tf.concat([conv out, attn out], axis=3)
Figure 8. Tensorﬂow code for 2D relative self-attention.
A.4. Attention visualizations.
In Figure 10, we present attention maps visualizations
for the input image shown in Figure 9. We see that attention
heads learn to specialize to different content and notably can
delineate object boundaries.


key: chroma:document
string_value:
for the input image shown in Figure 9. We see that attention
heads learn to specialize to different content and notably can
delineate object boundaries.
Figure 9. An input image. The red crosses indexed 1 to 4 represent
the pixel locations for which we show the attention maps in Figure
10.
Figure 10. Visualization of attention maps for an augmented convolution in the Attention-Augmented-ResNet50. Rows correspond to the 8 different heads and columns correspond to the 4
pixel locations depicted in the input image (See Figure 9).


key: chroma:document
string_value:
Learning Temporal Pose Estimation
from Sparsely-Labeled Videos
Gedas Bertasius1;2, Christoph Feichtenhofer1, Du Tran1, Jianbo Shi2, Lorenzo Torresani1
1Facebook AI,2University of Pennsylvania
Abstract
Modern approaches for multi-person pose estimation in video require large amounts
of dense annotations. However, labeling every frame in a video is costly and labor
intensive. To reduce the need for dense annotations, we propose a PoseWarper
network that leverages training videos with sparse annotations (every kframes) to
learn to perform dense temporal pose propagation and estimation. Given a pair
of video frames—a labeled Frame A and an unlabeled Frame B—we train our
model to predict human pose in Frame A using the features from Frame B by
means of deformable convolutions to implicitly learn the pose warping between A
and B. We demonstrate that we can leverage our trained PoseWarper for several
applications. First, at inference time we can reverse the application direction
of our network in order to propagate pose information from manually annotated
frames to unlabeled frames. This makes it possible to generate pose annotations
for the entire video given only a few manually-labeled frames. Compared to
modern label propagation methods based on optical ﬂow, our warping mechanism


key: chroma:document
string_value:
for the entire video given only a few manually-labeled frames. Compared to
modern label propagation methods based on optical ﬂow, our warping mechanism
is much more compact ( 6M vs 39M parameters), and also more accurate ( 88:7%
mAP vs 83:8%mAP). We also show that we can improve the accuracy of a
pose estimator by training it on an augmented dataset obtained by adding our
propagated poses to the original manual labels. Lastly, we can use our PoseWarper
to aggregate temporal pose information from neighboring frames during inference.
This allows us to obtain state-of-the-art pose detection results on PoseTrack2017
and PoseTrack2018 datasets. Code has been made available at: https://github.
com/facebookresearch/PoseWarper .
1 Introduction
In recent years, visual understanding methods [ 1–15] have made tremendous progress, partly because
of advances in deep learning [ 16–19], and partly due to the introduction of large-scale annotated
datasets [ 20,21]. In this paper we consider the problem of pose estimation, which has greatly
beneﬁtted from the recent creation of large-scale datasets [ 22,23]. Most of the recent advances
in this area, though, have concentrated on the task of pose estimation in still-images [ 3,23–27].


key: chroma:document
string_value:
in this area, though, have concentrated on the task of pose estimation in still-images [ 3,23–27].
However, directly applying these image-level models to video is challenging due to nuisance factors
such as motion blur, video defocus, and frequent pose occlusions. Additionally, the process of
collecting annotated pose data in multi-person videos is costly and time consuming. A video typically
contains hundreds of frames that need to be densely-labeled by human annotators. As a result,
datasets for video pose estimation [ 22] are typically smaller and less diverse compared to their image
counterparts [ 21]. This is problematic because modern deep models require large amounts of labeled
data to achieve good performance. At the same time, videos have high informational redundancy as
the content changes little from frame to frame. This raises the question of whether every single frame
in a training video needs to be labeled in order to achieve good pose estimation accuracy.
33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada.arXiv:1906.04016v3  [cs.CV]  11 Dec 2019
To reduce the reliance on densely annotated video pose data, in this work, we introduce the PoseWarper network, which operates on sparsely annotated videos, i.e., videos where pose annotations


key: chroma:document
string_value:
are given only every kframes. Given a pair of frames from the same video—a labeled Frame A
and an unlabeled Frame B—we train our model to detect pose in Frame A, using the features from
Frame B. To achieve this goal, our model leverages deformable convolutions [ 28] across space and
time. Through this mechanism, our model learns to sample features from an unlabeled Frame B to
maximize pose detection accuracy in a labeled Frame A.
Our trained PoseWarper can then be used for several applications. First, we can leverage PoseWarper
to propagate pose information from a few manually-labeled frames across the entire video. Compared
to modern optical ﬂow propagation methods such as FlowNet2 [ 29], our PoseWarper produces more
accurate pose annotations ( 88:7%mAP vs 83:8%mAP), while also employing a much more compact
warping mechanism ( 6M vs 39M parameters). Furthermore, we show that our propagated poses can
serve as effective pseudo labels for training a more accurate pose detector. Finally, our PoseWarper
can be used to aggregate temporal pose information from neighboring frames during inference. This
naturally renders the approach more robust to occlusion or motion blur in individual frames, and leads
to state-of-the-art pose detection results on the PoseTrack2017 and PoseTrack2018 datasets [22].


key: chroma:document
string_value:
to state-of-the-art pose detection results on the PoseTrack2017 and PoseTrack2018 datasets [22].
2 Related Work
Multi-Person Pose Detection in Images. The traditional approaches for pose estimation leverage
pictorial structures model [ 30–34], which represents human body as a tree-structured graph with
pairwise potentials between the connected body parts. These approaches have been highly successful
in the past, but they tend to fail if some of body parts are occluded. These issues have been partially
addressed by the models that assume a non-tree graph structure [ 35–38]. However, most modern
approaches for single image pose estimation are based on convolutional neural networks [ 3,6,23–
27,39–45]. The method in [ 3] regresses (x;y)joint coordinates directly from the images. More
recent work [ 25] instead predicts pose heatmaps, which leads to an easier optimization problem.
Several approaches [ 24,26,39,46] propose an iterative pose estimation pipeline where the predictions
are reﬁned at different stages inside a CNN or via a recurrent network. The methods in [ 6,23,45]
tackle pose estimation problem in a top-down fashion, ﬁrst detecting bounding boxes of people, and


key: chroma:document
string_value:
tackle pose estimation problem in a top-down fashion, ﬁrst detecting bounding boxes of people, and
then predicting the pose heatmaps from the cropped images. The work in [ 24] proposes part afﬁnity
ﬁelds module that captures pairwise relationships between different body parts. The approaches
in [42,43] leverage a bottom-up pipeline ﬁrst predicting the keypoints, and then assembling them
into instances. Lastly, a recent work in [ 27], proposes an architecture that preserves high resolution
feature maps, which is shown to be highly beneﬁcial for the multi-person pose estimation task.
Multi-Person Pose Detection in Video. Due to a limited number of large scale benchmarks for
video pose detection, there has been signiﬁcantly fewer methods in the video domain. Several prior
methods [ 22,47,48] tackle a video pose estimation task as a two-stage problem, ﬁrst detecting the
keypoints in individual frames, and then applying temporal smoothing techniques. The method in [ 49]
proposes a spatiotemporal CRF, which is jointly optimized for the pose prediction in video. The
work in [ 50] proposes a personalized video pose estimation framework, which is accomplished by


key: chroma:document
string_value:
proposes a spatiotemporal CRF, which is jointly optimized for the pose prediction in video. The
work in [ 50] proposes a personalized video pose estimation framework, which is accomplished by
ﬁnetuning the model on a few frames with high conﬁdence keypoints in each video. The approaches
in [51,52] leverage ﬂow based representations for aligning features temporally across multiple
frames, and then using such aligned features for pose detection in individual frames.
In contrast to these prior methods, our primary objective is to learn an effective video pose detector
from sparsely labeled videos. Our approach has similarities to the methods in [ 51,52], which use
ﬂow representations for feature alignment. However, unlike our model, the methods in [ 51,52] do
not optimize their ﬂow representations end-to-end with respect to the pose detection task. As we will
show in our experiments, this is important for strong performance.
3 The PoseWarper Network
Overview. Our goal is to design a model that learns to detect pose from sparsely labeled videos.
Speciﬁcally, we assume that pose annotations in training videos are available every kframes. Inspired
by a recent self-supervised approach for learning facial attribute embeddings [ 53], we formulate the


key: chroma:document
string_value:
by a recent self-supervised approach for learning facial attribute embeddings [ 53], we formulate the
following task. Given two video frames—a labeled Frame A and an unlabeled Frame B—our model
2
1. Training (Unlabeled      Labeled) 
2. Video Pose Propagation (Labeled      Unlabeled)


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:
Ground Truth Pose Propagated to Frame BFeature Warping
Ground Truth Pose in Frame AFeatures Relating the          Two FramesPose Detection
Feature WarpingFigure 1: A high level overview of our approach for using sparsely labeled videos for pose detection.
Faces in the ﬁgure are artiﬁcially masked for privacy reasons. In each training video, pose annotations
are available only every kframes. During training, our system considers a pair of frames–a labeled
Frame A, and an unlabeled Frame B, and aims to detect pose in Frame A, using the features from
Frame B. Our training procedure is designed to achieve two goals: 1) our model must be able to
extract motion offsets relating these two frames. 2) Using these motion offsets our model must
then be able to rewarp the detected pose heatmap extracted from an unlabeled Frame B in order to
optimize the accuracy of pose detection in a labeled Frame A. After training, we can apply our model
in reverse order to propagate pose information across the entire video from ground truth poses given
for only a few frames.
is allowed to compare Frame A to Frame B but it must predict Pose A (i.e., the pose in Frame A)
using the features from Frame B, as illustrated in Figure 1 (top).


key: chroma:document
string_value:
using the features from Frame B, as illustrated in Figure 1 (top).
At ﬁrst glance, this task may look overly challenging: how can we predict Pose A by merely using
features from Frame B? However, suppose that we had body joint correspondences between Frame A
and Frame B. In such a scenario, this task would become trivial, as we would simply need to spatially
“warp” the feature maps computed from frame B according to the set of correspondences relating
frame B to frame A. Based on this intuition, we design a learning scheme that achieves two goals: 1)
By comparing Frame A and Frame B, our model must be able to extract motion offsets relating these
two frames. 2) Using these motion offsets our model must be able to rewarp the pose extracted from
an unlabeled Frame B in order to optimize pose detection accuracy in a labeled Frame A.
To achieve these goals, we ﬁrst feed both frames through a backbone CNN that predicts pose heatmaps
for each of the frames. Then, the resulting heatmaps from both frames are used to determine which
points from Frame B should be sampled for detection in Frame A. Finally, the resampled pose
heatmap from Frame B is used to maximize accuracy of Pose A.


key: chroma:document
string_value:
points from Frame B should be sampled for detection in Frame A. Finally, the resampled pose
heatmap from Frame B is used to maximize accuracy of Pose A.
Backbone Network. Due to its high efﬁciency and accuracy, we use the state-of-the-art High
Resolution Network (HRNet-W48) [ 27] as our backbone CNN. However, we note that our system can
easily integrate other architectures as well. Thus, we envision that future improvements in still-image
pose estimation will further improve the effectiveness of our approach.
Deformable Warping. Initially, we feed Frame A and Frame B through our backbone CNN, which
outputs pose heatmaps fAandfB. Then, we compute the difference  A;B=fAfB. The resulting
feature tensor  A;Bis provided as input to a stack of 33simple residual blocks (as in standard
ResNet-18 or ResNet-34 models), which output a feature tensor A;B. The feature tensor A;Bis
then fed into ﬁve 33convolutional layers each using a different dilation rate d2f3;6;12;18;24g
3
Labeled Frame A (time t)
Offsets


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:
Pose      in Frame A A Stack of Residual            3x3 Blocks  Def. Conv. (dilation = 12)   2D Conv. (dilation = 12)   2D Conv. (dilation = 6)


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:
Def. Conv. (dilation = 18)   2D Conv. (dilation = 18)
   2D Conv. (dilation = 3)
Figure 2: An illustration of our PoseWarper architecture. Given a labeled Frame A and an unlabeled
Frame B, which are separated by steps in time, our goal is to detect pose in a labeled Frame A
using the features from an unlabeled Frame B. First, we predict pose heatmaps for both frames. Then,
we compute the difference between pose heatmaps in Frame A and Frame B and feed it through a
stack of 33residual blocks. Afterwards, we attach ﬁve 33convolutional layers with dilation
ratesd2f3;6;12;18;24gand predict ﬁve sets of offsets o(d)(pn)for each pixel location pn. The
predicted offsets are used to rewarp pose heatmap B. All ﬁve rewarped heatmaps are then summed
and the resulting tensor is used to predict pose in Frame A.
to predict ﬁve sets of offsets o(d)(pn)at all pixel locations pn. The motivation for using different
dilation rates at the offset prediction stage comes from the need to consider motion cues at different


key: chroma:document
string_value:
dilation rates at the offset prediction stage comes from the need to consider motion cues at different
spatial scales. When the body motion is small, a smaller dilation rate may be more useful as it
captures subtle motion cues. Conversely, if the body motion is large, using large dilation rate allows
us to incorporate relevant information further away. Next, the predicted offsets are used to spatially
rewarp the pose heatmap fB. We do this for each of the ﬁve sets of offsets o(d), and then sum up all
ﬁve rewarped pose heatmaps to obtain a ﬁnal output gA;B, which is used to predict pose in Frame A.
We implement the warping mechanism via a deformable convolution [ 28], which takes 1) the offsets
o(d)(pn), and 2) the pose heatmap fBas its inputs, and then outputs a newly sampled pose heatmap
gA;B. The subscript (A;B)is used to indicate that even though gA;Bwas resampled from tensor fB,
the offsets for rewarping were computed using  A;B, which contains information from both frames.
An illustration of our architecture is presented in Figure 2.
Loss Function. As in [ 27], we use a standard pose estimation loss function which computes a mean


key: chroma:document
string_value:
An illustration of our architecture is presented in Figure 2.
Loss Function. As in [ 27], we use a standard pose estimation loss function which computes a mean
squared error between the predicted, and the ground-truth heatmaps. The ground-truth heatmap is
generated by applying a 2D Gaussian around the location of each joint.
Pose Annotation Propagation. During training, we force our model to warp pose heatmap fBfrom
an unlabeled frame B such that it would match the ground-truth pose heatmap in a labeled Frame A.
Afterwards, we can reverse the application direction of our network. This then, allows us to propagate
pose information from manually annotated frames to unlabeled frames (i.e. from a labeled Frame
A to an unlabeled Frame B). Speciﬁcally, given a pose annotation in Frame A, we can generate its
respective ground-truth heatmap yAby applying a 2D Gaussian around the location of each joint
(identically to how it was done in [ 23,27]. Then, we can predict the offsets for warping ground-truth
heatmapyAto an unlabeled Frame B, from the feature difference  B;A=fBfA. Lastly, we


key: chroma:document
string_value:
heatmapyAto an unlabeled Frame B, from the feature difference  B;A=fBfA. Lastly, we
use our deformable warping scheme to warp the ground-truth pose heatmap yAto Frame B, thus,
propagating pose annotations to unlabeled frames in the same video. See Figure 1 (bottom) for a
high-level illustration of this scheme.
Temporal Pose Aggregation at Inference Time. Instead of using our model to propagate pose
annotations on training videos, we can also use our deformable warping mechanism to aggregate pose
information from nearby frames during inference in order to improve the accuracy of pose detection.
For every frame at time t, we want to aggregate information from all frames at times t+where
2f 3;2;1;0;1;2;3g. Such a pose aggregation procedure renders pose estimation more robust
to occlusions, motion blur, and video defocus.
4
   Ground Truth Reference             Frame (time t)PoseWarper (time t+1)FlowNet2 (time t+1)
Figure 3: The results of a video pose propagation task by our PoseWarper and FlowNet2 [ 29]. The
ﬁrst frame in each 3-frame sequence illustrates a labeled reference frame at time t. For simplicity,


key: chroma:document
string_value:
ﬁrst frame in each 3-frame sequence illustrates a labeled reference frame at time t. For simplicity,
we show only the “right ankle” body joint for one person, denoted by a pink circle in each of the
frames (please zoom in for a clearer view). The second frame depicts our propagated “right ankle”
detection from the labeled frame in time t to the unlabeled frame in time t+1. The third frame shows
the propagated detection in frame t+1 produced by the FlowNet2 baseline. In contrast to our method,
FlowNet2 fails to propagate poses when there is large motion, blurriness or occlusions.
Consider a pair of frames, ItandIt+. In this case, we want to use pose information from frame
It+to improve pose detection in frame It. To do this, we ﬁrst feed both frames through our trained
PoseWarper model, and obtain a spatially rewarped (resampled) pose heatmap gt;t+, which is aligned
with respect to frame Itusing the features from frame It+. We can repeat this procedure for every 
value, and then aggregate pose information from multiple frames via a summation asP
gt;t+.


key: chroma:document
string_value:
value, and then aggregate pose information from multiple frames via a summation asP
gt;t+.
Implementation Details. Following the framework in [ 27], for training, we crop a 384288
bounding box around each person and use it as input to our model. During training, we use ground
truth person bounding boxes. We also employ random rotations, scaling, and horizontal ﬂipping to
augment the data. To learn the network, we use the Adam optimizer [ 54] with a base learning rate
of104, which is reduced to 105and106after10, and 15epochs, respectively. The training is
performed using 4Tesla M40 GPUs, and is terminated after 20epochs. We initialize our model with
a HRNet-W48 [ 27] pretrained for a COCO keypoint estimation task. To train the deformable warping
module, we select Frame B, with a random time-gap 2[3;3]relative to Frame A. To compute
features relating the two frames, we use twenty 33residual blocks each with 128channels. Even
though this seems like many convolutional layers, due to a small number of channels in each layer,


key: chroma:document
string_value:
though this seems like many convolutional layers, due to a small number of channels in each layer,
this amounts to only 5:8M parameters (compared to 39M required to compute optical ﬂow in [ 29]).
To compute the offsets o(d), we use ﬁve 33convolutional layers, each using a different dilation
rate (d= 3;6;12;18;24). To resample the pose heatmap fB, we employ ﬁve 33deformable
convolutional layers, each applied to one of the ﬁve predicted offset maps o(d). The ﬁve deformable
convolution layers too employ different dilation rates of 3;6;12;18;24. During testing, we follow
the same two-stage framework used in [ 27,23]: ﬁrst, we detect the bounding boxes for each person in
the image using the detector in [ 48], and then feed the cropped images to our pose estimation model.
4 Experiments
In this section, we present our results on the PoseTrack [ 22] dataset. We demonstrate the effectiveness
of our approach on three applications: 1) video pose propagation, 2) training a network on annotations
augmented with propagated pose pseudo-labels, 3) temporal pose aggregation during inference.
5


key: chroma:document
string_value:
augmented with propagated pose pseudo-labels, 3) temporal pose aggregation during inference.
5
Table 1: The results of video pose propagation on the PoseTrack2017 [ 22] validation set (measured in
mAP). We propagate pose information across the entire video from the manual annotations provided
in few frames. To study the effect of different levels of dilated convolutions in our PoseWarper
architecture, we also include several ablation baselines (see the bottom half of the table).
Method Head Shoulder Elbow Wrist Hip Knee Ankle Mean
Pseudo-labeling w/ HRNet [27] 79.1 86.5 81.4 74.7 81.4 79.4 72.3 79.3
Optical Flow Propagation (Farneback [55]) 76.5 82.3 74.3 69.2 80.8 74.8 70.1 75.5
Optical Flow Propagation (FlowNet2 [29]) 82.7 91.0 83.8 78.4 89.7 83.6 78.1 83.8
PoseWarper (no dilated convs) 86.1 91.7 88.0 83.5 90.2 87.3 84.6 87.2


key: chroma:document
string_value:
PoseWarper (no dilated convs) 86.1 91.7 88.0 83.5 90.2 87.3 84.6 87.2
PoseWarper (1 dilated conv) 85.0 91.6 88.0 83.7 89.6 87.3 84.7 87.0
PoseWarper (2 dilated convs) 85.8 92.4 88.8 84.9 91.0 88.4 86.0 88.0
PoseWarper (3 dilated convs) 86.1 92.6 89.2 85.5 91.3 88.8 86.3 88.4
PoseWarper (4 dilated convs) 86.3 92.6 89.5 85.9 91.9 88.8 86.4 88.6
PoseWarper (5 dilated convs) 86.0 92.7 89.5 86.0 91.5 89.1 86.6 88.7
4.1 Video Pose Propagation
Quantitative Results. To verify that our model learns to capture pose correspondences, we apply it
to the task of video pose propagation, i.e., propagating poses across time from a few labeled frames.
Initially, we train our PoseWarper in a sparsely labeled video setting according to the procedure


key: chroma:document
string_value:
Initially, we train our PoseWarper in a sparsely labeled video setting according to the procedure
described above. In this setting, every 7thframe of a training video is labeled, i.e. there are 6
unlabeled frames between each pair of manually labeled frames. Since each video contains on
average 30 frames, we have approximately 5annotated frames uniformly spaced out in each video.
Our goal then, is to use our learned PoseWarper to propagate pose annotations from manually-labeled
frames to all unlabeled frames in the same video. Speciﬁcally, for each labeled frame in a video,
we propagate its pose information to the three preceding and three subsequent frames. We train our
PoseWarper on sparsely labeled videos from the training set of PoseTrack2017 [ 22] and then perform
our evaluations on the validation set.
To evaluate the effectiveness of our approach, we compare our model to several relevant baselines.
As our weakest baseline, we use our trained HRNet [ 27] model that simply predicts pose for every
single frame in a video. Furthermore, we also include a few propagation baselines based on warping
annotations using optical ﬂow. The ﬁrst of these uses a standard Farneback optical ﬂow [ 55] to
warp the manually-labeled pose in each labeled frame to its three preceding and three subsequent


key: chroma:document
string_value:
warp the manually-labeled pose in each labeled frame to its three preceding and three subsequent
frames. We also include a more advanced optical ﬂow propagation baseline that uses FlowNet2
optical ﬂow [29]. Finally, we evaluate our PoseWarper model.
In Table 1, we present our quantitative results for video pose propagation. The evaluation is done
using an mAP metric as in [ 42]. Our best model achieves a 88:7%mAP, while the optical ﬂow
propagation baseline using FlowNet2 [ 29] yields an accuracy of 83:8%mAP. We also note that
compared to the FlowNet2 [ 29] propagation baseline, our PoseWarper warping mechanism is not
only more accurate, but also signiﬁcantly more compact ( 6M vs 39M parameters).
Ablation Studies on Dilated Convolution. In Table 1, we also present the results investigating the
effect of different levels of dilated convolutions in our PoseWarper architecture. We evaluate all these
variants on the task of video pose propagation. First, we report that removing dilated convolution
blocks from the original architecture reduces the accuracy from 88:7mAP to 87:2mAP. We also
note that a network with a single dilated convolution (using a dilation rate of 3) yields 87:0mAP.


key: chroma:document
string_value:
note that a network with a single dilated convolution (using a dilation rate of 3) yields 87:0mAP.
Adding a second dilated convolution level (using dilation rates of 3;6) improves the accuracy to 88:0.
Three dilation levels (with dilation rates of 3;6;12) yield a mAP of 88:4and four levels (dilation
rates of 3;6;12;18) give a mAP of 88:6. A network with 5dilated convolution levels yields 88:7
mAP. Adding more dilated convolutions does not improve the performance further. Additionally, we
also experimented with two networks that use dilation rates of 1;2;3;4;5, and 4;8;16;24;32, and
report that such models yield mAPs of 88:6and88:5, respectively, which are slightly lower.
Qualitative Comparison to FlowNet2. In Figure 3, we include an illustration of the motion encoded
by PoseWarper, and compare it to the optical ﬂow computed by FlowNet2 for the video pose
propagation task. The ﬁrst frame in each 3-frame sequence illustrates a labeled reference frame at
time t. For a cleaner visualization, we show only the “right ankle” body joint for one person, which is


key: chroma:document
string_value:
time t. For a cleaner visualization, we show only the “right ankle” body joint for one person, which is
marked with a pink circle in each of the frames. The second frame depicts our propagated “right
6
a) Training HRNet with Propagated Pose Pseudo Labelsb) Temporal Pose Aggregation during Inference0 100 200 300# Labeled Training Videos707580Accuracy (mAP)GT (7x)GT (1x) + pGT (6x)GT (1x)02468# Labeled Frames Per Video707580Accuracy (mAP)GT (7x)GT (1x) + pGT (6x)GT (1x)02468# Labeled Frames Per Video707580Accuracy (mAP)GT (7x)GT (1x) + T-Agg.GT (1x)0 100 200 300# Labeled Training Videos707580Accuracy (mAP)GT (7x)GT (1x) + T-Agg.GT (1x)Figure 4: A ﬁgure illustrating the value of a) training a standard HRNet [ 27] using our propagated
pose pseudo labels (left), and b) our temporal pose aggregation scheme during inference. In both
settings, we study pose detection performance as a function of 1) number of sparsely-labeled training


key: chroma:document
string_value:
settings, we study pose detection performance as a function of 1) number of sparsely-labeled training
videos (with 1manually-labeled frame per video), and 2) number of labeled frames per video (with
50sparsely-labeled videos in total). All baselines are based on retraining the standard HRNet [ 27]
model on the different training sets. The "GT (1x)" baseline is trained in a standard way on sparsely
labeled video data. The "GT (7x)" baseline uses 7x more manually annotated data relative to the
"GT (1x)" baseline. Our approach on the left subﬁgure ("GT (1x) + pGT (6x)"), augments the
original sparsely labeled video data with our propagated pose pseudo labels ( 6nearby frames for
every manually-labeled frame). Lastly, in b) "GT (1x) + T-Agg" denotes the use of PoseWarper to
fuse pose information from multiple neighboring frames during inference (training is done as in "GT
(1x)" baseline). From the results, we observe that both application modalities of PoseWarper provide
an effective way to achieve strong pose accuracy while reducing the number of manual annotations.
ankle” detection from the labeled frame in time t to the unlabeled frame in time t+1. The third frame


key: chroma:document
string_value:
ankle” detection from the labeled frame in time t to the unlabeled frame in time t+1. The third frame
shows the propagated detection in frame t+1 produced by the FlowNet2 baseline. These results
suggest that FlowNet2 struggles to accurately warp poses if 1) there is large motion, 2) occlusions, or
3) blurriness. In contrast, our PoseWarper handles these cases robustly, which is also indicated by our
results in Table 1 (i.e., 88:7vs83:8mAP w.r.t. FlowNet2).
4.2 Data Augmentation with PoseWarper
Here we consider the task of propagating poses on sparsely labeled training videos using PoseWarper,
and then using them as pseudo-ground truth labels (in addition to the original manual labels) to
train a standard HRNet-W48 [ 27]. For this experiment, we study the pose detection accuracy as
a function of two variables: 1) the total number of sparsely-labeled videos, and 2) the number of
manually-annotated frames per video. We aim to study how much we can reduce manual labeling
through our mechanism of pose propagation, while maintaining strong pose accuracy. Note, that we
ﬁrst train our PoseWarper on sparsely labeled videos from the training set of PoseTrack2017 [ 22].


key: chroma:document
string_value:
ﬁrst train our PoseWarper on sparsely labeled videos from the training set of PoseTrack2017 [ 22].
Then, we propagate pose annotations on the same set of training videos. Afterwards, we retrain the
model on the joint training set comprised of sparse manual pose annotations and our propagated
poses. Lastly, we evaluate this trained model on the validation set.
All results are based on a standard HRNet [ 27] model trained on different forms of training data. "GT
(1x)" refers to a model trained on sparsely labeled videos using ground-truth annotations only. "GT
(7x)" baseline employs 7x more manually-annotated poses relative to "GT (1x)" (the annotations
are part of the PoseTrack2017 training set). In comparison, our approach ("GT (1x) + pGT (6x)"),
is trained on a joint training set consisting of sparse manual pose annotations (same as "GT (1x)"
baseline) and our propagated poses (on the training set of PoseTrack2017), which we use as pseudo
ground truth data (pGT). As before, for every labeled frame we propagate the ground truth pose to
the3previous and the 3subsequent frames, which allows us to expand the training set by 7times.


key: chroma:document
string_value:
the3previous and the 3subsequent frames, which allows us to expand the training set by 7times.
Based on the results in the left subﬁgure of Figure 4, we can draw several conclusions. First, we
note that when there are very few labeled videos (i.e., 5), all three baselines perform poorly (leftmost
ﬁgure). This indicates that in this setting there is not enough data to learn an effective pose detection
model. Second, we observe that when the number of labeled videos is somewhat reasonable (e.g.,
50100), our approach signiﬁcantly outperforms the "GT (1x)" baseline, and is only slightly worse
relative to the "GT (7x)" baseline. As we increase the number of labeled videos, the gaps among the
three methods shrink, suggesting that the model becomes saturated.
7
Table 2: Multi-person pose estimation results on the validation and test sets of PoseTrack2017
and PoseTrack2018 datasets. Even though our model is designed to improve pose detection in
scenarios involving sparsely-labeled videos, here we show that our temporal pose aggregation scheme
during inference is also useful for models trained on densely labeled videos. We improve upon the
state-of-the-art single-frame baselines [23, 27, 56].


key: chroma:document
string_value:
during inference is also useful for models trained on densely labeled videos. We improve upon the
state-of-the-art single-frame baselines [23, 27, 56].
Dataset Method Head Shoulder Elbow Wrist Hip Knee Ankle Mean
PoseTrack17 Val SetGirdhar et al. [48] 72.8 75.6 65.3 54.3 63.5 60.9 51.8 64.1
Xiu et al. [57] 66.7 73.3 68.3 61.1 67.5 67.0 61.3 66.5
Bin et al [23] 81.7 83.4 80.0 72.4 75.3 74.8 67.1 76.7
HRNet [27] 82.1 83.6 80.4 73.3 75.5 75.3 68.5 77.3
MDPN [56] 85.2 88.5 83.9 77.5 79.0 77.0 71.4 80.7
PoseWarper 81.4 88.3 83.9 78.0 82.4 80.5 73.6 81.2
PoseTrack17 Test SetGirdhar et al. [48] - - - - - - - 59.6


key: chroma:document
string_value:
PoseTrack17 Test SetGirdhar et al. [48] - - - - - - - 59.6
Xiu et al. [57] 64.9 67.5 65.0 59.0 62.5 62.8 57.9 63.0
Bin et al [23] 80.1 80.2 76.9 71.5 72.5 72.4 65.7 74.6
HRNet [27] 80.1 80.2 76.9 72.0 73.4 72.5 67.0 74.9
PoseWarper 79.5 84.3 80.1 75.8 77.6 76.8 70.8 77.9
PoseTrack18 Val SetAlphaPose [58] 63.9 78.7 77.4 71.0 73.7 73.0 69.7 71.9
MDPN [56] 75.4 81.2 79.0 74.1 72.4 73.0 69.9 75.0
PoseWarper 79.9 86.3 82.4 77.5 79.8 78.8 73.2 79.7
PoseTrack18 Test SetAlphaPose++ [56, 58] - - - 66.2 - - 65.0 67.6
MDPN [56] - - - 74.5 - - 69.0 76.4


key: chroma:document
string_value:
MDPN [56] - - - 74.5 - - 69.0 76.4
PoseWarper 78.9 84.4 80.9 76.8 75.6 77.5 71.8 78.0
As we vary the number of labeled frames per video (second leftmost ﬁgure), we notice several
interesting patterns. First, we note that for a small number of labeled frames per video (i.e., 12) our
approach outperforms the "GT (1x)" baseline by a large margin. Second, we note that the performance
of our approach and the "GT (7x)" becomes very similar as we add 2or more labeled frames per
video. These ﬁndings further strengthen our previous observation that PoseWarper allows us to reduce
the annotation cost without a signiﬁcant loss in performance.
4.3 Improved Pose Estimation via Temporal Pose Aggregation
In this subsection we assess the ability of PoseWarper to improve the accuracy of pose estimation at
test time by using our deformable warping mechanism to aggregate pose information from nearby
frames. We visualize our results in Figure 4 b), where we evaluate the effectiveness of our temporal
pose aggregation during inference for models trained a) with a different number of labeled videos


key: chroma:document
string_value:
frames. We visualize our results in Figure 4 b), where we evaluate the effectiveness of our temporal
pose aggregation during inference for models trained a) with a different number of labeled videos
(second rightmost ﬁgure), and b) with a different number of manually-labeled frames per video
(rightmost ﬁgure). We compare our approach ("GT (1x) + T-Agg.") to the same "GT (7x)" and "GT
(1x)" baselines deﬁned in the previous subsection. Note that our method in this case is trained exactly
as "GT (1x)" baseline, the only difference comes from the inference procedure.
When the number of training videos and/or manually labeled frames is small, our approach provides
a signiﬁcant accuracy boost with respect to the "GT (1x)" baseline. However, once, we increase the
number of labeled videos/frames, the gap between all three baselines shrinks, and the model becomes
more saturated. Thus, our temporal pose aggregation scheme during inference is another effective
way to maintain strong performance in a sparsely-labeled video setting.
4.4 Comparison to State-of-the-Art
We also test the effectiveness of our temporal pose aggregation scheme, when the model is trained on


key: chroma:document
string_value:
4.4 Comparison to State-of-the-Art
We also test the effectiveness of our temporal pose aggregation scheme, when the model is trained on
the full PoseTrack [ 22] dataset. Table 2 compares our method to the most recent approaches in this
area [ 48,57,23,27]. These results suggest that although we designed our method to improve pose
estimation when training videos are sparsely-labeled, our temporal pose aggregation scheme applied
at inference is also useful for models trained on densely-labeled videos. Our PoseWarper obtains
81:2mAP and 77:9mAP on PoseTrack2017 validation and test sets respectively, and 79:7mAP and
78:0mAP on PoseTrack2018 validation and test sets respectively, thus outperforming prior single
frame baselines [48, 57, 23, 27].
8
Frame tFrame t+5Farneback FlowPredicted Human         MotionOffset MagnitudesChannel 99 (x,y)Channel 123 (x,y)
Figure 5: In the ﬁrst two columns, we show a pair of video frames used as input for our model.
The3rdand4thcolumns depict 2randomly selected offset channels visualized as a motion ﬁeld.
Different channels appear to capture the motion of different body parts. In the 5thcolumn, we display


key: chroma:document
string_value:
Different channels appear to capture the motion of different body parts. In the 5thcolumn, we display
the offset magnitudes, which highlight salient human motion. Finally, the last two columns illustrate
the standard Farneback ﬂow, and the human motion predicted from our learned offsets. To predict
human motion we train a linear classiﬁer to regress the ground-truth (x;y)displacement of each
joint from the offset maps. The color wheel, at the bottom right corner encodes motion direction.
4.5 Interpreting Learned Offsets
Understanding what information is encoded in our learned offsets is nearly as difﬁcult as analyzing
any other CNN features [ 59,60]. The main challenge comes from the high dimensionality of offsets:
we are predicting ckhkw(x;y)displacements for every pixel for each of the ﬁve dilation
ratesd, wherecis the number of channels, and kh;kware the convolutional kernel height and width
respectively.
In columns 3;4of Figure 5, we visualize two randomly-selected offset channels as a motion ﬁeld.
Based on this ﬁgure, it appears that different offset maps encode different motions rather than all


key: chroma:document
string_value:
Based on this ﬁgure, it appears that different offset maps encode different motions rather than all
predicting the same solution (say, the optical ﬂow between the two frames). This makes sense, as the
network may decide to ignore motions of uninformative regions, and instead capture the motion of
different human body parts in different offset maps (say, a hand as opposed to the head). We also note
that the magnitudes of our learned offsets encode salient human motion (see Column 5 of Figure 5).
Lastly, to verify that our learned offsets encode human motion, for each point pndenoting a body joint,
we extract our predicted offsets and train a linear classiﬁer to regress the ground truth (x;y)motion
displacement of this body joint. In Column 7 of Figure 5, we visualize our predicted motion outputs
for every pixel. We show Farneback’s optical ﬂow in Column 6. Note that in regions containing
people, our predicted human motion matches Farneback optical ﬂow. Furthermore, we point out that
compared to the standard Farneback optical ﬂow, our motion ﬁelds look less noisy.
5 Conclusions
In this work, we introduced PoseWarper, a novel architecture for pose detection in sparsely labeled


key: chroma:document
string_value:
5 Conclusions
In this work, we introduced PoseWarper, a novel architecture for pose detection in sparsely labeled
videos. Our PoseWarper can be effectively used for multiple applications, including video pose
propagation, and temporal pose aggregation. In these settings, we demonstrated that our approach
reduces the need for densely labeled video data, while producing strong pose detection performance.
Furthermore, our state-of-the-art results on PoseTrack2017 and PoseTrack2018 datasets demonstrate
that our PoseWarper is useful even when the training videos are densely-labeled. Our future work
involves improving our model ability to propagate labels and aggregate temporal information when
the input frames are far away from each other. We are also interested in exploring self-supervised
learning objectives for our task, which may further reduce the need of pose annotations in video. We
will release our source code and our trained models upon publication of the paper.
References
[1]Gedas Bertasius, Jianbo Shi, and Lorenzo Torresani. Deepedge: A multi-scale bifurcated deep network
for top-down contour detection. In The IEEE Conference on Computer Vision and Pattern Recognition
(CVPR) , June 2015.
9


key: chroma:document
string_value:
for top-down contour detection. In The IEEE Conference on Computer Vision and Pattern Recognition
(CVPR) , June 2015.
9
[2]Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In CVPR , 2017.
[3]Alexander Toshev and Christian Szegedy. Deeppose: Human pose estimation via deep neural networks. In
CVPR , 2014.
[4]Gedas Bertasius, Lorenzo Torresani, Stella X. Yu, and Jianbo Shi. Convolutional random walk networks
for semantic image segmentation. In The IEEE Conference on Computer Vision and Pattern Recognition
(CVPR) , July 2017.
[5]Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Spatial pyramid pooling in deep convolutional
networks for visual recognition. In Computer Vision – ECCV 2014 , 2014.
[6]Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R-CNN. In Proceedings of the
International Conference on Computer Vision (ICCV) , 2017.


key: chroma:document
string_value:
International Conference on Computer Vision (ICCV) , 2017.
[7]Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal Loss for Dense Object
Detection. In Proceedings of the International Conference on Computer Vision (ICCV) , 2017.
[8]Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards real-time object
detection with region proposal networks. In Neural Information Processing Systems (NIPS) , 2015.
[9]Ross Girshick. Fast R-CNN. In Proceedings of the International Conference on Computer Vision (ICCV) ,
2015.
[10] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate
object detection and semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision
and Pattern Recognition (CVPR) , 2014.
[11] Saurabh Gupta, Ross Girshick, Pablo Arbelaez, and Jitendra Malik. Learning rich features from RGB-D
images for object detection and segmentation. In ECCV , 2014.
[12] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and


key: chroma:document
string_value:
[12] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and
Alexander C. Berg. Ssd: Single shot multibox detector. In ECCV , 2016.
[13] Jifeng Dai, Yi Li, Kaiming He, and Jian Sun. R-fcn: Object detection via region-based fully convolutional
networks. In Advances in Neural Information Processing Systems 29 , pages 379–387. Curran Associates,
Inc., 2016.
[14] Joseph Redmon, Santosh Kumar Divvala, Ross B. Girshick, and Ali Farhadi. You only look once: Uniﬁed,
real-time object detection. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR
2016, Las Vegas, NV , USA, June 27-30, 2016 , pages 779–788, 2016.
[15] Joseph Redmon and Ali Farhadi. YOLO9000: better, faster, stronger. In 2017 IEEE Conference on
Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017 , pages
6517–6525, 2017.


key: chroma:document
string_value:
Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017 , pages
6517–6525, 2017.
[16] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classiﬁcation with deep convolutional
neural networks. In NIPS , 2012.
[17] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru
Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Computer Vision
and Pattern Recognition (CVPR) , 2015.
[18] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. In
ICLR , 2015.
[19] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition.
2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 770–778, 2016.
[20] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image


key: chroma:document
string_value:
Database. In CVPR09 , 2009.
[21] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár,
and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference on
Computer Vision (ECCV) , Zürich, September 2014.
[22] Umar Iqbal, Anton Milan, and Juergen Gall. Posetrack: Joint multi-person pose estimation and tracking.
InIEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2017.
10
[23] Bin Xiao, Haiping Wu, and Yichen Wei. Simple baselines for human pose estimation and tracking. In
European Conference on Computer Vision (ECCV) , 2018.
[24] Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. Realtime multi-person 2d pose estimation using
part afﬁnity ﬁelds. In CVPR , 2017.
[25] Alejandro Newell, Kaiyu Yang, and Jia Deng. Stacked hourglass networks for human pose estimation. In
Computer Vision - ECCV 2016 - 14th European Conference, Amsterdam, The Netherlands, October 11-14,


key: chroma:document
string_value:
Computer Vision - ECCV 2016 - 14th European Conference, Amsterdam, The Netherlands, October 11-14,
2016, Proceedings, Part VIII , pages 483–499, 2016.
[26] Shih-En Wei, Varun Ramakrishna, Takeo Kanade, and Yaser Sheikh. Convolutional pose machines. In
CVPR , pages 4724–4732. IEEE Computer Society, 2016.
[27] Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang. Deep high-resolution representation learning for human
pose estimation. In CVPR , 2019.
[28] J. Dai, H. Qi, Y . Xiong, Y . Li, G. Zhang, H. Hu, and Y . Wei. Deformable convolutional networks. In 2017
IEEE International Conference on Computer Vision (ICCV) , volume 00, pages 764–773, Oct. 2017.
[29] Eddy Ilg, Nikolaus Mayer, Tonmoy Saikia, Margret Keuper, Alexey Dosovitskiy, and Thomas Brox.
Flownet 2.0: Evolution of optical ﬂow estimation with deep networks. CoRR , abs/1612.01925, 2016.
[30] Mykhaylo Andriluka, Stefan Roth, and Bernt Schiele. Pictorial structures revisited: People detection and


key: chroma:document
string_value:
[30] Mykhaylo Andriluka, Stefan Roth, and Bernt Schiele. Pictorial structures revisited: People detection and
articulated pose estimation. In CVPR , pages 1014–1021. IEEE Computer Society, 2009.
[31] M. Andriluka, S. Roth, and B. Schiele. Monocular 3d pose estimation and tracking by detection. In 2010
IEEE Computer Society Conference on Computer Vision and Pattern Recognition , pages 623–630, June
2010.
[32] Sam Johnson and Mark Everingham. Clustered pose and nonlinear appearance models for human pose
estimation. In Proc. BMVC , pages 12.1–11, 2010. doi:10.5244/C.24.12.
[33] Leonid Pishchulin, Mykhaylo Andriluka, Peter V . Gehler, and Bernt Schiele. Strong appearance and
expressive spatial models for human pose estimation. In ICCV , pages 3487–3494. IEEE Computer Society,
2013.
[34] Yi Yang and D. Ramanan. Articulated pose estimation with ﬂexible mixtures-of-parts. In Proceedings of
the 2011 IEEE Conference on Computer Vision and Pattern Recognition , CVPR ’11, pages 1385–1392,


key: chroma:document
string_value:
the 2011 IEEE Conference on Computer Vision and Pattern Recognition , CVPR ’11, pages 1385–1392,
Washington, DC, USA, 2011. IEEE Computer Society.
[35] M. Dantone, J. Gall, C. Leistner, and L. van Gool. Human pose estimation using body parts dependent
joint regressors. In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR) , pages 3041–3048,
Portland, OR, USA, June 2013. IEEE.
[36] Xiangyang Lan and D. P. Huttenlocher. Beyond trees: common-factor models for 2d human pose recovery.
InTenth IEEE International Conference on Computer Vision (ICCV’05) Volume 1 , volume 1, pages
470–477 V ol. 1, Oct 2005.
[37] L. Sigal and M. J. Black. Measure locally, reason globally: Occlusion-sensitive articulated pose estimation.
InProc. IEEE Conf. on Computer Vision and Pattern Recognition, CVPR , volume 2, pages 2041–2048,
New York, NY , June 2006.
[38] Yang Wang and Greg Mori. Multiple tree models for occlusion and spatial constraints in human pose
estimation. In David Forsyth, Philip Torr, and Andrew Zisserman, editors, Computer Vision – ECCV 2008 ,


key: chroma:document
string_value:
estimation. In David Forsyth, Philip Torr, and Andrew Zisserman, editors, Computer Vision – ECCV 2008 ,
pages 710–724, Berlin, Heidelberg, 2008. Springer Berlin Heidelberg.
[39] J. Carreira, P. Agrawal, K. Fragkiadaki, and J. Malik. Human pose estimation with iterative error feedback.
In2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 4733–4742, June
2016.
[40] Xianjie Chen and Alan L Yuille. Articulated pose estimation by a graphical model with image dependent
pairwise relations. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger,
editors, Advances in Neural Information Processing Systems 27 , pages 1736–1744. Curran Associates,
Inc., 2014.
[41] W. Ouyang, X. Chu, and X. Wang. Multi-source deep learning for human pose estimation. In 2014 IEEE
Conference on Computer Vision and Pattern Recognition , pages 2337–2344, June 2014.
11


key: chroma:document
string_value:
Conference on Computer Vision and Pattern Recognition , pages 2337–2344, June 2014.
11
[42] Eldar Insafutdinov, Leonid Pishchulin, Bjoern Andres, Mykhaylo Andriluka, and Bernt Schieke. Deepercut:
A deeper, stronger, and faster multi-person pose estimation model. In European Conference on Computer
Vision (ECCV) , 2016.
[43] Leonid Pishchulin, Eldar Insafutdinov, Siyu Tang, Bjoern Andres, Mykhaylo Andriluka, Peter Gehler, and
Bernt Schiele. Deepcut: Joint subset partition and labeling for multi person pose estimation. In IEEE
Conference on Computer Vision and Pattern Recognition (CVPR) , 2016.
[44] Jonathan J Tompson, Arjun Jain, Yann LeCun, and Christoph Bregler. Joint training of a convolutional
network and a graphical model for human pose estimation. In Z. Ghahramani, M. Welling, C. Cortes, N. D.
Lawrence, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 27 , pages
1799–1807. Curran Associates, Inc., 2014.


key: chroma:document
string_value:
Lawrence, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 27 , pages
1799–1807. Curran Associates, Inc., 2014.
[45] George Papandreou, Tyler Zhu, Nori Kanazawa, Alexander Toshev, Jonathan Tompson, Chris Bregler, and
Kevin Murphy. Towards accurate multi-person pose estimation in the wild. In CVPR , pages 3711–3719.
IEEE Computer Society, 2017.
[46] Vasileios Belagiannis and Andrew Zisserman. Recurrent human pose estimation. In International
Conference on Automatic Face and Gesture Recognition . IEEE, 2017.
[47] Eldar Insafutdinov, Mykhaylo Andriluka, Leonid Pishchulin, Siyu Tang, Evgeny Levinkov, Bjoern Andres,
and Bernt Schiele. Articulated multi-person tracking in the wild. In 2017 IEEE Conference on Computer
Vision and Pattern Recognition (CVPR) , pages 1293–1301. IEEE, July 2017. Oral.
[48] Rohit Girdhar, Georgia Gkioxari, Lorenzo Torresani, Manohar Paluri, and Du Tran. Detect-and-Track:
Efﬁcient Pose Estimation in Videos. In CVPR , 2018.


key: chroma:document
string_value:
Efﬁcient Pose Estimation in Videos. In CVPR , 2018.
[49] Jie Song, Limin Wang, Luc Van Gool, and Otmar Hilliges. Thin-slicing network: A deep structured model
for pose estimation in videos. In CVPR , pages 4420–4229, 2017.
[50] J. Charles, T. Pﬁster, D. Magee, D. Hogg, and A. Zisserman. Personalizing human video pose estimation.
InIEEE Conference on Computer Vision and Pattern Recognition , 2016.
[51] T. Pﬁster, J. Charles, and A. Zisserman. Flowing convnets for human pose estimation in videos. In IEEE
International Conference on Computer Vision , 2015.
[52] Dingwen Zhang, Guangyu Guo, Dong Huang, and Junwei Han. Poseﬂow: A deep motion representation
for understanding human behaviors in videos. In The IEEE Conference on Computer Vision and Pattern
Recognition (CVPR) , June 2018.
[53] O. Wiles, A.S. Koepke, and A. Zisserman. Self-supervised learning of a facial attribute embedding from
video. In British Machine Vision Conference , 2018.


key: chroma:document
string_value:
video. In British Machine Vision Conference , 2018.
[54] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR , 2015.
[55] Gunnar Farnebäck. Two-frame motion estimation based on polynomial expansion. In Proceedings of the
13th Scandinavian Conference on Image Analysis , SCIA’03, pages 363–370, Berlin, Heidelberg, 2003.
Springer-Verlag.
[56] Hengkai Guo, Tang Tang, Guozhong Luo, Riwei Chen, Yongchen Lu, and Linfu Wen. Multi-domain pose
network for multi-person pose estimation and tracking. In Computer Vision - ECCV 2018 Workshops Munich, Germany, September 8-14, 2018, Proceedings, Part II , pages 209–216, 2018.
[57] Yuliang Xiu, Jiefeng Li, Haoyu Wang, Yinghong Fang, and Cewu Lu. Pose Flow: Efﬁcient online pose
tracking. In BMVC , 2018.
[58] Hao-Shu Fang, Shuqin Xie, Yu-Wing Tai, and Cewu Lu. Rmpe: Regional multi-person pose estimation. In
The IEEE International Conference on Computer Vision (ICCV) , Oct 2017.


key: chroma:document
string_value:
The IEEE International Conference on Computer Vision (ICCV) , Oct 2017.
[59] Matthew D. Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. CoRR ,
abs/1311.2901, 2013.
[60] Jason Yosinski, Jeff Clune, Anh Mai Nguyen, Thomas J. Fuchs, and Hod Lipson. Understanding neural
networks through deep visualization. CoRR , abs/1506.06579, 2015.
12


key: chroma:document
string_value:
Classifying, Segmenting, and Tracking
Object Instances in Video with Mask Propagation
Gedas Bertasius, Lorenzo Torresani
Facebook AI
Abstract
We introduce a method for simultaneously classifying,
segmenting and tracking object instances in a video sequence. Our method, named MaskProp, adapts the popular Mask R-CNN to video by adding a mask propagation
branch that propagates frame-level object instance masks
from each video frame to all the other frames in a video clip.
This allows our system to predict clip-level instance tracks
with respect to the object instances segmented in the middle frame of the clip. Clip-level instance tracks generated
densely for each frame in the sequence are ﬁnally aggregated to produce video-level object instance segmentation
and classiﬁcation. Our experiments demonstrate that our
clip-level instance segmentation makes our approach robust
to motion blur and object occlusions in video. MaskProp
achieves the best reported accuracy on the YouTube-VIS
dataset, outperforming the ICCV 2019 video instance segmentation challenge winner despite being much simpler and
using orders of magnitude less labeled data (1.3M vs 1B images and 860K vs 14M bounding boxes).
1. Introduction


key: chroma:document
string_value:
using orders of magnitude less labeled data (1.3M vs 1B images and 860K vs 14M bounding boxes).
1. Introduction
In this paper, we tackle the recently introduced video instance segmentation problem [42]. This task requires segmenting all instances of a predeﬁned set of object classes
in each frame, classifying them, and linking individual instances over the entire sequence.
In recent years, convolutional networks have obtained remarkable results in still-image object detection [16, 33, 14,
15], and segmentation [27, 45, 8, 4]. However, extending
these models to video instance segmentation is challenging. In order to localize objects precisely, these methods
have to operate at very large spatial resolution. As a result,
detectors based on the popular ResNet-101 or ResNet-152
backbones [17] can rarely ﬁt more than one image per GPU
during training. In the context of video instance segmentation this is problematic because tracking objects over time
requires analyzing multiple video frames simultaneously.
To address this issue, one could reduce the spatial resFigure 1: In this paper, we tackle the problem of video instance segmentation, which requires classifying, segmenting, and tracking object instances in a given video sequence.
Our proposed Mask Propagation framework (MaskProp)
provides a simple and effective way for solving this task.


key: chroma:document
string_value:
Our proposed Mask Propagation framework (MaskProp)
provides a simple and effective way for solving this task.
olution of the input and ﬁt more video frames in a GPU.
However, doing so typically leads to a signiﬁcant drop in
segmentation or detection performance. Alternatively, one
could perform high-resolution instance segmentation on individual frames and then link segmentations temporally in
a separate post-processing stage. However, performing instance segmentation and tracking in two disjoint steps often produces suboptimal results, because these two tasks
are closely intertwined. The key challenge then becomes
designing a uniﬁed model that can track objects in video
while maintaining strong detection accuracy.
Currently, the best method for video instance segmentation is the ICCV 2019 challenge winner [28]. It tackles
video instance segmentation by dividing it into four problems: 1) detection, 2) classiﬁcation, 3) segmentation, and
4) tracking. These four problems are solved independently
using several off-the-shelf components and their respective
solutions are combined and adapted to the video instance
segmentation task. However, despite effective performance,
such an approach is disadvantageous because it requires designing and tuning a separate model (or, in some cases, an


key: chroma:document
string_value:
segmentation task. However, despite effective performance,
such an approach is disadvantageous because it requires designing and tuning a separate model (or, in some cases, an
ensemble of models) for each of the four tasks. This renders the approach costly and cumbersome. On the other
end of the complexity spectrum, MaskTrack R-CNN [42] is
a simple uniﬁed approach trained end-to-end but it achieves
signiﬁcantly lower performance ( 30:3vs44:8video mAP).
To address the shortcomings of these prior methods wearXiv:1912.04573v4  [cs.CV]  9 Jul 2021
MaskTrack R-CNN [42] ICCV19 Challenge Winner [28] MaskProp
ModelClassiﬁcation cls head Mask R-CNN [16], ResNeXt-101 32x48d [30] cls head
Localization bbox head Mask R-CNN [16] bbox head
Segmentation mask head DeepLabv3 [9], Box2Seg [29] mask head
Tracking tracking head UnOVOST [47], ReID Net [18, 31] mask propagation head
Optical Flow - PWC-Net [35] Pre-training DatasetsImageNet [34] (1.3M images) X X X


key: chroma:document
string_value:
Optical Flow - PWC-Net [35] Pre-training DatasetsImageNet [34] (1.3M images) X X X
COCO [25] (860K bboxes) X X X
Instagram [30] (1B images) - X OpenImages [23] (14M bboxes) - X Performancevideo mAP 30.3 44.8 46.6
video AP@75 32.6 48.9 51.2
Table 1: A table comparing our work to prior video instance segmentation methods [42, 28]. The ICCV 2019 Challenge
Winner [28] decomposes video instance segmentation into four different problems, solves each of them independently using
ensembles of different models, and then combines these solutions. In contrast, our approach relies on a single uniﬁed model
trained end-to-end. Despite being simpler, and using several orders of magnitude less pretraining data (1.3M vs 1B images
and 860K vs 14M bounding boxes) our model achieves higher accuracy. Furthermore, compared to MaskTrack R-CNN [42],
our work yields a 16.3% gain in mAP (46.6% vs 30.3%).
introduce MaskProp, a simple mask propagation framework
for simultaneously classifying, segmenting and tracking object instances in video. Our method adapts the popular


key: chroma:document
string_value:
introduce MaskProp, a simple mask propagation framework
for simultaneously classifying, segmenting and tracking object instances in video. Our method adapts the popular
Mask R-CNN [16] to video by adding a branch that propagates frame-level instance masks from each video frame
to other frames within a temporal neighborhood (which we
refer to as a clip). This allows our method to compute cliplevel instance tracks centered at each individual frame of the
video. These densely estimated clip-level tracks are then
aggregated to form accurate and coherent object instance
sequences for the entire video, regardless of its length. This
renders our approach capable of handling challenging cases
of occlusions, disocclusions, and motion blur. Our method
achieves the best reported accuracy on the YouTube-VIS
dataset [42], outperforming the ICCV 2019 challenge winner [28] despite being much simpler and using signiﬁcantly
less labeled data (1000x fewer images and 10x fewer bounding boxes). In Table 1, we compare our approach vs these
prior methods in terms of accuracy and other characteristics.
2. Related Work
Instance Segmentation in Images. Compared to instance
segmentation in images [11, 16, 44, 1, 2, 22, 26], the problem considered in this paper requires not only to segment
object instances in individual frames, but also to determine


key: chroma:document
string_value:
object instances in individual frames, but also to determine
instance correspondences across multiple frames. We leverage the Mask R-CNN model [16] for still-image instance
segmentation and adapt it to track object instances in video.
Object Detection in Video. Object detection in video requires classifying and localizing objects in every frame of
a given video. Most modern video object detection systems [46, 5, 40, 13] implement some form of spatiotemporal feature alignment for improving object detection accuracy in individual video frames. However, these systems
are typically not designed for tracking object instances. In
contrast, our mask propagation produces clip-level instancesegmentations rather than frame-level bounding boxes.
Video Object Segmentation. The task of video object
segmentation requires segmenting foreground objects in a
class-agnostic fashion [43, 21, 36, 38], often by leveraging
ground truth masks available for the ﬁrst frame during inference [6, 32, 10, 19, 37]. Instead, video instance segmentation requires ﬁnding all instances of a predeﬁned set of
object classes in each frame, classifying them and linking
them over the entire sequence.
Video Instance Segmentation. The recently introduced
video instance segmentation task [42] requires classifying,
segmenting and tracking object instances in videos. This


key: chroma:document
string_value:
Video Instance Segmentation. The recently introduced
video instance segmentation task [42] requires classifying,
segmenting and tracking object instances in videos. This
is the task considered in this work. There are only a few
video instance segmentation methods we can compare our
approach to. The MaskTrack R-CNN [42] presents a uniﬁed model for video instance segmentation. It augments the
original Mask R-CNN [16] with a tracking branch that establishes associations among object instances segmented in
separate frames. Furthermore, we include the ICCV 2019
video instance segmentation challenge winner [28] in our
comparison. This approach divides video instance segmentation into four separate subproblems: classiﬁcation, detection, segmentation, and tracking. A separate model (or
an ensemble of models) is used to solve each of these subproblems, and these solutions are then combined to produce
video instance segmentation results. For brevity, from now
on we refer to it as EnsembleVIS to indicate that it is an ensemble approach designed for video instance segmentation.
Our MaskProp framework provides advantages over
both of these methods [42, 28]. Similarly to MaskTrack
R-CNN [42], our method is a uniﬁed and simple approach.


key: chroma:document
string_value:
both of these methods [42, 28]. Similarly to MaskTrack
R-CNN [42], our method is a uniﬁed and simple approach.
However, our mask propagation branch is much more effective than the tracking branch of MaskTrack R-CNN, achieving much higher accuracy relative to this baseline. Furthermore, compared to EnsembleVIS [28], our method 1)
is much simpler, 2) uses signiﬁcantly less labeled data, and


key: chroma:document
string_value:
clip-level     instance tracktemporally propagated instance featuresfeature tensorinstance feature computation at time t    frame-level    instance mask time ttime t⌦<latexit


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:
time ttime t+10time t-5time t+5time t+10time ttime t-10propagated instance        segmentationfeature tensors for each frame             in a video clip
time t-5time t+5time t+10time ttime t-10
Figure 2: An illustration of our MaskProp system, which takes as input a video clip centered around frame t, and outputs
a clip-level instance track. Our mask propagation framework can be summarized in three high-level steps: 1) An instancespeciﬁc feature map at time tis computed by masking the frame features at time twith the given instance segmentation
for framet(one for each instance detected in frame t). 2) Next, we use our mask propagation mechanism to temporally
propagate instance-speciﬁc features from frame tto all the other frames in the clip. 3) Lastly, our model predicts instancespeciﬁc segmentations in every frame of the clip by implicitly matching the propagated instance features with the frame-level
features computed at each time step. This last step yields clip-level instance tracks centered around frame t.
3) produces higher accuracy on YouTube-VIS [42].
3. Video Instance Segmentation
Problem Deﬁnition. Let us denote with V2RL3HW


key: chroma:document
string_value:
3. Video Instance Segmentation
Problem Deﬁnition. Let us denote with V2RL3HW
an input video consisting of LRGB frames of spatial size
HW. The aim of our system is to segment and temporally
link all object instances that are visible for at least one frame
inVand that belong to a predeﬁned set of categories C=
f1;:::;Kg. To achieve this goal, our model outputs a videolevel instance mask track Mi2RLHWwith a category
labelci2f1;:::;Kgand a conﬁdence score si2[0;1]for
each object instance idetected in the video.
Evaluation Metric. Video instance segmentation is evaluated according to the metrics of average precision (AP) and
average recall (AR). Unlike in the image domain, these metrics are evaluated over the video sequence. Thus, to evaluate spatiotemporal consistency of the predicted mask sequences, the video Intersection over Union (IoU) between
a predicted object instance iand a ground truth object instancejis computed as:
IoU(i;j) =PT
t=1jMi(t)\~Mj(t)j
PT


key: chroma:document
string_value:
IoU(i;j) =PT
t=1jMi(t)\~Mj(t)j
PT
t=1jMi(t)[~Mj(t)j(1)
where ~Mj(t)is the ground-truth segmentation of object j
in framet. To achieve a large IoU, a model must not only
accurately classify and segment object instances at a framelevel, but also reliably track them over the video sequence.
As in the COCO benchmark for image segmentation [25], the metrics of AP and AR are computed separatelyfor each object category, and then averaged over ten IoU
thresholds from 50% to95% at increments of 5%. Lastly,
the resulting AP and AR metrics are averaged over the category set, which yields the ﬁnal evaluation metric.
4. Mask Propagation
MaskProp takes a video Vof arbitrary length Las input and outputs video-level instance segmentation tracks
Mi, category labels ciand conﬁdence scores sifor all
objectsidetected in the video. In order to achieve this
goal, our method ﬁrst builds clip-level object instance tracks
Mi
tT:t+T2 R(2T+1)1HWfor each individual clip


key: chroma:document
string_value:
Mi
tT:t+T2 R(2T+1)1HWfor each individual clip
VtT:t+T2R(2T+1)3HWof length (2T+ 1) in the
video, i.e., for t= 1;2;:::;L (clips at the beginning and
the end of the video will include fewer frames).
We want to use clips that are long enough to allow us
to jointly solve instance segmentation and tracking while
handling challenging cases of occlusion and motion blur.
At the same time, the clip should be short enough to allow
us to ﬁt it at high spatial resolution in the memory of a GPU.
The resulting clip-level instance masks Mi
tT:t+Tproduced densely for all overlapping clips t= 1;:::;L are
then aggregated to produce video-level instance masks Mi.
Our approach for clip-level instance segmentation is described in subsections 4.1 and 4.2. We also illustrate it in
Figure 2. The subsequent clip-level instance mask aggregation method is presented in subsection 4.3.
Frame t
Bbox...
ClsMask
Frame t+...     Mask  Propagation     CNN


key: chroma:document
string_value:


key: chroma:document
string_value:
Figure 3: We adapt Mask R-CNN [16] to video by adding
a mask propagation branch, for tracking object segmentation instances in video. Given a video clip centered around
framet, our system outputs a clip-level instance segmentation track as well as a classiﬁcation score and a bounding
box for each object instanced detected in frame t. For compactness, in this ﬁgure, we illustrate our system processing a
pair of frames but typically the propagation is applied from
the middle frame to all the other frames in the clip.
4.1. Video Mask R-CNN
Our video instance segmentation system is based on the
Mask R-CNN [16] model, which we adapt to video by
adding a mask propagation branch (See Figure 3). We train
our system with a multi-task loss Lt=Lcls
t+Lbox
t+
Lmask
t +Lprop
t wheretdenotes a time-step of a center
frame. We use identical loss terms Lcls
t;Lbox
t,Lmask
t as
in Mask R-CNN. The mask propagation loss is deﬁned as:
Lprop
t =~NtX
it+TX
t0=tT1sIoU (Mi
tT:t+T(t0);~Mi


key: chroma:document
string_value:
t =~NtX
it+TX
t0=tT1sIoU (Mi
tT:t+T(t0);~Mi
tT:t+T(t0))(2)
whereMi
tT:t+T(t0)2[0;1]is the segmentation at time
t0for an instance ipredicted from a clip centered at tand
~Mi
tT:t+T(t0)is the corresponding ground truth mask at
timet0.~Ntis the number of ground truth object instances
in framet, andsIoU is deﬁned as:
sIoU (A;B) =P
pA(p)B(p)P
pA(p) +B(p)A(p)B(p)(3)
where the summations in numerator and denominator are
performed over every pixel location p. The loss above is a
soft IoU loss, which we observed to work slightly better
than the standard cross entropy loss for our task.
4.2. Mask Propagation Branch
Overview. Our main technical contribution is the design of
a mask propagation branch, that allows our method to track


key: chroma:document
string_value:
4.2. Mask Propagation Branch
Overview. Our main technical contribution is the design of
a mask propagation branch, that allows our method to track
object instances. Given a video clip VtT:t+Tcenteredat framet, our system outputs clip-level instance masks
Mi
tT:t+Tfor each predicted object instance iin framet.
Our mask propagation branch can be described in three
high-level steps: 1) instance-speciﬁc feature computation,
2) temporal propagation of instance features, and 3) propagated instance segmentation. We will now describe each of
these steps in more detail. We introduce our mask propagation with the example of propagating object instance masks
from frametto framet+where2[T:T].
Computing Instance Speciﬁc Features. The mask
branch of our model predicts frame-level instance masks
Mi
t2R1H0W0from single frame inputs. We then
use these frame-level instance masks to compute instancespeciﬁc features for frame t. Speciﬁcally, for each object instancei, we compute an element-wise product between Mi
t
and the feature tensor from the backbone network ft. This
then yields a set of new feature tensors fi


key: chroma:document
string_value:
t
and the feature tensor from the backbone network ft. This
then yields a set of new feature tensors fi
t2RCH0W0,
wherei= 1;::;Nt, andNtis the number of object instances
detected in frame t. In other words, for each object instance
i, we are zeroing out the feature values in ftthat correspond
to pixels not belonging to that object instance.
Temporally Propagating Instance Features. Given
frame-level features ft;ft+2RCH0W0and instancespeciﬁc feature tensor fi
t, our method generates a propagated instance feature tensor gi
t;t+. Intuitively, gi
t;t+represents the features predicted by our model for object instanceiin framet+from an instance-speciﬁc feature tensorfi
t. The tensor gi
t;t+is generated by warping features
fi
tusing the alignment computed from frame-level features
ftandft+. We implement the propagation mechanism via
a deformable convolution [12], which has previously been
used for aligning features computed from separate frames


key: chroma:document
string_value:
ftandft+. We implement the propagation mechanism via
a deformable convolution [12], which has previously been
used for aligning features computed from separate frames
of a video [5, 3]. Speciﬁcally, we compute the elementwise difference of tensors ft;ft+and feed it through a
simple residual block [17], which predicts motion offsets
ot;t+2R2k2H0W0. These offsets contain (x;y)sampling locations for each entry of a kkdeformable convolution kernel [12]. The propagation step takes as inputs 1)
the offsetsot;t+and 2) the instance feature tensor fi
t, and
then applies deformable convolution to output the propagated instance feature tensor gi
t;t+for each instance i. We
use subscript t;t+to denote the propagated instance feature because, although gis obtained by propagating the feature tensorfi
t, the offset computation uses both frame tand
framet+. We stress that no explicit ground truth alignment is available between frames. The deformable convolutional kernels are supervised implicitly by optimizing Eq. 2.
Segmenting Propagated Instances. Lastly, we use our
propagated feature map gi


key: chroma:document
string_value:
Segmenting Propagated Instances. Lastly, we use our
propagated feature map gi
t;t+for predicting a corresponding object instance mask in frame t+. To do this we
ﬁrst, construct a new feature tensor i
t;t+=gi
t;t++ft+.
The addition effectively overimposes the tensor gi
t;t+predicted from time tfor object instance iin framet+, with
Tensor t


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:


key: chroma:document
string_value:
4: An illustration of the 3 steps implemented by our mask propagation branch: 1)For every detected instance in frame


key: chroma:document
string_value:
t, we compute an instance-speciﬁc feature tensor via element-wise multiplication between tensor tand the given frame-level
instance mask at frame t.2)Next, the element-wise difference of the feature tensors associated with frames tandt+is
used to predict motion offsets between frames tandt+. The resulting offsets are used to propagate the instance-speciﬁc
tensors from time tto timet+via deformable convolution. The propagated tensors represent instance-speciﬁc features
predicted for time t+using the tensors computed at time t.3)Lastly, we add the propagated instance feature tensors to the
tensor effectively computed at t+. A convolutional layer applied to these tensors predicts instance masks in frame t+.
The pixels that do not belong to any object instances are zeroed out using an instance-agnostic attention map.
the tensorft+effectively computed from the frame at time
t+. If the object instance prediction is consistent with the
feature computation, the feature tensors will be aligned and
thus, they will reinforce each other in the predicted region.
Finally, the resulting feature tensors i
t;t+are fed into
a11convolution layer that outputs instance masks for


key: chroma:document
string_value:
Finally, the resulting feature tensors i
t;t+are fed into
a11convolution layer that outputs instance masks for
each object instance iin framet+. The masks are normalized with the softmax nonlinearity across all Ntinstances. To zero-out pixels that do not belong to any object
instance, we use a single 33convolution that computes
an instance-agnostic attention map At+from feature tensorft+. We then multiply At+with each of our predicted
instance masks. A detailed illustration of our mask propagation branch is presented in Figure 4.
4.3. Video-Level Segmentation Instances
Given a video of length L, our goal is to produce videolevel segmentation instances Mi2 RLHW. Conceptually, this requires linking clip-level instance tracks
Mi
tT:t+TandMj
t0T:t0+Twheniandjrepresent the same
object instance, i.e., when the instances are matching. We
achieve this by assigning a video-level instance ID to each
of our predicted clip-level instance tracks. Matching instance tracks are assigned the same video-level instance ID.
Matching Clip-Level Instance Tracks. Consider a pair of
clip-level instance tracks Mi


key: chroma:document
string_value:
Matching Clip-Level Instance Tracks. Consider a pair of
clip-level instance tracks Mi
tT:t+TandMj
t0T:t0+T, that
are centered around frames tandt0respectively. They overlap in time ifjtt0j<2T+ 1. Let us denote their overlapping time interval as \t;t0. Given two overlapping in-stance tracks, we can check if they match by comparing
their predicted instance masks in the overlapping frames.
If the masks of the two instance tracks align well, then they
are likely to encode the same object instance. Otherwise,
each track represents a different object instance. We compute a matching score mi;j
t;t0between two clip-level instance
tracks using our previously deﬁned soft IoU metric as:
mi;j
t;t0=1
j\t;t0jX
~t2\t;t0sIoU (Mi
tT:t+T(~t); Mj
t0T:t0+T(~t))(4)
Video-Level Instance ID Assignment. We denote withY
the set of video-level instance IDs. The set Yis built incrementally by matching clip-level instance tracks Mi
tT:t+T


key: chroma:document
string_value:
the set of video-level instance IDs. The set Yis built incrementally by matching clip-level instance tracks Mi
tT:t+T
in order from time t= 1 to timet=L. Initially, we set
Y=f1;2;:::;N 1gwhereN1is the number of object instances detected at time t= 1 (i.e., in the ﬁrst clip of the
video). Let yi
t2Y denote the video-level instance ID assigned to clip-level instance track Mi
tT:t+T. As we move
forward in time t>1, the video-level instance ID yi
tis assigned by matching clip-level instance Mi
tT:t+Tto all previously processed instance tracks Mj
t0T:t0+Tthat overlap
with this clip, i.e., such that \t;t06=;. For each video-level
instance IDy2Y already in the ID set, we compute a score
qi
t(y)capturing how well Mi
tT:t+Tmatches the tracks that
have been already assigned video-level ID y:
qi
t(y) =P
t0s.t.\t;t06=;PNt0
j=11fyj


key: chroma:document
string_value:
qi
t(y) =P
t0s.t.\t;t06=;PNt0
j=11fyj
t0=ygmi;j
t;t0
P
t0s.t.\t;t06=;PNt0
j=11fyj
t0=yg(5)
time ttime t+5time t+10time t+15time t-15time t-10time t-5
Figure 5: An illustration of instance-speciﬁc features propagated from frame tto other frames in the given video clip. Here,
we visualize propagated activations from one randomly selected feature channel. The activations in the two rows correspond
to two different object instances detected at time t. Our visualizations suggest that MaskProp reliably propagates features
that are speciﬁc to each instance even when instances appear next to each other, and despite the changes in shape, pose and
the nuisances effects of deformation and occlusion.
where 1fyj
t0=ygis an indicator function that is equal
to1whenyj
t0=y, and 0otherwise.Nt0is the number
of detected instances at time t0, andmi;j
t;t0is the previously
deﬁned matching score between instance tracks Mi


key: chroma:document
string_value:
of detected instances at time t0, andmi;j
t;t0is the previously
deﬁned matching score between instance tracks Mi
tT:t+T
andMj
t0T:t0+T. The score qi
t(y)is effectively an average
of the soft IoU computed between instance track Mi
tT:t+T,
and all its overlapping instance tracks that have been assigned video-level instance ID y.
Letq= maxy2Yqi
t(y)be the maximum score obtained
by considering all possible video-level instance IDs y2Y.
Ifqis greater than a certain threshold, the current instance track Mi
tT:t+Tis assigned the video-level instance
IDy= arg maxy2Yqi
t(y). Otherwise, the clip-level track
does not match any of the current video-level instances. In
such case we create a new video-level instance ID and assign it to the the clip-level track while also expanding the
setY, i.e.,yi
t=jYj+ 1andY=Y[fjYj + 1g.
Finally, for each video-level instance ID y2 Y , we
generate the ﬁnal sequence of segmentation instance masks


key: chroma:document
string_value:
Finally, for each video-level instance ID y2 Y , we
generate the ﬁnal sequence of segmentation instance masks
My2RLHWas:
My(t) =(
Mi
tT:t+T(t)ifyi
t=y
0 otherwise .4.4. Implementation Details
Backbone Network. As our backbone we use a Spatiotemporal Sampling Network [5] based on a Deformable
ResNeXt-101-64x4d [12, 41] with a feature pyramid network (FPN) [24] attached on top of it.
Detection Network. For detection, we use a Hybrid Task
Cascade Network [7] with a 3stage cascade.
Mask Propagation Branch. The residual block in the
mask propagation branch consists of two 33convolutions with 128output channels each. The instance feature
propagation is applied to the FPN feature map with the second largest spatial resolution. To propagate instance features from one frame to another, we predict 9 (x;y)offsets
for every pixel, which are then used as input to a 33deformable convolutional layer with 256channels. To capture
motion at different scales, we use three levels of dilated deformable convolutions with dilation rates 3;6;12as in [3].


key: chroma:document
string_value:
motion at different scales, we use three levels of dilated deformable convolutions with dilation rates 3;6;12as in [3].
High-Resolution Mask Reﬁnement. Predicting masks
from RoI features typically leads to low resolution predictions. We address this issue via a high-resolution mask reﬁnement step. Given the center of a detected bounding box,
we crop a 384384patch around the object, preserving
the original aspect ratio. We then feed the RGB patch and
the predicted low-resolution mask through 3residual blocks
each with 128channels to obtain a high-resolution mask.
Method Pre-training Data mAP AP@75 AR@1 AR@10
DeepSORTz[39] Imagenet [34], COCO [25] 26.1 26.1 27.8 31.3
FEELVOSz[37] Imagenet [34], COCO [25] 26.9 29.7 29.9 33.4
OSMNz[43] Imagenet [34], COCO [25] 27.5 29.1 28.6 33.1
MaskTrack R-CNNz[42] Imagenet [34], COCO [25] 30.3 32.6 31.0 35.5


key: chroma:document
string_value:
MaskTrack R-CNNz[42] Imagenet [34], COCO [25] 30.3 32.6 31.0 35.5
MaskTrack R-CNNImagenet [34], COCO [25] 36.9 40.2 34.3 42.9
EnsembleVIS [28] Imagenet [34], COCO [25], Instagram [30], OpenImages [23] 44.8 48.9 42.7 51.7
MaskProp Imagenet [34], COCO [25] 46.6 51.2 44.0 52.6
MaskProp Imagenet [34], COCO [25], OpenImages [23] 50.0 55.9 44.6 54.5
Table 2: The results of video instance segmentation on the YouTube-VIS [42] validation dataset. We evaluate the performance
of each method according to mean average precision (mAP), average precision at 75% IoU threshold (AP@75), and average
recall given top 1(AR@1) and top 10(AR@10) detections. The baselines denoted withzwere implemented by the authors
in [42], whereas the methods marked withwere implemented by us, and use the same backbone and detection networks as
our approach. Despite its simplicity, MaskProp outperforms all prior video instance segmentation methods by a large margin.


key: chroma:document
string_value:
our approach. Despite its simplicity, MaskProp outperforms all prior video instance segmentation methods by a large margin.
Scoring Video-Level Sequences. Each video-level sequence contains a list of classiﬁcation scores and predicted
object labels. To assign a conﬁdence score to each videolevel sequence, we average classiﬁcation scores associated
with that sequence (separately for each object category).
Training. We use a similar training setup as in [7]. The loss
weights for each of the three cascade stages are set to 1;0:5;
and0:25respectively. The loss weight for the semantic segmentation branch is set to 0:1. We train our model on pairs
of frames, where the second frame in a pair is randomly selected with a time gap 2[25;25]relative to ﬁrst frame.
We use a multi-scale training approach implemented by resizing the shorter side of the frame randomly between 400
and800pixels. Our model is trained in a distributed setting using 64GPUs, each GPU holding a single clip. The
training is done for 20epochs with an initial learning rate
of0:008, which is decreased by 10at16and19epochs.
We initialize our model with a Mask R-CNN pretrained on


key: chroma:document
string_value:
of0:008, which is decreased by 10at16and19epochs.
We initialize our model with a Mask R-CNN pretrained on
COCO for the instance segmentation. The hyperparameters
of RPN and FPN are the same as in [7].
Inference. During testing, we run the bounding box prediction branch on 1000 proposals, apply non-maximum suppression, and use boxes with a score higher than 0:1as input to the mask prediction and mask propagation branches.
During inference, our MaskProp is applied to video clips
consisting of 13frames.
5. Experimental Results
In this section, we evaluate MaskProp for video instance
segmentation on YouTube-VIS [42], which contains 2;238
training, 302validation, and 343test videos. Each video
is annotated with per-pixel segmentation, category, and instance labels. The dataset contains 40object categories.
Since the evaluation on the test set is currently closed, we
perform our evaluations on the validation set.
5.1. Quantitative Results
Video instance segmentation is a very recent task [42],
and thus, there are only a few established baselines that we
7 9 11 13 15 17 19 21


key: chroma:document
string_value:
Video instance segmentation is a very recent task [42],
and thus, there are only a few established baselines that we
7 9 11 13 15 17 19 21
Instance Track Clip Length43.54444.54545.54646.547Accuracy (Video mAP)Figure 6: We plot video mAP as a function of an instance
track clip length (denoted as 2T+ 1in the paper). Based on
these results, we observe that optimal video instance segmentation performance is achieved when we propagate instance masks to T= 6previous and subsequent frames.
can compare our work to. We include in our comparison
MaskTrack R-CNN [42], and the EnsembleVIS method,
which won the ICCV 2019 video instance segmentation
challenge [28]. Additionally, to make the comparison with
MaskTrack R-CNN more fair, we reimplement it using the
same backbone and detection networks as our MaskProp
(see MaskTrack R-CNN* in Table 2).
We present our quantitative results in Table 2, where we
assess each method according to 1) mean video average precision (mAP), 2) video average precision at IoU threshold
of75%, and 3) average recall given 1and10highest scored
instances per video. From these results, we observe that
our MaskProp outperforms all the other baselines according


key: chroma:document
string_value:
of75%, and 3) average recall given 1and10highest scored
instances per video. From these results, we observe that
our MaskProp outperforms all the other baselines according
to all four evaluation metrics, thus achieving state-of-theart results in video instance segmentation on YouTube-VIS.
It can be noted that we outperform EnsembleVIS [28] by
1:8mAP even though our approach is much simpler and
even if it uses orders of magnitude less labeled data for
pre-training. Furthermore, compared to the MaskTrack RCNN, our method achieves a 16:3improvement in mAP. We
also note that our implementation of MaskTrack [42] signiﬁcantly improves upon the original work, but it is still 9:7
Method mAP AP@75
FlowNet2 Propagation 31.4 33.6
MaskTrack R-CNN36.9 40.2
MaskProp 46.6 51.2
Table 3: Here, we study the effectiveness of our mask propagation branch. If we replace it with the FlowNet2 propagation scheme, where masks are propagated using the optical
ﬂow predicted by a FlowNet2 network [20], the accuracy
drops from 46:6mAP to 31:4mAP. Similarly, if we replace
our mask propagation branch with the tracking branch from


key: chroma:document
string_value:
drops from 46:6mAP to 31:4mAP. Similarly, if we replace
our mask propagation branch with the tracking branch from
MaskTrack R-CNN, the accuracy drops to 36:9mAP. Note
that all of these baselines are implemented using the same
backbone and detection networks.
worse in mAP compared to our model. Lastly, we note that
additionally pre-training MaskProp on the OpenImages [23]
dataset as was done in [28] further boosts our performance
to50:0mAP.
5.2. Ablation Experiments
Mask Propagation Branch. To investigate the effectiveness of our mask propagation branch, we compare our
method with a FlowNet2 [20] propagation baseline. For
this baseline, we use exactly the same setup as for our
MaskProp, except that instance masks are propagated using the optical ﬂow predicted by a FlowNet2 network [20]
rather than our proposed mask propagation scheme. For a
more complete comparison, we also include the MaskTrack
R-CNNfrom Table 2, which uses the originally proposed
tracking branch [42], but is implemented using the same
backbone and detection networks as our MaskProp.
These baselines allow us to directly compare the effectiveness of our mask propagation scheme versus the propagation mechanisms employed by FlowNet2 and MaskTrack


key: chroma:document
string_value:
backbone and detection networks as our MaskProp.
These baselines allow us to directly compare the effectiveness of our mask propagation scheme versus the propagation mechanisms employed by FlowNet2 and MaskTrack
R-CNN [42] methods. The results in Table 3 show that
MaskProp outperforms these baselines by a large margin.
Instance Track Clip Length. Due to occlusions, object instances in video may not be visible in some frames. If there
are occlusions separated by 2T0+ 2time-steps, we can use
T >T0to predict longer clip-level instance tracks. In Figure 6, we study video instance segmentation performance as
a function of instance track clip length (denoted as 2T+ 1
in the paper). Our results indicate that the best accuracy is
achieved when we use a clip length of 13, meaning that we
propagate instances to 6previous and 6subsequent frames.
High-Resolution Mask Reﬁnement. We also study the impact of our high-resolution mask reﬁnement, described in
Subsection 4.4. We report that removing this reﬁnement
causes a drop of 1:9%in video instance segmentation mAP.
Importance of Frame-Level Instance Masks. As described in our main draft, we use frame-level instance masks


key: chroma:document
string_value:
Importance of Frame-Level Instance Masks. As described in our main draft, we use frame-level instance masks
for instance-speciﬁc feature computation. To investigate theBackbone Network mAP AP@75
ResNet-50 40.0 42.9
ResNet-101 42.5 45.6
ResNeXt-101-64x4d 44.3 48.3
STSN [5]-ResNeXt-101-64x4d 46.6 51.2
Table 4: We study the effect of frame-level instance masks
to our system’s performance. We evaluate our method’s
accuracy when using instance masks obtained from Mask
R-CNN with several different backbones. Our results indicate that frame-level instance masks obtained from stronger
models lead to better video instance segmentation results.
contribution of these masks to the performance of our system, we experiment with masks obtained from several different Mask R-CNN models. In Table 4, we present our
results for this ablation. Our results indicate that framelevel instance masks obtained from stronger models allow
us to achieve better video instance segmentation performance. Thus, we expect that future improvements in image
instance segmentation will further beneﬁt our method.
5.3. Qualitative Results
In Figure 7, we compare our predicted clip-level instance


key: chroma:document
string_value:
instance segmentation will further beneﬁt our method.
5.3. Qualitative Results
In Figure 7, we compare our predicted clip-level instance
tracks (last row of predictions for each clip) with the MaskTrack R-CNN predictions (ﬁrst row of predictions). We
use different colors to represent different object instances.
Our qualitative results suggest that our MaskProp produces
more robust and temporally coherent instance tracks than
MaskTrack R-CNN. Such differences in performance are
especially noticeable when a video contains large object
motion, occlusions, or overlapping objects.
In Figure 5, we also visualize instance-speciﬁc features
that are propagated from frame tto other frames in the
given video clip for two different object instances detected
in framet. Here, we show activations from a randomly selected feature channel. Based on these results, we observe
that our MaskProp reliably propagates features that are speciﬁc to each instance despite motion blur, object deformations and large variations in object appearance.
6. Conclusion
In this work, we introduced MaskProp, a novel architecture for video instance segmentation. Our method is conceptually simple, it does not require large amounts of labeled data for pre-training, and it produces state-of-the-art


key: chroma:document
string_value:
results on YouTube-VIS dataset. In future, we plan to extend MaskProp to scenarios where only bounding box annotations are available. We are also interested in applying our
method to problems such as pose estimation and tracking.
Video ClipMaskPropMaskTrack      R-CNN
Video ClipMaskPropMaskTrack      R-CNN
Video Clip
MaskProp
MaskTrack      R-CNNVideo Clip
MaskPropMaskTrack      R-CNN
Figure 7: We compare our video instance segmentation results with MaskTrack R-CNN [42] predictions. Different object
instances are encoded with different colors. The ﬁrst row for each video shows the original frames. The second row illustrates
the mask predictions of MaskTrack R-CNN and the third row those obtained with our MaskProp. Compared to MaskTrack
R-CNN, our MaskProp tracks object instances more robustly even when they are occluded or overlap with each other.
References
[1] Anurag Arnab and Philip H. S. Torr. Pixelwise instance segmentation with a dynamically instantiated network. In 2017
IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017 ,
pages 879–888, 2017. 2
[2] M. Bai and R. Urtasun. Deep watershed transform for instance segmentation. In 2017 IEEE Conference on Computer


key: chroma:document
string_value:
pages 879–888, 2017. 2
[2] M. Bai and R. Urtasun. Deep watershed transform for instance segmentation. In 2017 IEEE Conference on Computer
Vision and Pattern Recognition (CVPR) , pages 2858–2866,
July 2017. 2
[3] Gedas Bertasius, Christoph Feichtenhofer, Du Tran, Jianbo
Shi, and Lorenzo Torresani. Learning temporal pose estimation from sparsely labeled videos. In Advances in Neural
Information Processing Systems 33 . 2019. 4, 6
[4] Gedas Bertasius, Jianbo Shi, and Lorenzo Torresani. Semantic segmentation with boundary neural ﬁelds. In The
IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , June 2016. 1
[5] Gedas Bertasius, Lorenzo Torresani, and Jianbo Shi. Object
detection in video with spatiotemporal sampling networks.
InECCV (12) , volume 11216 of Lecture Notes in Computer
Science , pages 342–357. Springer, 2018. 2, 4, 6, 8
[6] Sergi Caelles, Kevis-Kokitsi Maninis, Jordi Pont-Tuset,


key: chroma:document
string_value:
[6] Sergi Caelles, Kevis-Kokitsi Maninis, Jordi Pont-Tuset,
Laura Leal-Taixé, Daniel Cremers, and Luc Van Gool. Oneshot video object segmentation. In Computer Vision and Pattern Recognition (CVPR) , 2017. 2
[7] Kai Chen, Jiangmiao Pang, Jiaqi Wang, Yu Xiong, Xiaoxiao
Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jianping Shi,
Wanli Ouyang, Chen Change Loy, and Dahua Lin. Hybrid
task cascade for instance segmentation. In IEEE Conference
on Computer Vision and Pattern Recognition, CVPR 2019,
Long Beach, CA, USA, June 16-20, 2019 , pages 4974–4983.
Computer Vision Foundation / IEEE, 2019. 6, 7
[8] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos,
Kevin Murphy, and Alan L. Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Trans. Pattern Anal.
Mach. Intell. , 40(4):834–848, 2018. 1


key: chroma:document
string_value:
Mach. Intell. , 40(4):834–848, 2018. 1
[9] Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian
Schroff, and Hartwig Adam. Encoder-decoder with atrous
separable convolution for semantic image segmentation. In
Computer Vision - ECCV 2018 - 15th European Conference,
Munich, Germany, September 8-14, 2018, Proceedings, Part
VII, pages 833–851, 2018. 2
[10] Yuhua Chen, Jordi Pont-Tuset, Alberto Montes, and Luc Van
Gool. Blazingly fast video object segmentation with pixelwise metric learning. In CVPR , pages 1189–1198. IEEE
Computer Society, 2018. 2
[11] Jifeng Dai, Kaiming He, and Jian Sun. Instance-aware semantic segmentation via multi-task network cascades. In
CVPR , 2016. 2
[12] J. Dai, H. Qi, Y . Xiong, Y . Li, G. Zhang, H. Hu, and Y . Wei.
Deformable convolutional networks. In 2017 IEEE International Conference on Computer Vision (ICCV) , volume 00,


key: chroma:document
string_value:
Deformable convolutional networks. In 2017 IEEE International Conference on Computer Vision (ICCV) , volume 00,
pages 764–773, Oct. 2017. 4, 6[13] Christoph Feichtenhofer, Axel Pinz, and Andrew Zisserman.
Detect to track and track to detect. In International Conference on Computer Vision (ICCV) , 2017. 2
[14] Ross Girshick. Fast R-CNN. In Proceedings of the International Conference on Computer Vision (ICCV) , 2015. 1
[15] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra
Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition
(CVPR) , 2014. 1
[16] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R-CNN. In Proceedings of the International
Conference on Computer Vision (ICCV) , 2017. 1, 2, 4
[17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
Deep residual learning for image recognition. 2016 IEEE
Conference on Computer Vision and Pattern Recognition
(CVPR) , pages 770–778, 2016. 1, 4


key: chroma:document
string_value:
Deep residual learning for image recognition. 2016 IEEE
Conference on Computer Vision and Pattern Recognition
(CVPR) , pages 770–778, 2016. 1, 4
[18] Alexander Hermans*, Lucas Beyer*, and Bastian Leibe. In
Defense of the Triplet Loss for Person Re-Identiﬁcation.
arXiv preprint arXiv:1703.07737 , 2017. 2
[19] Alba Herrera-Palacio, Carles Ventura, and Xavier Giro-i Nieto. Video object linguistic grounding. In 1st International
Workshop on Multimodal Understanding and Learning for
Embodied Applications , MULEA ’19, pages 49–51, New
York, NY , USA, 2019. ACM. 2
[20] Eddy Ilg, Nikolaus Mayer, Tonmoy Saikia, Margret Keuper,
Alexey Dosovitskiy, and Thomas Brox. Flownet 2.0: Evolution of optical ﬂow estimation with deep networks. CoRR ,
abs/1612.01925, 2016. 8
[21] Suyog Dutt Jain, Bo Xiong, and Kristen Grauman. Fusionseg: Learning to combine motion and appearance for fully
automatic segmentation of generic objects in videos. In 2017


key: chroma:document
string_value:
automatic segmentation of generic objects in videos. In 2017
IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017 ,
pages 2117–2126, 2017. 2
[22] Alexander Kirillov, Evgeny Levinkov, Bjoern Andres, Bogdan Savchynskyy, and Carsten Rother. InstanceCut: from
edges to instances with multicut. In CVPR , 2017. 2
[23] Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan
Popov, Matteo Malloci, Tom Duerig, and Vittorio Ferrari.
The open images dataset v4: Uniﬁed image classiﬁcation,
object detection, and visual relationship detection at scale,
2018. 2, 7, 8
[24] Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He,
Bharath Hariharan, and Serge Belongie. Feature pyramid
networks for object detection. In CVPR , 2017. 6
[25] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays,


key: chroma:document
string_value:
networks for object detection. In CVPR , 2017. 6
[25] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays,
Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence
Zitnick. Microsoft coco: Common objects in context. In
European Conference on Computer Vision (ECCV) , Zürich,
September 2014. 2, 3, 7
[26] S. Liu, J. Jia, S. Fidler, and R. Urtasun. Sgn: Sequential
grouping networks for instance segmentation. In 2017 IEEE
International Conference on Computer Vision (ICCV) , pages
3516–3524, Oct 2017. 2
[27] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully
convolutional networks for semantic segmentation. In
CVPR , 2015. 1
[28] Jonathon Luiten, Philip H.S. Torr, and Bastian Leibe. Video
instance segmentation 2019: A winning approach for combined detection, segmentation, classiﬁcation and tracking.
In2019 IEEE International Conference on Computer Vision
Workshops, ICCV Workshops 2019, Seoul, South Korea, October 27-November 2, 2019 . IEEE, 2019. 1, 2, 7, 8


key: chroma:document
string_value:
Workshops, ICCV Workshops 2019, Seoul, South Korea, October 27-November 2, 2019 . IEEE, 2019. 1, 2, 7, 8
[29] Jonathon Luiten, Paul V oigtlaender, and Bastian Leibe. Premvos: Proposal-generation, reﬁnement and merging for
video object segmentation. In Asian Conference on Computer Vision , 2018. 2
[30] Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan,
Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe,
and Laurens van der Maaten. Exploring the limits of weakly
supervised pretraining. In ECCV , 2018. 2, 7
[31] Aljoša Ošep, Paul V oigtlaender, Jonathon Luiten, Stefan
Breuers, and Bastian Leibe. Large-scale object mining for
object discovery from unlabeled video. ICRA , 2019. 2
[32] F. Perazzi, A. Khoreva, R. Benenson, B. Schiele, and
A.Sorkine-Hornung. Learning video object segmentation
from static images. In Computer Vision and Pattern Recognition , 2017. 2


key: chroma:document
string_value:
A.Sorkine-Hornung. Learning video object segmentation
from static images. In Computer Vision and Pattern Recognition , 2017. 2
[33] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.
Faster R-CNN: Towards real-time object detection with region proposal networks. In Neural Information Processing
Systems (NIPS) , 2015. 1
[34] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy,
Aditya Khosla, Michael Bernstein, Alexander C. Berg, and
Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. arXiv:1409.0575 , 2014. 2, 7
[35] Deqing Sun, Xiaodong Yang, Ming-Yu Liu, and Jan Kautz.
PWC-Net: CNNs for optical ﬂow using pyramid, warping,
and cost volume. 2018. 2
[36] P. Tokmakov, K. Alahari, and C. Schmid. Learning video
object segmentation with visual memory. In 2017 IEEE International Conference on Computer Vision (ICCV) , pages
4491–4500, Oct 2017. 2


key: chroma:document
string_value:
object segmentation with visual memory. In 2017 IEEE International Conference on Computer Vision (ICCV) , pages
4491–4500, Oct 2017. 2
[37] Paul V oigtlaender, Yuning Chai, Florian Schroff, Hartwig
Adam, Bastian Leibe, and Liang-Chieh Chen. Feelvos: Fast
end-to-end embedding learning for video object segmentation. In CVPR , 2019. 2, 7
[38] Paul V oigtlaender, Michael Krause, Aljosa Osep, Jonathon
Luiten, Berin Balachandar Gnana Sekar, Andreas Geiger,
and Bastian Leibe. Mots: Multi-object tracking and segmentation. In Proceedings IEEE Conf. on Computer Vision and
Pattern Recognition (CVPR) , June 2019. 2
[39] Nicolai Wojke, Alex Bewley, and Dietrich Paulus. Simple
online and realtime tracking with a deep association metric.
CoRR , abs/1703.07402, 2017. 7
[40] Fanyi Xiao and Yong Jae Lee. Video object detection with an
aligned spatial-temporal memory. In European Conference
on Computer Vision (ECCV) , 2018. 2


key: chroma:document
string_value:
[40] Fanyi Xiao and Yong Jae Lee. Video object detection with an
aligned spatial-temporal memory. In European Conference
on Computer Vision (ECCV) , 2018. 2
[41] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and
Kaiming He. Aggregated residual transformations for deep
neural networks. In CVPR , 2017. 6[42] Linjie Yang, Yuchen Fan, and Ning Xu. Video instance segmentation. In ICCV , 2019. 1, 2, 3, 7, 8, 9
[43] Linjie Yang, Yanran Wang, Xuehan Xiong, Jianchao Yang,
and Aggelos K. Katsaggelos. Efﬁcient video object segmentation via network modulation. In 2018 IEEE Conference on
Computer Vision and Pattern Recognition, CVPR 2018, Salt
Lake City, UT, USA, June 18-22, 2018 , pages 6499–6507,
2018. 2, 7
[44] Jifeng Dai Xiangyang Ji Yi Li, Haozhi Qi and Yichen Wei.
Fully convolutional instance-aware semantic segmentation.
InCVPR , 2017. 2
[45] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang


key: chroma:document
string_value:
InCVPR , 2017. 2
[45] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang
Wang, and Jiaya Jia. Pyramid scene parsing network. In 2017
IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017 ,
pages 6230–6239, 2017. 1
[46] Xizhou Zhu, Yujie Wang, Jifeng Dai, Lu Yuan, and Yichen
Wei. Flow-guided feature aggregation for video object detection. In International Conference on Computer Vision
(ICCV) , 2017. 2
[47] Idil Esen Zulﬁkar, Jonathon Luiten, and Bastian Leibe. Unovost: Unsupervised ofﬂine video object segmentation and
tracking for the 2019 unsupervised davis challenge. 2


key: chroma:document
string_value:
Quantifying Attention Flow in Transformers
Samira Abnar
ILLC, University of Amsterdam
s.abnar@uva.nlWillem Zuidema
ILLC, University of Amsterdam
w.h.zuidema@uva.nl
Abstract
In the Transformer model, “self-attention”
combines information from attended embeddings into the representation of the focal embedding in the next layer. Thus, across layers of the Transformer, information originating
from different tokens gets increasingly mixed.
This makes attention weights unreliable as explanations probes. In this paper, we consider
the problem of quantifying this ﬂow of information through self-attention. We propose two
methods for approximating the attention to input tokens given attention weights, attention
rollout andattention ﬂow , as post hoc methods
when we use attention weights as the relative
relevance of the input tokens. We show that
these methods give complementary views on
the ﬂow of information, and compared to raw
attention, both yield higher correlations with
importance scores of input tokens obtained using an ablation method and input gradients.
1 Introduction
Attention (Bahdanau et al., 2015; Vaswani et al.,
2017) has become the key building block of neural sequence processing models, and visualizing
attention weights is the easiest and most popular


key: chroma:document
string_value:
2017) has become the key building block of neural sequence processing models, and visualizing
attention weights is the easiest and most popular
approach to interpret a model’s decisions and to
gain insights about its internals (Vaswani et al.,
2017; Xu et al., 2015; Wang et al., 2016; Lee et al.,
2017; Dehghani et al., 2019; Rockt ¨aschel et al.,
2016; Chen and Ji, 2019; Coenen et al., 2019; Clark
et al., 2019). Although it is wrong to equate attention with explanation (Pruthi et al., 2019; Jain and
Wallace, 2019), it can offer plausible and meaningful interpretations (Wiegreffe and Pinter, 2019;
Vashishth et al., 2019; Vig, 2019). In this paper,
we focus on problems arising when we move to the
higher layers of a model, due to lack of token identiﬁability of the embeddings in higher layers (Brunner et al., 2020).We propose two simple but effective methods to
compute attention scores to input tokens (i.e., token
attention ) at each layer, by taking raw attentions
(i.e., embedding attention ) of that layer as well as
those from the precedent layers. These methods


key: chroma:document
string_value:
attention ) at each layer, by taking raw attentions
(i.e., embedding attention ) of that layer as well as
those from the precedent layers. These methods
are based on modelling the information ﬂow in the
network with a DAG (Directed Acyclic Graph), in
which the nodes are input tokens and hidden embeddings, edges are the attentions from the nodes
in each layer to those in the previous layer, and
the weights of the edges are the attention weights.
The ﬁrst method, attention rollout , assumes that
the identities of input tokens are linearly combined
through the layers based on the attention weights.
To adjust attention weights, it rolls out the weights
to capture the propagation of information from input tokens to intermediate hidden embeddings. The
second method, attention ﬂow , considers the attention graph as a ﬂow network. Using a maximum
ﬂow algorithm, it computes maximum ﬂow values,
from hidden embeddings (sources) to input tokens
(sinks). In both methods, we take the residual connection in the network into account to better model
the connections between input tokens and hidden
embedding. We show that compared to raw attention, the token attentions from attention rollout and
attention ﬂow have higher correlations with the importance scores obtained from input gradients as


key: chroma:document
string_value:
embedding. We show that compared to raw attention, the token attentions from attention rollout and
attention ﬂow have higher correlations with the importance scores obtained from input gradients as
well as blank-out , an input ablation based attribution method. Furthermore, we visualize the token
attention weights and demonstrate that they are better approximations of how input tokens contribute
to a predicted output, compared to raw attention.
It is noteworthy that the techniques we propose
in this paper, are not toward making hidden embeddings more identiﬁable, or providing better attention weights for better performance, but a new set
of attention weights that take token identity problem into consideration and can serve as a better
diagnostic tool for visualization and debugging.arXiv:2005.00928v2  [cs.LG]  31 May 2020
(a) Embedding attentions
 (b) Attention rollout
 (c) Attention ﬂow
Figure 1: Visualisation of attention weights.
Figure 2: Raw Attention maps for the CLS token at
different layers.
2 Setups and Problem Statement
In our analysis, we focus on the verb number prediction task, i.e., predicting singularity or plurality
of a verb of a sentence, when the input is the sentence up to the verb position. We use the subjectverb agreement dataset (Linzen et al., 2016). This


key: chroma:document
string_value:
of a verb of a sentence, when the input is the sentence up to the verb position. We use the subjectverb agreement dataset (Linzen et al., 2016). This
task and dataset are convenient choices, as they offer a clear hypothesis about what part of the input
is essential to get the right solution. For instance,
given “ the key to the cabinets ” as the input, we
know that attending to “key” helps the model predict singular as output while attending to “cabinets”
(anagreement attractor , with the opposite number)
is unhelpful.
We train a Transformer encoder, with GPT2 Transformer blocks as described in (Radford
et al., 2019; Wolf et al., 2019) (without masking).
The model has 6 layers, and 8 heads, with hidden/embedding size of 128. Similar to Bert (Devlin et al., 2019) we add a CLS token and use
its embedding in the ﬁnal layer as the input to
the classiﬁer. The accuracy of the model on the
subject-verb agreement task is 0:96. To facilitate
replication of our experiments we will make the
implementations of the models we use and algorithms we introduce publicly available at https:
//github.com/samiraabnar/attention_flow .


key: chroma:document
string_value:
implementations of the models we use and algorithms we introduce publicly available at https:
//github.com/samiraabnar/attention_flow .
We start by visualizing raw attention in Figure 1a
(like Vig 2019). The example given here is correctly classiﬁed. Crucially, only in the ﬁrst couple
of layers, there are some distinctions in the attention patterns for different positions, while in higher
layers the attention weights are rather uniform. Fig-ure 2 (left) gives raw attention scores of the CLS
token over input tokens (x-axis) at different layers (y-axis), which similarly lack an interpretable
pattern.These observations reﬂect the fact that as
we go deeper into the model, the embeddings are
more contextualized and may all carry similar information. This underscores the need to track down
attention weights all the way back to the input layer
and is in line with ﬁndings of Serrano and Smith
(2019), who show that attention weights do not
necessarily correspond to the relative importance
of input tokens.
To quantify the usefulness of raw attention
weights, and the two alternatives that we consider
in the next section, besides input gradients, we
employ an input ablation method, blank-out , to estimate an importance score for each input token.


key: chroma:document
string_value:
in the next section, besides input gradients, we
employ an input ablation method, blank-out , to estimate an importance score for each input token.
Blank-out replaces each token in the input, one
by one, with UNK and measures how much it affects the predicted probability of the correct class.
We compute the Spearman’s rank correlation coefﬁcient between the attention weights of the CLS
embedding in the ﬁnal layer and the importance
scores from blank-out. As shown in the ﬁrst row
of Table 1, the correlation between raw attention
weights of the CLS token and blank-out scores is
rather low, except for the ﬁrst layer. As we can see
in Table 2 this is also the case when we compute
the correlations with input gradients.
L1 L2 L3 L4 L5 L6
Raw 0.690.27 0.100.43 -0.110.49 -0.090.52 0.200.45 0.290.39
Rollout 0.320.26 0.380.27 0.510.26 0.620.26 0.700.25 0.710.24


key: chroma:document
string_value:
Flow 0.320.26 0.440.29 0.700.25 0.700.22 0.710.22 0.700.22
Table 1: SpearmanR correlation of attention based importance with blank-out scores for 2000 samples from
the test set for the verb number prediction model.
3 Attention Rollout and Attention Flow
Attention rollout and attention ﬂow recursively
compute the token attentions in each layer of a
L1 L2 L3 L4 L5 L6
Raw 0.530.33 0.160.38 -0.060.42 0.000.47 0.240.40 0.460.35
Rollout 0.220.31 0.270.32 0.390.32 0.470.32 0.530.32 0.540.31
Flow 0.220.31 0.310.34 0.540.32 0.610.28 0.600.28 0.610.28
Table 2: SpearmanR correlation of attention based importance with input gradients for 2000 samples from
the test set for the verb number prediction model.


key: chroma:document
string_value:
Table 2: SpearmanR correlation of attention based importance with input gradients for 2000 samples from
the test set for the verb number prediction model.
given model given the embedding attentions as input. They differ in the assumptions they make
about how attention weights in lower layers affect
the ﬂow of information to the higher layers and
whether to compute the token attentions relative to
each other or independently.
To compute how information propagates from
the input layer to the embeddings in higher layers, it is crucial to take the residual connections
in the model into account as well as the attention
weights. In a Transformer block, both self-attention
and feed-forward networks are wrapped by residual connections, i.e., the input to these modules is
added to their output. When we only use attention
weights to approximate the ﬂow of information in
Transformers, we ignore the residual connections.
But these connections play a signiﬁcant role in
tying corresponding positions in different layers.
Hence, to compute attention rollout and attention
ﬂow, we augment the attention graph with extra
weights to represent residual connections. Given
the attention module with residual connection, we
compute values in layer l+1asVl+1=Vl+WattVl,
where Wattis the attention matrix. Thus, we have


key: chroma:document
string_value:
compute values in layer l+1asVl+1=Vl+WattVl,
where Wattis the attention matrix. Thus, we have
Vl+1= (Watt+I)Vl. So, to account for residual
connections, we add an identity matrix to the attention matrix and re-normalize the weights. This
results in A= 0:5Watt+ 0:5I, where Ais the raw
attention updated by residual connections.
Furthermore, analyzing individual heads requires accounting for mixing of information between heads through a position-wise feed-forward
network in Transformer block. Using attention rollout and attention ﬂow, it is also possible to analyze
each head separately. We explain in more details
in Appendix A.1. However, in our analysis in this
paper, for simplicity, we average the attention at
each layer over all heads.
Attention rollout Attention rollout is an intuitive
way of tracking down the information propagated
from the input layer to the embeddings in the higher
layers. Given a Transformer with Llayers, we want
to compute the attention from all positions in layerlito all positions in layer lj, where j < i . In the
attention graph, a path from node vat position k
inli, to node uat position minlj, is a series of


key: chroma:document
string_value:
attention graph, a path from node vat position k
inli, to node uat position minlj, is a series of
edges that connect these two nodes. If we look
at the weight of each edge as the proportion of
information transferred between two nodes, we
can compute how much of the information at v
is propagated to uthrough a particular path by
multiplying the weights of all edges in that path.
Since there may be more than one path between
two nodes in the attention graph, to compute the
total amount of information propagated from vtou,
we sum over all possible paths between these two
nodes. At the implementation level, to compute the
attentions from litolj, we recursively multiply the
attention weights matrices in all the layers below.
~A(li) =
A(li)~A(li1)ifi > j
A(li) ifi=j(1)
In this equation, ~Ais attention rollout, Ais raw attention and the multiplication operation is a matrix
multiplication. With this formulation, to compute
input attention we set j= 0.
Attention ﬂow In graph theory, a ﬂow network
is a directed graph with a “capacity” associated


key: chroma:document
string_value:
input attention we set j= 0.
Attention ﬂow In graph theory, a ﬂow network
is a directed graph with a “capacity” associated
with each edge. Formally, given G= (V; E)is a
graph, where Vis the set of nodes, and Eis the set
of edges in G;C=fcuv2Rj8u; vwhere eu;v2
E^u6=vgdenotes the capacities of the edges and
s; t2Vare the source and target (sink) nodes respectively; ﬂow is a mapping of edges to real numbers,f:E!R, that satisﬁes two conditions: (a)
capacity constraint : for each edge the ﬂow value
should not exceed its capacity, jfuvcuvj; (b)
ﬂow conservation : for all nodes except sandtthe
input ﬂow should be equal to output ﬂow –sum
of the ﬂow of outgoing edges should be equal to
sum of the ﬂow of incoming edges. Given a ﬂow
network, a maximum ﬂow algorithm ﬁnds a ﬂow
which has the maximum possible value between s
andt(Cormen et al., 2009).


key: chroma:document
string_value:
network, a maximum ﬂow algorithm ﬁnds a ﬂow
which has the maximum possible value between s
andt(Cormen et al., 2009).
Treating the attention graph as a ﬂow network,
where the capacities of the edges are attention
weights, using any maximum ﬂow algorithm, we
can compute the maximum attention ﬂow from any
node in any of the layers to any of the input nodes.
We can use this maximum-ﬂow-value as an approximation of the attention to input nodes. In attention
ﬂow, the weight of a single path is the minimum
value of the weights of the edges in the path, instead of the product of the weights. Besides, we
attention rollout
 attention ﬂow
Figure 3: Attention maps for the CLS token
.
can not compute the attention for node sto node
tby adding up the weights of all paths between
these two nodes, since there might be an overlap
between the paths and this might result in overﬂow
in the overlapping edges.
It is noteworthy that both of the proposed methods can be computed in polynomial time. O(dn2)
for attention rollout and O(d2n4)for attention
ﬂow, where dis the depth of the model, and nis


key: chroma:document
string_value:
for attention rollout and O(d2n4)for attention
ﬂow, where dis the depth of the model, and nis
the number of tokens.
4 Analysis and Discussion
Now, we take a closer look at these three views of
attention. Figure 1 depicts raw attention, attention
rollout and attention ﬂow for a correctly classiﬁed
example across different layers. It is noteworthy
that the ﬁrst layer of attention rollout and attention
ﬂow are the same, and their only difference with
raw attention is the addition of residual connections. As we move to the higher layers, we see that
the residual connections fade away. Moreover, in
contrast to raw attention, the patterns of attention
rollout and attention ﬂow become more distinctive
in the higher layers.
Figures 2 and 3 show the weights from raw attention, attention rollout and attention ﬂow for the
CLS embedding over input tokens (x-axis) in all
6 layers (y-axis) for three examples. The ﬁrst example is the same as the one in Figure 1. The second example is “ the article on NNP large systems
<?>”. The model correctly classiﬁes this example and changing the subject of the missing verb


key: chroma:document
string_value:
<?>”. The model correctly classiﬁes this example and changing the subject of the missing verb
from “article” to “article s” ﬂips the decision of the
model. The third example is “ here the NNS differ
in that the female <?>”, which is a miss-classiﬁed
example and again changing “NNS” (plural noun)
to “NNP” (singular proper noun) ﬂips the decision
of the model.
For all cases, the raw attention weights are almost uniform above layer three (discussed before).
raw attention
attention rollout
attention ﬂow
(a) “The author talked to Sara about mask book .”
raw attention
attention rollout
attention ﬂow
(b)“Mary convinced John of mask love.”
Figure 4: Bert attention maps. We look at the attention
weights from the mask embedding to the two potential
references for it, e.g. “author” and “Sara” in (a) and
“Mary” and “John” in (b). The bars, at the left, show
the relative predicted probability for the two possible


key: chroma:document
string_value:
“Mary” and “John” in (b). The bars, at the left, show
the relative predicted probability for the two possible
pronouns, “his” and “her”.
In the case of the correctly classiﬁed example, we
observe that both attention rollout and attention
ﬂow assign relatively high weights to both the subject of the verb, “article’ and the attractor, “systems”. For the miss-classiﬁed example, both attention rollout and attention ﬂow assign relatively
high scores to the “NNS” token which is not the
subject of the verb. This can explain the wrong
prediction of the model.
The main difference between attention rollout
and attention ﬂow is that attention ﬂow weights are
amortized among the set of most attended tokens,
as expected. Attention ﬂow can indicate a set of
input tokens that are important for the ﬁnal decision.
Thus we do not get sharp distinctions among them.
On the other hand, attention rollout weights are
more focused compared to attention ﬂow weights,
which is sensible for the third example but not as
much for the second one.
L1 L3 L5 L6


key: chroma:document
string_value:
more focused compared to attention ﬂow weights,
which is sensible for the third example but not as
much for the second one.
L1 L3 L5 L6
Raw 0.120.21 0.090.21 0.080.20 0.090.21
Rollout 0.110.19 0.120.21 0.130.21 0.130.20
Flow 0.110.19 0.110.21 0.120.22 0.140.21
Table 3: SpearmanR correlation of attention based importance with input gradients for 100 samples from the
test set for the DistillBERT model ﬁne tuned on SST-2.
Furthermore, as shown in Table 1 and 2 both
attention rollout and attention ﬂow, are better
correlated with blank-out scores and input gradients compared to raw attention, but attention ﬂow
weights are more reliable than attention rollout.
The difference between these two methods is rooted
in their different views of attention weights. Attention ﬂow views them as capacities, and at every
step of the algorithm, it uses as much of the capacity as possible. Hence, attention ﬂow computes the
maximum possibility of token identities to propagate to the higher layers. Whereas attention rollout


key: chroma:document
string_value:
maximum possibility of token identities to propagate to the higher layers. Whereas attention rollout
views them as proportion factors and at every step,
it allows token identities to be propagated to higher
layers exactly based on this proportion factors. This
makes attention rollout stricter than attention ﬂow,
and so we see that attention rollout provides us with
more focused attention patterns. However, since
we are making many simplifying assumptions, the
strictness of attention rollout does not lead to more
accurate results, and the relaxation of attention ﬂow
seems to be a useful property.
At last, to illustrate the application of attention ﬂow and attention rollout on different tasks
and different models, we examine them on two
pretrained BERT models. We use the models
available at https://github.com/huggingface/
transformers .
Table 3 shows the correlation of the importance
score obtained from raw attention, attention rollout
and attention ﬂow from a DistillBERT (Sanh et al.,
2019) model ﬁne-tuned to solve “SST-2” (Socher
et al., 2013), the sentiment analysis task from the
glue benchmark (Wang et al., 2018). Even though
for this model, all three methods have very low
correlation with the input gradients, we can still see


key: chroma:document
string_value:
glue benchmark (Wang et al., 2018). Even though
for this model, all three methods have very low
correlation with the input gradients, we can still see
that attention rollout and attention ﬂow are slightly
better than raw attention.
Furthermore, in Figure 4, we show an example
of applying these methods to a pre-trained Bert
to see how it resolves the pronouns in a sentence.
What we do here is to feed the model with a sentence, masking a pronoun. Next, we look at the
prediction of the model for the masked word and
compare the probabilities assigned to “her” and
“his”. Then we look at raw attention, attention rollout and attention ﬂow weights of the embeddings
for the masked pronoun at all the layers. In the
ﬁrst example, in Figure 4a, attention rollout and
attention ﬂow are consistent with each other and
the prediction of the model. Whereas, the ﬁnal
layer of raw attention does not seem to be consistent with the prediction of the models, and it varies
a lot across different layers. In the second example, in Figure 4b, only attention ﬂow weights areconsistent with the prediction of the model.
5 Conclusion
Translating embedding attentions to token attentions can provide us with better explanations about


key: chroma:document
string_value:
5 Conclusion
Translating embedding attentions to token attentions can provide us with better explanations about
models’ internals. Yet, we should be cautious about
our interpretation of these weights, because, we
are making many simplifying assumptions when
we approximate information ﬂow in a model with
the attention weights. Our ideas are simple and
task/architecture agnostic. In this paper, we insisted on sticking with simple ideas that only require attention weights and can be easily employed
in any task or architecture that uses self-attention.
We should note that all our analysis in this paper is
for a Transformer encoder, with no casual masking.
Since in Transformer decoder, future tokens are
masked, naturally there is more attention toward
initial tokens in the input sequence, and both attention rollout and attention ﬂow will be biased toward
these tokens. Hence, to apply these methods on a
Transformer decoder, we should ﬁrst normalize
based on the receptive ﬁeld of attention.
Following this work, we can build the attention
graph with effective attention weights (Brunner
et al., 2020) instead of raw attentions. Furthermore,
we can come up with a new method that adjusts the
attention weights using gradient-based attribution
methods (Ancona et al., 2019).
Acknowledgements


key: chroma:document
string_value:
we can come up with a new method that adjusts the
attention weights using gradient-based attribution
methods (Ancona et al., 2019).
Acknowledgements
We thank Mostafa Dehghani, Wilker Aziz, and the
anonymous reviewers for their valuable feedback
and comments on this work. The work presented
here was funded by the Netherlands Organization
for Scientiﬁc Research (NWO), through a Gravitation Grant 024.001.006 to the Language in Interaction Consortium.
References
Marco Ancona, Enea Ceolini, Cengiz ¨Oztireli, and
Markus Gross. 2019. Gradient-Based Attribution
Methods , pages 169–191. Springer International
Publishing.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly
learning to align and translate. In proceedings of
the 2015 International Conference on Learning Representations .
Gino Brunner, Yang Liu, Damian Pascual, Oliver
Richter, Massimiliano Ciaramita, and Roger Wattenhofer. 2020. On identiﬁability in transformers. In
International Conference on Learning Representations .
Hanjie Chen and Yangfeng Ji. 2019. Improving the interpretability of neural sentiment classiﬁers via data


key: chroma:document
string_value:
International Conference on Learning Representations .
Hanjie Chen and Yangfeng Ji. 2019. Improving the interpretability of neural sentiment classiﬁers via data
augmentation. arXiv preprint arXiv:1909.04225 .
Kevin Clark, Urvashi Khandelwal, Omer Levy, and
Christopher D. Manning. 2019. What does BERT
look at? an analysis of BERT’s attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP:
Analyzing and Interpreting Neural Networks for
NLP, pages 276–286, Florence, Italy. Association
for Computational Linguistics.
Andy Coenen, Emily Reif, Ann Yuan, Been Kim,
Adam Pearce, Fernanda Vi ´egas, and Martin Wattenberg. 2019. Visualizing and measuring the geometry
of bert. arXiv preprint arXiv:1906.02715 .
Thomas H. Cormen, Charles E. Leiserson, Ronald L.
Rivest, and Clifford Stein. 2009. Introduction to
Algorithms, Third Edition , 3rd edition. The MIT
Press.
Mostafa Dehghani, Stephan Gouws, Oriol Vinyals,


key: chroma:document
string_value:
Algorithms, Third Edition , 3rd edition. The MIT
Press.
Mostafa Dehghani, Stephan Gouws, Oriol Vinyals,
Jakob Uszkoreit, and Łukasz Kaiser. 2019. Universal transformers. In proceedings of the 2019 International Conference on Learning Representations .
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and
Kristina Toutanova. 2019. BERT: Pre-training of
deep bidirectional transformers for language understanding. In proceedings of the 2019 Conference
of the North American Chapter of the Association
for Computational Linguistics: Human Language
Technologies . Association for Computational Linguistics.
Sarthak Jain and Byron C. Wallace. 2019. Attention
is not Explanation. In proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 3543–3556. Association
for Computational Linguistics.
Jaesong Lee, Joong-Hwi Shin, and Jun-Seok Kim.
2017. Interactive visualization and manipulation
of attention-based neural machine translation. In
proceedings of the 2017 Conference on Empirical
Methods in Natural Language Processing: System
Demonstrations , pages 121–126.
Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg.


key: chroma:document
string_value:
Methods in Natural Language Processing: System
Demonstrations , pages 121–126.
Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg.
2016. Assessing the ability of LSTMs to learn
syntax-sensitive dependencies. Transactions of the
Association for Computational Linguistics , 4:521–
535.
Danish Pruthi, Mansi Gupta, Bhuwan Dhingra, Graham Neubig, and Zachary C Lipton. 2019. Learning
to deceive with attention-based explanations. arXiv
preprint arXiv:1909.07913 .Alec Radford, Jeffrey Wu, Rewon Child, David Luan,
Dario Amodei, and Ilya Sutskever. 2019. Language
models are unsupervised multitask learners. OpenAI
Blog , 1(8).
Tim Rockt ¨aschel, Edward Grefenstette, Karl Moritz
Hermann, Tomas Kocisky, and Phil Blunsom. 2016.
Reasoning about entailment with neural attention.
InInternational Conference on Learning Representations (ICLR) .
Victor Sanh, Lysandre Debut, Julien Chaumond, and
Thomas Wolf. 2019. Distilbert, a distilled version of
bert: smaller, faster, cheaper and lighter.


key: chroma:document
string_value:
Thomas Wolf. 2019. Distilbert, a distilled version of
bert: smaller, faster, cheaper and lighter.
Soﬁa Serrano and Noah A. Smith. 2019. Is attention
interpretable? In proceedings of the 57th Annual
Meeting of the Association for Computational Linguistics . Association for Computational Linguistics.
Richard Socher, Alex Perelygin, Jean Wu, Jason
Chuang, Christopher D. Manning, Andrew Ng, and
Christopher Potts. 2013. Recursive deep models
for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on
Empirical Methods in Natural Language Processing ,
pages 1631–1642, Seattle, Washington, USA. Association for Computational Linguistics.
Shikhar Vashishth, Shyam Upadhyay, Gaurav Singh
Tomar, and Manaal Faruqui. 2019. Attention interpretability across nlp tasks. arXiv preprint
arXiv:1909.11218 .
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob
Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz
Kaiser, and Illia Polosukhin. 2017. Attention is all
you need. In Advances in neural information processing systems , pages 5998–6008.


key: chroma:document
string_value:
Kaiser, and Illia Polosukhin. 2017. Attention is all
you need. In Advances in neural information processing systems , pages 5998–6008.
Jesse Vig. 2019. Visualizing attention in transformerbased language models. arXiv preprint
arXiv:1904.02679 .
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018.
GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP , pages 353–355, Brussels, Belgium.
Association for Computational Linguistics.
Yequan Wang, Minlie Huang, Li Zhao, et al. 2016.
Attention-based lstm for aspect-level sentiment classiﬁcation. In proceedings of the 2016 conference on
empirical methods in natural language processing ,
pages 606–615.
Sarah Wiegreffe and Yuval Pinter. 2019. Attention is
not not explanation. In proceedings of the 2019 Conference on Empirical Methods in Natural Language
Processing and the 9th International Joint Conference on Natural Language Processing (EMNLPIJCNLP) . Association for Computational Linguistics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien


key: chroma:document
string_value:
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien
Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R’emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface’s transformers: State-of-the-art natural language processing. ArXiv , abs/1910.03771.
Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho,
Aaron Courville, Ruslan Salakhudinov, Rich Zemel,
and Yoshua Bengio. 2015. Show, attend and tell:
Neural image caption generation with visual attention. In proceedings of International Conference on
Machine Learning , pages 2048–2057.
A Appendices
A.1 Single Head Analysis
For analysing the attention weights, with multihead setup, we could either analyze attention heads
separately, or we could average all heads and have a
single attention graph. However, we should be careful that treating attention heads separately could
potentially mean that we are assuming there is no
mixing of information between heads, which is not
true as we combine information of heads in the
position-wise feed-forward network on top of selfattention in a transformer block. It is possible to
analyse the role of each head in isolation of all other


key: chroma:document
string_value:
position-wise feed-forward network on top of selfattention in a transformer block. It is possible to
analyse the role of each head in isolation of all other
heads using attention rollout and attention ﬂow. To
not make the assumption that there is no mixing
of information between heads, for computing the
“input attention”, we will treat all the layers below
the layer of interest as single head layers, i.e., we
sum the attentions of all heads in the layers below.
For example, we can compute attention rollout for
headkat layer ias~A(i; k) =A(i; k)A(i), where,
A(i)is attention rollout computed for layer iwith
the single head assumption.


key: chroma:document
string_value:
Language (Technology) is Power: A Critical Survey of “Bias” in NLP
Su Lin Blodgett
College of Information and Computer Sciences
University of Massachusetts Amherst
blodgett@cs.umass.eduSolon Barocas
Microsoft Research
Cornell University
solon@microsoft.com
Hal Daumé III
Microsoft Research
University of Maryland
me@hal3.nameHanna Wallach
Microsoft Research
wallach@microsoft.com
Abstract
We survey 146 papers analyzing “bias” in
NLP systems, ﬁnding that their motivations
are often vague, inconsistent, and lacking
in normative reasoning, despite the fact that
analyzing “bias” is an inherently normative
process. We further ﬁnd that these papers’
proposed quantitative techniques for measuring or mitigating “bias” are poorly matched to
their motivations and do not engage with the
relevant literature outside of NLP. Based on
these ﬁndings, we describe the beginnings of a
path forward by proposing three recommendations that should guide work analyzing “bias”
in NLP systems. These recommendations rest
on a greater recognition of the relationships
between language and social hierarchies,
encouraging researchers and practitioners
to articulate their conceptualizations of


key: chroma:document
string_value:
in NLP systems. These recommendations rest
on a greater recognition of the relationships
between language and social hierarchies,
encouraging researchers and practitioners
to articulate their conceptualizations of
“bias”—i.e., what kinds of system behaviors
are harmful, in what ways, to whom, and why,
as well as the normative reasoning underlying
these statements—and to center work around
the lived experiences of members of communities affected by NLP systems, while interrogating and reimagining the power relations
between technologists and such communities.
1 Introduction
A large body of work analyzing “bias” in natural
language processing (NLP) systems has emerged
in recent years, including work on “bias” in embedding spaces (e.g., Bolukbasi et al., 2016a; Caliskan
et al., 2017; Gonen and Goldberg, 2019; May
et al., 2019) as well as work on “bias” in systems
developed for a breadth of tasks including language
modeling (Lu et al., 2018; Bordia and Bowman,2019), coreference resolution (Rudinger et al.,
2018; Zhao et al., 2018a), machine translation (Vanmassenhove et al., 2018; Stanovsky et al., 2019),
sentiment analysis (Kiritchenko and Mohammad,


key: chroma:document
string_value:
2018; Zhao et al., 2018a), machine translation (Vanmassenhove et al., 2018; Stanovsky et al., 2019),
sentiment analysis (Kiritchenko and Mohammad,
2018), and hate speech/toxicity detection (e.g.,
Park et al., 2018; Dixon et al., 2018), among others.
Although these papers have laid vital groundwork by illustrating some of the ways that NLP
systems can be harmful, the majority of them fail
to engage critically with what constitutes “bias”
in the ﬁrst place. Despite the fact that analyzing
“bias” is an inherently normative process—in
which some system behaviors are deemed good
and others harmful—papers on “bias” in NLP
systems are rife with unstated assumptions about
what kinds of system behaviors are harmful, in
what ways, to whom, and why. Indeed, the term
“bias” (or “gender bias” or “racial bias”) is used
to describe a wide range of system behaviors, even
though they may be harmful in different ways, to
different groups, or for different reasons. Even
papers analyzing “bias” in NLP systems developed
for the same task often conceptualize it differently.
For example, the following system behaviors


key: chroma:document
string_value:
papers analyzing “bias” in NLP systems developed
for the same task often conceptualize it differently.
For example, the following system behaviors
are all understood to be self-evident statements of
“racial bias”: (a) embedding spaces in which embeddings for names associated with African Americans
are closer (compared to names associated with
European Americans) to unpleasant words than
pleasant words (Caliskan et al., 2017); (b) sentiment analysis systems yielding different intensity
scores for sentences containing names associated
with African Americans and sentences containing
names associated with European Americans (Kiritchenko and Mohammad, 2018); and (c) toxicityarXiv:2005.14050v2  [cs.CL]  29 May 2020
detection systems scoring tweets containing features associated with African-American English as
more offensive than tweets without these features
(Davidson et al., 2019; Sap et al., 2019). Moreover,
some of these papers focus on “racial bias”
expressed in written text, while others focus on
“racial bias” against authors. This use of imprecise
terminology obscures these important differences.
We survey 146 papers analyzing “bias” in NLP
systems, ﬁnding that their motivations are often
vague and inconsistent. Many lack any normative


key: chroma:document
string_value:
We survey 146 papers analyzing “bias” in NLP
systems, ﬁnding that their motivations are often
vague and inconsistent. Many lack any normative
reasoning for why the system behaviors that are
described as “bias” are harmful, in what ways, and
to whom. Moreover, the vast majority of these
papers do not engage with the relevant literature
outside of NLP to ground normative concerns when
proposing quantitative techniques for measuring
or mitigating “bias.” As a result, we ﬁnd that many
of these techniques are poorly matched to their
motivations, and are not comparable to one another.
We then describe the beginnings of a path
forward by proposing three recommendations
that should guide work analyzing “bias” in NLP
systems. We argue that such work should examine
the relationships between language and social hierarchies; we call on researchers and practitioners
conducting such work to articulate their conceptualizations of “bias” in order to enable conversations
about what kinds of system behaviors are harmful,
in what ways, to whom, and why; and we recommend deeper engagements between technologists
and communities affected by NLP systems. We
also provide several concrete research questions
that are implied by each of our recommendations.
2 Method
Our survey includes all papers known to us


key: chroma:document
string_value:
and communities affected by NLP systems. We
also provide several concrete research questions
that are implied by each of our recommendations.
2 Method
Our survey includes all papers known to us
analyzing “bias” in NLP systems—146 papers in
total. We omitted papers about speech, restricting
our survey to papers about written text only. To
identify the 146 papers, we ﬁrst searched the ACL
Anthology1for all papers with the keywords “bias”
or “fairness” that were made available prior to May
2020. We retained all papers about social “bias,”
and discarded all papers about other deﬁnitions of
the keywords (e.g., hypothesis-only bias, inductive
bias, media bias). We also discarded all papers using “bias” in NLP systems to measure social “bias”
in text or the real world (e.g., Garg et al., 2018).
To ensure that we did not exclude any relevant
1https://www.aclweb.org/anthology/NLP task Papers
Embeddings (type-level or contextualized) 54
Coreference resolution 20
Language modeling or dialogue generation 17
Hate-speech detection 17
Sentiment analysis 15
Machine translation 8
Tagging or parsing 5


key: chroma:document
string_value:
Coreference resolution 20
Language modeling or dialogue generation 17
Hate-speech detection 17
Sentiment analysis 15
Machine translation 8
Tagging or parsing 5
Surveys, frameworks, and meta-analyses 20
Other 22
Table 1: The NLP tasks covered by the 146 papers.
papers without the keywords “bias” or “fairness,”
we also traversed the citation graph of our initial
set of papers, retaining any papers analyzing “bias”
in NLP systems that are cited by or cite the papers
in our initial set. Finally, we manually inspected
any papers analyzing “bias” in NLP systems from
leading machine learning, human–computer interaction, and web conferences and workshops, such
as ICML, NeurIPS, AIES, FAccT, CHI, and WWW,
along with any relevant papers that were made
available in the “Computation and Language” and
“Computers and Society” categories on arXiv prior
to May 2020, but found that they had already been
identiﬁed via our traversal of the citation graph. We
provide a list of all 146 papers in the appendix. In
Table 1, we provide a breakdown of the NLP tasks
covered by the papers. We note that counts do not


key: chroma:document
string_value:
provide a list of all 146 papers in the appendix. In
Table 1, we provide a breakdown of the NLP tasks
covered by the papers. We note that counts do not
sum to 146, because some papers cover multiple
tasks. For example, a paper might test the efﬁcacy
of a technique for mitigating “bias” in embedding spaces in the context of sentiment analysis.
Once identiﬁed, we then read each of the 146 papers with the goal of categorizing their motivations
and their proposed quantitative techniques for measuring or mitigating “bias.” We used a previously
developed taxonomy of harms for this categorization, which differentiates between so-called allocational andrepresentational harms (Barocas et al.,
2017; Crawford, 2017). Allocational harms arise
when an automated system allocates resources (e.g.,
credit) or opportunities (e.g., jobs) unfairly to different social groups; representational harms arise
when a system (e.g., a search engine) represents
some social groups in a less favorable light than
others, demeans them, or fails to recognize their
existence altogether. Adapting and extending this
taxonomy, we categorized the 146 papers’ motivations and techniques into the following categories:
.Allocational harms.
Papers
Category Motivation Technique


key: chroma:document
string_value:
taxonomy, we categorized the 146 papers’ motivations and techniques into the following categories:
.Allocational harms.
Papers
Category Motivation Technique
Allocational harms 30 4
Stereotyping 50 58
Other representational harms 52 43
Questionable correlations 47 42
Vague/unstated 23 0
Surveys, frameworks, and
meta-analyses20 20
Table 2: The categories into which the 146 papers fall.
.Representational harms:2
.Stereotyping that propagates negative generalizations about particular social groups.
.Differences in system performance for different social groups, language that misrepresents the distribution of different social
groups in the population, or language that
isdenigrating to particular social groups.
.Questionable correlations between system behavior and features of language that are typically associated with particular social groups.
.Vague descriptions of “bias” (or “gender
bias” or “racial bias”) or no description at all.
.Surveys, frameworks, and meta-analyses .
In Table 2 we provide counts for each of the
six categories listed above. (We also provide a
list of the papers that fall into each category in the
appendix.) Again, we note that the counts do not
sum to 146, because some papers state multiple


key: chroma:document
string_value:
list of the papers that fall into each category in the
appendix.) Again, we note that the counts do not
sum to 146, because some papers state multiple
motivations, propose multiple techniques, or propose a single technique for measuring or mitigating
multiple harms. Table 3, which is in the appendix,
contains examples of the papers’ motivations and
techniques across a range of different NLP tasks.
3 Findings
Categorizing the 146 papers’ motivations and proposed quantitative techniques for measuring or mitigating “bias” into the six categories listed above enabled us to identify several commonalities, which
we present below, along with illustrative quotes.
2We grouped several types of representational harms into
two categories to reﬂect that the main point of differentiation
between the 146 papers’ motivations and proposed quantitative
techniques for measuring or mitigating “bias” is whether or not
they focus on stereotyping. Among the papers that do not focus on stereotyping, we found that most lack sufﬁciently clear
motivations and techniques to reliably categorize them further.3.1 Motivations
Papers state a wide range of motivations,
multiple motivations, vague motivations, and
sometimes no motivations at all. We found that
the papers’ motivations span all six categories, with
several papers falling into each one. Appropriately,


key: chroma:document
string_value:
sometimes no motivations at all. We found that
the papers’ motivations span all six categories, with
several papers falling into each one. Appropriately,
papers that provide surveys or frameworks for analyzing “bias” in NLP systems often state multiple
motivations (e.g., Hovy and Spruit, 2016; Bender,
2019; Sun et al., 2019; Rozado, 2020; Shah et al.,
2020). However, as the examples in Table 3 (in the
appendix) illustrate, many other papers (33%) do
so as well. Some papers (16%) state only vague
motivations or no motivations at all. For example,
“[N]o human should be discriminated on the basis
of demographic attributes by an NLP system. ”
—Kaneko and Bollegala (2019)
“[P]rominent word embeddings [...] encode
systematic biases against women and black people
[...] implicating many NLP systems in scaling up
social injustice. ” —May et al. (2019)
These examples leave unstated what it might mean
for an NLP system to “discriminate,” what constitutes “systematic biases,” or how NLP systems
contribute to “social injustice” (itself undeﬁned).


key: chroma:document
string_value:
contribute to “social injustice” (itself undeﬁned).
Papers’ motivations sometimes include no normative reasoning. We found that some papers
(32%) are not motivated by any apparent normative
concerns, often focusing instead on concerns about
system performance. For example, the ﬁrst quote
below includes normative reasoning—namely that
models should not use demographic information
to make predictions—while the other focuses on
learned correlations impairing system performance.
“In [text classiﬁcation], models are expected to
make predictions with the semantic information
rather than with the demographic group identity
information ( e.g., ‘gay’, ‘black’) contained in the
sentences. ” —Zhang et al. (2020a)
“An over-prevalence of some gendered forms in the
training data leads to translations with identiﬁable
errors. Translations are better for sentences
involving men and for sentences containing
stereotypical gender roles. ”
—Saunders and Byrne (2020)
Even when papers do state clear motivations,
they are often unclear about why the system behaviors that are described as “bias” are harmful, in what ways, and to whom. We found that


key: chroma:document
string_value:
they are often unclear about why the system behaviors that are described as “bias” are harmful, in what ways, and to whom. We found that
even papers with clear motivations often fail to explain what kinds of system behaviors are harmful,
in what ways, to whom, and why. For example,
“Deploying these word embedding algorithms in
practice, for example in automated translation
systems or as hiring aids, runs the serious risk of
perpetuating problematic biases in important
societal contexts. ” —Brunet et al. (2019)
“[I]f the systems show discriminatory behaviors in
the interactions, the user experience will be
adversely affected. ” —Liu et al. (2019)
These examples leave unstated what “problematic
biases” or non-ideal user experiences might look
like, how the system behaviors might result in
these things, and who the relevant stakeholders
or users might be. In contrast, we ﬁnd that papers
that provide surveys or frameworks for analyzing
“bias” in NLP systems often name who is harmed,
acknowledging that different social groups may
experience these systems differently due to their
different relationships with NLP systems or
different social positions. For example, Ruane
et al. (2019) argue for a “deep understanding of


key: chroma:document
string_value:
different relationships with NLP systems or
different social positions. For example, Ruane
et al. (2019) argue for a “deep understanding of
the user groups [sic] characteristics, contexts, and
interests” when designing conversational agents.
Papers about NLP systems developed for the
same task often conceptualize “bias” differently. Even papers that cover the same NLP task
often conceptualize “bias” in ways that differ substantially and are sometimes inconsistent. Rows 3
and 4 of Table 3 (in the appendix) contain machine
translation papers with different conceptualizations
of “bias,” leading to different proposed techniques,
while rows 5 and 6 contain papers on “bias” in embedding spaces that state different motivations, but
propose techniques for quantifying stereotyping.
Papers’ motivations conﬂate allocational and
representational harms. We found that the papers’ motivations sometimes (16%) name immediate representational harms, such as stereotyping,
alongside more distant allocational harms, which,
in the case of stereotyping, are usually imagined as
downstream effects of stereotypes on résumé ﬁltering. Many of these papers use the imagined downstream effects to justify focusing on particular system behaviors, even when the downstream effects
are not measured. Papers on “bias” in embedding


key: chroma:document
string_value:
are not measured. Papers on “bias” in embedding
spaces are especially likely to do this because embeddings are often used as input to other systems:
“However, none of these papers [on embeddings]
have recognized how blatantly sexist the
embeddings are and hence risk introducing biases
of various types into real-world systems. ”
—Bolukbasi et al. (2016a)“It is essential to quantify and mitigate gender bias
in these embeddings to avoid them from affecting
downstream applications. ” —Zhou et al. (2019)
In contrast, papers that provide surveys or frameworks for analyzing “bias” in NLP systems treat
representational harms as harmful in their own
right. For example, Mayﬁeld et al. (2019) and
Ruane et al. (2019) cite the harmful reproduction
of dominant linguistic norms by NLP systems (a
point to which we return in section 4), while Bender
(2019) outlines a range of harms, including seeing
stereotypes in search results and being made invisible to search engines due to language practices.
3.2 Techniques
Papers’ techniques are not well grounded in the
relevant literature outside of NLP. Perhaps unsurprisingly given that the papers’ motivations are
often vague, inconsistent, and lacking in normative


key: chroma:document
string_value:
Papers’ techniques are not well grounded in the
relevant literature outside of NLP. Perhaps unsurprisingly given that the papers’ motivations are
often vague, inconsistent, and lacking in normative
reasoning, we also found that the papers’ proposed
quantitative techniques for measuring or mitigating
“bias” do not effectively engage with the relevant
literature outside of NLP. Papers on stereotyping
are a notable exception: the Word Embedding
Association Test (Caliskan et al., 2017) draws on
the Implicit Association Test (Greenwald et al.,
1998) from the social psychology literature, while
several techniques operationalize the well-studied
“Angry Black Woman” stereotype (Kiritchenko
and Mohammad, 2018; May et al., 2019; Tan
and Celis, 2019) and the “double bind” faced by
women (May et al., 2019; Tan and Celis, 2019), in
which women who succeed at stereotypically male
tasks are perceived to be less likable than similarly
successful men (Heilman et al., 2004). Tan and
Celis (2019) also examine the compounding effects
of race and gender, drawing on Black feminist
scholarship on intersectionality (Crenshaw, 1989).
Papers’ techniques are poorly matched to their


