ence of the North American Chapter of the As-
sociation for Computational Linguistics: Hu-
man Language Technologies, Volume 1 (Long
and Short Papers), pages 622–628, Minneapo-
lis, Minnesota. Association for Computational
Linguistics.
J. S. McCarley, Rishav Chakravarti, and Avirup
Sil. 2020. Structured Pruning of a BERT-based
Question Answering Model. arXiv:1910.06360
[cs].
R. Thomas McCoy, Tal Linzen, Ewan Dunbar, and
Paul Smolensky. 2019a. RNNs implicitly imple-
ment tensor-product representations. In Interna-
tional Conference on Learning Representations.
Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019b.
Right for the Wrong Reasons: Diagnosing Syn-
tactic Heuristics in Natural Language Inference.
In Proceedings of the 57th Annual Meeting of
the Association for Computational Linguistics,
pages 3428–3448, Florence, Italy. Association
for Computational Linguistics.
Alessio Miaschi and Felice Dell’Orletta. 2020.
Contextual and Non-Contextual Word Embed-
dings: An in-depth Linguistic Investigation. In
Proceedings of the 5th Workshop on Representa-
tion Learning for NLP, pages 110–119.
Paul Michel, Omer Levy, and Graham Neubig.
2019. Are Sixteen Heads Really Better than
One? Advances in Neural Information Process-
ing Systems 32 (NIPS 2019).
Timothee Mickus, Denis Paperno, Mathieu Con-
stant, and Kees van Deemeter. 2019. What do
you mean, BERT? assessing BERT as a dis-
tributional semantics model. arXiv preprint
arXiv:1911.05758.
Microsoft. 2020. Turing-NLG: A 17-billion-
parameter language model by microsoft.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S.
Corrado, and Jeff Dean. 2013. Distributed repre-
sentations of words and phrases and their compo-
sitionality. In Advances in Neural Information
Processing Systems 26 (NIPS 2013), pages 3111–
3119.
Jiaqi Mu and Pramod Viswanath. 2018. All-but-
the-top: Simple and effective postprocessing for
word representations. In International Confer-
ence on Learning Representations.
Timothy Niven and Hung-Yu Kao. 2019. Probing
Neural Network Comprehension of Natural Lan-
guage Arguments. In Proceedings of the 57th
Annual Meeting of the Association for Computa-
tional Linguistics, pages 4658–4664, Florence,
Italy. Association for Computational Linguistics.
Matthew E. Peters, Mark Neumann, Robert Logan,
Roy Schwartz, Vidur Joshi, Sameer Singh, and
Noah A. Smith. 2019a. Knowledge Enhanced
Contextual Word Representations. In Proceed-
ings of the 2019 Conference on Empirical Meth-
ods in Natural Language Processing and the
9th International Joint Conference on Natural
Language Processing (EMNLP-IJCNLP), pages
43–54, Hong Kong, China. Association for Com-
putational Linguistics.
Matthew E. Peters, Sebastian Ruder, and Noah A.
Smith. 2019b. To Tune or Not to Tune? Adapt-
ing Pretrained Representations to Diverse Tasks.
In Proceedings of the 4th Workshop on Repre-
sentation Learning for NLP (RepL4NLP-2019),
pages 7–14, Florence, Italy. Association for
Computational Linguistics.
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel,
Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and
Alexander Miller. 2019. Language Models as
Knowledge Bases? In Proceedings of the 2019
Conference on Empirical Methods in Natural
Language Processing and the 9th International
Joint Conference on Natural Language Process-
ing (EMNLP-IJCNLP), pages 2463–2473, Hong
Kong, China. Association for Computational
Linguistics.
Jason Phang, Thibault Févry, and Samuel R. Bow-
man. 2019. Sentence Encoders on STILTs: Sup-
plementary Training on Intermediate Labeled-
Data Tasks. arXiv:1811.01088 [cs].
Tiago Pimentel, Josef Valvoda, Rowan Hall Maud-
slay, Ran Zmigrod, Adina Williams, and Ryan
Cotterell. 2020. Information-Theoretic Probing
for Linguistic Structure. arXiv:2004.03061 [cs].
Nina Poerner, Ulli Waltinger, and Hinrich Schütze.
2019. BERT is not a knowledge base
(yet): Factual knowledge vs. name-based rea-
soning in unsupervised qa. arXiv preprint
arXiv:1911.03681.