Sastry, Amanda Askell, Sandhini Agarwal, Ariel
Herbert-V oss, Gretchen Krueger, Tom Henighan,
Rewon Child, Aditya Ramesh, Daniel M.
Ziegler, Jeffrey Wu, Clemens Winter, Christo-
pher Hesse, Mark Chen, Eric Sigler, Mateusz
Litwin, Scott Gray, Benjamin Chess, Jack Clark,
Christopher Berner, Sam McCandlish, Alec
Radford, Ilya Sutskever, and Dario Amodei.
2020. Language Models are Few-Shot Learners.
arXiv:2005.14165 [cs].
Gino Brunner, Yang Liu, Damian Pascual, Oliver
Richter, Massimiliano Ciaramita, and Roger
Wattenhofer. 2020. On Identiﬁability in Trans-
formers. In International Conference on Learn-
ing Representations.
Tianlong Chen, Jonathan Frankle, Shiyu Chang,
Sijia Liu, Yang Zhang, Zhangyang Wang, and
Michael Carbin. 2020. The Lottery Ticket
Hypothesis for Pre-trained BERT Networks.
arXiv:2007.12223 [cs, stat].
Xingyi Cheng, Weidi Xu, Kunlong Chen, Wei
Wang, Bin Bi, Ming Yan, Chen Wu, Luo Si, Wei
Chu, and Taifeng Wang. 2019. Symmetric Reg-
ularization based BERT for Pair-Wise Semantic
Reasoning. arXiv:1909.03405 [cs].
Kevin Clark, Urvashi Khandelwal, Omer Levy,
and Christopher D. Manning. 2019. What Does
BERT Look at? An Analysis of BERT’s Atten-
tion. In Proceedings of the 2019 ACL Workshop
BlackboxNLP: Analyzing and Interpreting Neu-
ral Networks for NLP, pages 276–286, Florence,
Italy. Association for Computational Linguistics.
Kevin Clark, Minh-Thang Luong, Quoc V . Le, and
Christopher D. Manning. 2020. ELECTRA: Pre-
Training Text Encoders as Discriminators Rather
Than Generators. In International Conference
on Learning Representations.
Stephane Clinchant, Kweon Woo Jung, and Vas-
silina Nikoulina. 2019. On the use of BERT
for Neural Machine Translation. In Proceedings
of the 3rd Workshop on Neural Generation and
Translation, pages 108–117, Hong Kong. Asso-
ciation for Computational Linguistics.
Alexis Conneau, Kartikay Khandelwal, Naman
Goyal, Vishrav Chaudhary, Guillaume Wen-
zek, Francisco Guzmán, Edouard Grave, Myle
Ott, Luke Zettlemoyer, and Veselin Stoyanov.
2019. Unsupervised Cross-Lingual Representa-
tion Learning at Scale. arXiv:1911.02116 [cs].
Gonçalo M. Correia, Vlad Niculae, and André F. T.
Martins. 2019. Adaptively Sparse Transform-
ers. In Proceedings of the 2019 Conference on
Empirical Methods in Natural Language Pro-
cessing and the 9th International Joint Confer-
ence on Natural Language Processing (EMNLP-
IJCNLP), pages 2174–2184, Hong Kong, China.
Association for Computational Linguistics.
Matt Crane. 2018. Questionable Answers in Ques-
tion Answering Research: Reproducibility and
Variability of Published Results. Transactions of
the Association for Computational Linguistics,
6:241–252.
Leyang Cui, Sijie Cheng, Yu Wu, and Yue Zhang.
2020. Does BERT Solve Commonsense Task via
Commonsense Knowledge? arXiv:2008.03945
[cs].
Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin,
Ziqing Yang, Shijin Wang, and Guoping Hu.
2019. Pre-Training with Whole Word Masking
for Chinese BERT. arXiv:1906.08101 [cs].
Jeff Da and Jungo Kasai. 2019. Cracking the
Contextual Commonsense Code: Understand-
ing Commonsense Reasoning Aptitude of Deep
Contextual Representations. In Proceedings of
the First Workshop on Commonsense Inference
in Natural Language Processing , pages 1–12,
Hong Kong, China. Association for Computa-
tional Linguistics.
Joe Davison, Joshua Feldman, and Alexander Rush.
2019. Commonsense Knowledge Mining from
Pretrained Models. In Proceedings of the 2019
Conference on Empirical Methods in Natural
Language Processing and the 9th International
Joint Conference on Natural Language Process-
ing (EMNLP-IJCNLP), pages 1173–1178, Hong
Kong, China. Association for Computational
Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and
Kristina Toutanova. 2019. BERT: Pre-training
of Deep Bidirectional Transformers for Lan-
guage Understanding. In Proceedings of the
2019 Conference of the North American Chapter
of the Association for Computational Linguis-
tics: Human Language Technologies, Volume 1