Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali
Farhadi, Hannaneh Hajishirzi, and Noah Smith.
2020. Fine-Tuning Pretrained Language Models:
Weight Initializations, Data Orders, and Early
Stopping. arXiv:2002.06305 [cs].
Yanai Elazar, Shauli Ravfogel, Alon Jacovi, and
Yoav Goldberg. 2020. When Bert Forgets How
To POS: Amnesic Probing of Linguistic Proper-
ties and MLM Predictions. arXiv:2006.00995
[cs].
Kawin Ethayarajh. 2019. How Contextual are
Contextualized Word Representations? Compar-
ing the Geometry of BERT, ELMo, and GPT-2
Embeddings. In Proceedings of the 2019 Con-
ference on Empirical Methods in Natural Lan-
guage Processing and the 9th International Joint
Conference on Natural Language Processing
(EMNLP-IJCNLP), pages 55–65, Hong Kong,
China. Association for Computational Linguis-
tics.
Allyson Ettinger. 2019. What BERT is
not: Lessons from a new suite of psy-
cholinguistic diagnostics for language models.
arXiv:1907.13528 [cs].
Angela Fan, Edouard Grave, and Armand Joulin.
2019. Reducing Transformer Depth on Demand
with Structured Dropout. In International Con-
ference on Learning Representations.
Maxwell Forbes, Ari Holtzman, and Yejin Choi.
2019. Do Neural Language Representations
Learn Physical Commonsense? In Proceedings
of the 41st Annual Conference of the Cognitive
Science Society (CogSci 2019), page 7.
Jonathan Frankle and Michael Carbin. 2019. The
Lottery Ticket Hypothesis: Finding Sparse,
Trainable Neural Networks. In International
Conference on Learning Representations.
Prakhar Ganesh, Yao Chen, Xin Lou, Moham-
mad Ali Khan, Yin Yang, Deming Chen, Mari-
anne Winslett, Hassan Sajjad, and Preslav Nakov.
2020. Compressing large-scale transformer-
based models: A case study on BERT. arXiv
preprint arXiv:2002.11985.
Siddhant Garg, Thuy Vu, and Alessandro Moschitti.
2020. TANDA: Transfer and Adapt Pre-Trained
Transformer Models for Answer Sentence Selec-
tion. In AAAI.
Michael Glass, Alﬁo Gliozzo, Rishav Chakravarti,
Anthony Ferritto, Lin Pan, G P Shrivatsa Bhar-
gav, Dinesh Garg, and Avi Sil. 2020. Span
Selection Pre-training for Question Answering.
In Proceedings of the 58th Annual Meeting of
the Association for Computational Linguistics,
pages 2773–2782, Online. Association for Com-
putational Linguistics.
Goran Glavaš and Ivan Vuli ´c. 2020. Is Super-
vised Syntactic Parsing Beneﬁcial for Language
Understanding? An Empirical Investigation.
arXiv:2008.06788 [cs].
Adele Goldberg. 2006. Constructions at Work: The
Nature of Generalization in Language. Oxford
University Press, USA.
Yoav Goldberg. 2019. Assessing BERT’s syntactic
abilities. arXiv preprint arXiv:1901.05287.
Linyuan Gong, Di He, Zhuohan Li, Tao Qin, Liwei
Wang, and Tieyan Liu. 2019. Efﬁcient training
of BERT by progressively stacking. In Interna-
tional Conference on Machine Learning, pages
2337–2346.
Mitchell A Gordon, Kevin Duh, and Nicholas An-
drews. 2020. Compressing BERT: Studying the
effects of weight pruning on transfer learning.
arXiv preprint arXiv:2002.08307.
Saurabh Goyal, Anamitra Roy Choudhary, Venkate-
san Chakaravarthy, Saurabh ManishRaje, Yogish
Sabharwal, and Ashish Verma. 2020. Power-
bert: Accelerating BERT inference for classiﬁ-
cation tasks. arXiv preprint arXiv:2001.08950.
Fu-Ming Guo, Sijia Liu, Finlay S. Mungall, Xue
Lin, and Yanzhi Wang. 2019. Reweighted Prox-
imal Pruning for Large-Scale Language Repre-
sentation. arXiv:1909.12486 [cs, stat].
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pa-
supat, and Ming-Wei Chang. 2020. REALM:
Retrieval-Augmented Language Model Pre-
Training. arXiv:2002.08909 [cs].
Yaru Hao, Li Dong, Furu Wei, and Ke Xu. 2019.
Visualizing and Understanding the Effective-
ness of BERT. In Proceedings of the 2019
Conference on Empirical Methods in Natural
Language Processing and the 9th International
Joint Conference on Natural Language Process-
ing (EMNLP-IJCNLP), pages 4143–4152, Hong