International Conference on Learning Represen-
tations.
Olga Kovaleva, Alexey Romanov, Anna Rogers,
and Anna Rumshisky. 2019. Revealing the Dark
Secrets of BERT. In Proceedings of the 2019
Conference on Empirical Methods in Natural
Language Processing and the 9th International
Joint Conference on Natural Language Process-
ing (EMNLP-IJCNLP), pages 4356–4365, Hong
Kong, China. Association for Computational
Linguistics.
Kalpesh Krishna, Gaurav Singh Tomar, Ankur P.
Parikh, Nicolas Papernot, and Mohit Iyyer. 2020.
Thieves on Sesame Street! Model Extraction of
BERT-Based APIs. In ICLR 2020.
Varun Kumar, Ashutosh Choudhary, and Eunah
Cho. 2020. Data Augmentation using Pre-
Trained Transformer Models. arXiv:2003.02245
[cs].
Ilia Kuznetsov and Iryna Gurevych. 2020. A Mat-
ter of Framing: The Impact of Linguistic For-
malism on Probing Results. arXiv:2004.14999
[cs].
Guillaume Lample and Alexis Conneau. 2019.
Cross-Lingual Language Model Pretraining.
arXiv:1901.07291 [cs].
Zhenzhong Lan, Mingda Chen, Sebastian Good-
man, Kevin Gimpel, Piyush Sharma, and Radu
Soricut. 2020a. ALBERT: A Lite BERT for
Self-Supervised Learning of Language Repre-
sentations. In ICLR.
Zhenzhong Lan, Mingda Chen, Sebastian Good-
man, Kevin Gimpel, Piyush Sharma, and Radu
Soricut. 2020b. ALBERT: A Lite BERT for
Self-supervised Learning of Language Represen-
tations. In ICLR 2020.
Cheolhyoung Lee, Kyunghyun Cho, and Wanmo
Kang. 2019. Mixout: Effective regularization to
ﬁnetune large-scale pretrained language models.
arXiv preprint arXiv:1909.11299.
Mike Lewis, Yinhan Liu, Naman Goyal, Mar-
jan Ghazvininejad, Abdelrahman Mohamed,
Omer Levy, Ves Stoyanov, and Luke Zettle-
moyer. 2019. BART: Denoising Sequence-to-
Sequence Pre-Training for Natural Language
Generation, Translation, and Comprehension.
arXiv:1910.13461 [cs, stat].
Changmao Li and Jinho D. Choi. 2020. Transform-
ers to Learn Hierarchical Contexts in Multiparty
Dialogue for Span-based Question Answering.
In Proceedings of the 58th Annual Meeting of
the Association for Computational Linguistics,
pages 5709–5714, Online. Association for Com-
putational Linguistics.
Zhuohan Li, Eric Wallace, Sheng Shen, Kevin Lin,
Kurt Keutzer, Dan Klein, and Joseph E Gonzalez.
2020. Train large, then compress: Rethinking
model size for efﬁcient training and inference of
transformers. arXiv preprint arXiv:2002.11794.
Yongjie Lin, Yi Chern Tan, and Robert Frank. 2019.
Open Sesame: Getting inside BERT’s Linguistic
Knowledge. In Proceedings of the 2019 ACL
Workshop BlackboxNLP: Analyzing and Inter-
preting Neural Networks for NLP , pages 241–
253.
Nelson F. Liu, Matt Gardner, Yonatan Belinkov,
Matthew E. Peters, and Noah A. Smith. 2019a.
Linguistic Knowledge and Transferability of
Contextual Representations. In Proceedings
of the 2019 Conference of the North Ameri-
can Chapter of the Association for Computa-
tional Linguistics: Human Language Technolo-
gies, Volume 1 (Long and Short Papers), pages
1073–1094, Minneapolis, Minnesota. Associa-
tion for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du,
Mandar Joshi, Danqi Chen, Omer Levy, Mike
Lewis, Luke Zettlemoyer, and Veselin Stoyanov.
2019b. RoBERTa: A Robustly Optimized BERT
Pretraining Approach. arXiv:1907.11692 [cs].
Xiaofei Ma, Zhiguo Wang, Patrick Ng, Ramesh
Nallapati, and Bing Xiang. 2019. Universal Text
Representation from BERT: An Empirical Study.
arXiv:1910.07973 [cs].
Christopher D. Manning, Kevin Clark, John He-
witt, Urvashi Khandelwal, and Omer Levy. 2020.
Emergent linguistic structure in artiﬁcial neural
networks trained by self-supervision. Proceed-
ings of the National Academy of Sciences, page
201907367.
Chandler May, Alex Wang, Shikha Bordia,
Samuel R. Bowman, and Rachel Rudinger. 2019.
On Measuring Social Biases in Sentence En-
coders. In Proceedings of the 2019 Confer-