Sai Prasanna, Anna Rogers, and Anna Rumshisky.
2020. When BERT Plays the Lottery, All Tick-
ets Are Winning. In Proceedings of the 2020
Conference on Empirical Methods in Natural
Language Processing, Online. Association for
Computational Linguistics.
Oﬁr Press, Noah A. Smith, and Omer Levy. 2020.
Improving Transformer Models by Reordering
their Sublayers. In Proceedings of the 58th An-
nual Meeting of the Association for Computa-
tional Linguistics, pages 2996–3005, Online. As-
sociation for Computational Linguistics.
Yada Pruksachatkun, Jason Phang, Haokun Liu,
Phu Mon Htut, Xiaoyi Zhang, Richard Yuanzhe
Pang, Clara Vania, Katharina Kann, and
Samuel R. Bowman. 2020. Intermediate-Task
Transfer Learning with Pretrained Language
Models: When and Why Does It Work? In Pro-
ceedings of the 58th Annual Meeting of the As-
sociation for Computational Linguistics, pages
5231–5247, Online. Association for Computa-
tional Linguistics.
Colin Raffel, Noam Shazeer, Adam Roberts,
Katherine Lee, Sharan Narang, Michael Matena,
Yanqi Zhou, Wei Li, and Peter J. Liu.
2019. Exploring the Limits of Transfer Learn-
ing with a Uniﬁed Text-to-Text Transformer.
arXiv:1910.10683 [cs, stat].
Alessandro Raganato, Yves Scherrer, and Jörg
Tiedemann. 2020. Fixed Encoder Self-Attention
Patterns in Transformer-Based Machine Transla-
tion. arXiv:2002.10260 [cs].
Alessandro Raganato and Jörg Tiedemann. 2018.
An Analysis of Encoder Representations in
Transformer-Based Machine Translation. In Pro-
ceedings of the 2018 EMNLP Workshop Black-
boxNLP: Analyzing and Interpreting Neural Net-
works for NLP, pages 287–297, Brussels, Bel-
gium. Association for Computational Linguis-
tics.
Marco Tulio Ribeiro, Tongshuang Wu, Carlos
Guestrin, and Sameer Singh. 2020. Beyond Ac-
curacy: Behavioral Testing of NLP Models with
CheckList. In Proceedings of the 58th Annual
Meeting of the Association for Computational
Linguistics, pages 4902–4912, Online. Associa-
tion for Computational Linguistics.
Kyle Richardson, Hai Hu, Lawrence S. Moss, and
Ashish Sabharwal. 2020. Probing Natural Lan-
guage Inference Models through Semantic Frag-
ments. In AAAI 2020.
Kyle Richardson and Ashish Sabharwal. 2019.
What Does My QA Model Know? Devis-
ing Controlled Probes using Expert Knowledge.
arXiv:1912.13337 [cs].
Adam Roberts, Colin Raffel, and Noam Shazeer.
2020. How Much Knowledge Can You Pack Into
the Parameters of a Language Model? arXiv
preprint arXiv:2002.08910.
Anna Rogers, Olga Kovaleva, Matthew Downey,
and Anna Rumshisky. 2020. Getting Closer to
AI Complete Question Answering: A Set of Pre-
requisite Real Tasks. In AAAI, page 11.
Rudolf Rosa and David Mare ˇcek. 2019. Induc-
ing syntactic trees from BERT representations.
arXiv preprint arXiv:1906.11511.
Victor Sanh, Lysandre Debut, Julien Chaumond,
and Thomas Wolf. 2019a. Distilbert, a distilled
version of BERT: smaller, faster, cheaper and
lighter. arXiv preprint arXiv:1910.01108.
Victor Sanh, Lysandre Debut, Julien Chaumond,
and Thomas Wolf. 2019b. DistilBERT, a dis-
tilled version of BERT: Smaller, faster, cheaper
and lighter. In 5th Workshop on Energy Efﬁcient
Machine Learning and Cognitive Computing -
NeurIPS 2019.
Victor Sanh, Thomas Wolf, and Alexander M.
Rush. 2020. Movement Pruning: Adaptive Spar-
sity by Fine-Tuning. arXiv:2005.07683 [cs].
Timo Schick and Hinrich Schütze. 2020.
BERTRAM: Improved Word Embeddings
Have Big Impact on Contextualized Model
Performance. In Proceedings of the 58th
Annual Meeting of the Association for Compu-
tational Linguistics, pages 3996–4007, Online.
Association for Computational Linguistics.
Florian Schmidt and Thomas Hofmann. 2020.
BERT as a Teacher: Contextual Embeddings
for Sequence-Level Reward. arXiv preprint
arXiv:2003.02738.
Roy Schwartz, Jesse Dodge, Noah A. Smith,
and Oren Etzioni. 2019. Green AI.
arXiv:1907.10597 [cs, stat].