On Reading-comprehension. arXiv preprint
arXiv:1912.06638.
Shubham Toshniwal, Haoyue Shi, Bowen Shi,
Lingyu Gao, Karen Livescu, and Kevin Gim-
pel. 2020. A Cross-Task Analysis of Text Span
Representations. In Proceedings of the 5th Work-
shop on Representation Learning for NLP, pages
166–176, Online. Association for Computational
Linguistics.
Henry Tsai, Jason Riesa, Melvin Johnson, Naveen
Arivazhagan, Xin Li, and Amelia Archer. 2019.
Small and Practical BERT Models for Sequence
Labeling. arXiv preprint arXiv:1909.00100.
Iulia Turc, Ming-Wei Chang, Kenton Lee, and
Kristina Toutanova. 2019. Well-Read Students
Learn Better: The Impact of Student Initializa-
tion on Knowledge Distillation. arXiv preprint
arXiv:1908.08962.
Marten van Schijndel, Aaron Mueller, and Tal
Linzen. 2019. Quantity doesn’t buy quality syn-
tax with neural language models. In Proceed-
ings of the 2019 Conference on Empirical Meth-
ods in Natural Language Processing and the
9th International Joint Conference on Natural
Language Processing (EMNLP-IJCNLP), pages
5831–5837, Hong Kong, China. Association for
Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar,
Jakob Uszkoreit, Llion Jones, Aidan N Gomez,
Łukasz Kaiser, and Illia Polosukhin. 2017. At-
tention is All you Need. In Advances in neu-
ral information processing systems, pages 5998–
6008.
Jesse Vig. 2019. Visualizing Attention in
Transformer-Based Language Representation
Models. arXiv:1904.02679 [cs, stat].
Jesse Vig and Yonatan Belinkov. 2019. Analyzing
the Structure of Attention in a Transformer Lan-
guage Model. In Proceedings of the 2019 ACL
Workshop BlackboxNLP: Analyzing and Inter-
preting Neural Networks for NLP, pages 63–76,
Florence, Italy. Association for Computational
Linguistics.
David Vilares, Michalina Strzyz, Anders Søgaard,
and Carlos Gómez-Rodríguez. 2020. Parsing as
pretraining. In Thirty-Fourth AAAI Conference
on Artiﬁcial Intelligence (AAAI-20).
Elena V oita, Rico Sennrich, and Ivan Titov. 2019a.
The Bottom-up Evolution of Representations in
the Transformer: A Study with Machine Transla-
tion and Language Modeling Objectives. In Pro-
ceedings of the 2019 Conference on Empirical
Methods in Natural Language Processing and
the 9th International Joint Conference on Nat-
ural Language Processing (EMNLP-IJCNLP),
pages 4387–4397.
Elena V oita, David Talbot, Fedor Moiseev, Rico
Sennrich, and Ivan Titov. 2019b. Analyzing
Multi-Head Self-Attention: Specialized Heads
Do the Heavy Lifting, the Rest Can Be Pruned.
arXiv preprint arXiv:1905.09418.
Elena V oita and Ivan Titov. 2020. Information-
Theoretic Probing with Minimum Description
Length. arXiv:2003.12298 [cs].
Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gard-
ner, and Sameer Singh. 2019a. Universal Ad-
versarial Triggers for Attacking and Analyzing
NLP. In Proceedings of the 2019 Conference
on Empirical Methods in Natural Language Pro-
cessing and the 9th International Joint Confer-
ence on Natural Language Processing (EMNLP-
IJCNLP), pages 2153–2162, Hong Kong, China.
Association for Computational Linguistics.
Eric Wallace, Yizhong Wang, Sujian Li, Sameer
Singh, and Matt Gardner. 2019b. Do NLP Mod-
els Know Numbers? Probing Numeracy in Em-
beddings. arXiv preprint arXiv:1909.07940.
Alex Wang, Amapreet Singh, Julian Michael, Felix
Hill, Omer Levy, and Samuel R. Bowman. 2018.
GLUE: A Multi-Task Benchmark and Analysis
Platform for Natural Language Understanding.
In Proceedings of the 2018 EMNLP Workshop
BlackboxNLP: Analyzing and Interpreting Neu-
ral Networks for NLP, pages 353–355, Brussels,
Belgium. Association for Computational Lin-
guistics.
Ruize Wang, Duyu Tang, Nan Duan, Zhongyu
Wei, Xuanjing Huang, Jianshu ji, Guihong Cao,
Daxin Jiang, and Ming Zhou. 2020a. K-Adapter:
Infusing Knowledge into Pre-Trained Models
with Adapters. arXiv:2002.01808 [cs].
Wei Wang, Bin Bi, Ming Yan, Chen Wu, Zuyi
Bao, Liwei Peng, and Luo Si. 2019a. Struct-
BERT: Incorporating Language Structures into