References
Gustavo Aguilar, Yuan Ling, Yu Zhang, Benjamin
Yao, Xing Fan, and Edward Guo. 2019. Knowl-
edge Distillation from Internal Representations.
arXiv preprint arXiv:1910.03723.
Alan Akbik, Tanja Bergmann, and Roland V oll-
graf. 2019. Pooled Contextualized Embeddings
for Named Entity Recognition. In Proceedings
of the 2019 Conference of the North Ameri-
can Chapter of the Association for Computa-
tional Linguistics: Human Language Technolo-
gies, Volume 1 (Long and Short Papers), pages
724–728, Minneapolis, Minnesota. Association
for Computational Linguistics.
Yuki Arase and Jun’ichi Tsujii. 2019. Transfer
Fine-Tuning: A BERT Case Study. In Proceed-
ings of the 2019 Conference on Empirical Meth-
ods in Natural Language Processing and the
9th International Joint Conference on Natural
Language Processing (EMNLP-IJCNLP), pages
5393–5404, Hong Kong, China. Association for
Computational Linguistics.
Ekaterina Arkhangelskaia and Sourav Dutta. 2019.
Whatcha lookin’at? DeepLIFTing BERT’s At-
tention in Question Answering. arXiv preprint
arXiv:1910.06431.
Mikel Artetxe, Sebastian Ruder, and Dani Yo-
gatama. 2019. On the Cross-lingual Trans-
ferability of Monolingual Representations.
arXiv:1911.03310 [cs].
Matthias Aßenmacher and Christian Heumann.
2020. On the comparability of Pre-Trained Lan-
guage Models. arXiv:2001.00781 [cs, stat].
Joris Baan, Maartje ter Hoeve, Marlies van der
Wees, Anne Schuth, and Maarten de Rijke.
2019. Understanding Multi-Head Attention
in Abstractive Summarization. arXiv preprint
arXiv:1911.03898.
Alexei Baevski, Sergey Edunov, Yinhan Liu, Luke
Zettlemoyer, and Michael Auli. 2019. Cloze-
driven Pretraining of Self-Attention Networks.
In Proceedings of the 2019 Conference on Em-
pirical Methods in Natural Language Process-
ing and the 9th International Joint Confer-
ence on Natural Language Processing (EMNLP-
IJCNLP), pages 5360–5369, Hong Kong, China.
Association for Computational Linguistics.
He Bai, Peng Shi, Jimmy Lin, Luchen
Tan, Kun Xiong, Wen Gao, and Ming Li.
2020. SegaBERT: Pre-training of Segment-
aware BERT for Language Understanding.
arXiv:2004.14996 [cs].
Sriram Balasubramanian, Naman Jain, Gaurav Jin-
dal, Abhijeet Awasthi, and Sunita Sarawagi.
2020. What’s in a Name? Are BERT Named En-
tity Representations just as Good for any other
Name? In Proceedings of the 5th Workshop on
Representation Learning for NLP , pages 205–
214, Online. Association for Computational Lin-
guistics.
Hangbo Bao, Li Dong, Furu Wei, Wenhui Wang,
Nan Yang, Xiaodong Liu, Yu Wang, Songhao
Piao, Jianfeng Gao, Ming Zhou, and Hsiao-
Wuen Hon. 2020. UniLMv2: Pseudo-Masked
Language Models for Uniﬁed Language Model
Pre-Training. arXiv:2002.12804 [cs].
Yonatan Belinkov and James Glass. 2019. Anal-
ysis Methods in Neural Language Processing:
A Survey. Transactions of the Association for
Computational Linguistics, 7:49–72.
Eyal Ben-David, Carmel Rabinovitz, and Roi Re-
ichart. 2020. PERL: Pivot-based Domain Adap-
tation for Pre-trained Deep Contextualized Em-
bedding Models. arXiv:2006.09075 [cs].
Rishi Bommasani, Kelly Davis, and Claire Cardie.
2020. Interpreting Pretrained Contextualized
Representations via Reductions to Static Em-
beddings. In Proceedings of the 58th Annual
Meeting of the Association for Computational
Linguistics, pages 4758–4781.
Zied Bouraoui, Jose Camacho-Collados, and
Steven Schockaert. 2019. Inducing Relational
Knowledge from BERT. arXiv:1911.12753
[cs].
Samuel Broscheit. 2019. Investigating Entity
Knowledge in BERT with Simple Neural End-
To-End Entity Linking. In Proceedings of the
23rd Conference on Computational Natural Lan-
guage Learning (CoNLL), pages 677–685, Hong
Kong, China. Association for Computational
Linguistics.
Tom B. Brown, Benjamin Mann, Nick Ryder,
Melanie Subbiah, Jared Kaplan, Prafulla Dhari-
wal, Arvind Neelakantan, Pranav Shyam, Girish