Soﬁa Serrano and Noah A. Smith. 2019. Is Atten-
tion Interpretable? arXiv:1906.03731 [cs].
Sheng Shen, Zhen Dong, Jiayu Ye, Linjian Ma,
Zhewei Yao, Amir Gholami, Michael W Ma-
honey, and Kurt Keutzer. 2019. Q-BERT: Hes-
sian Based Ultra Low Precision Quantization of
BERT. arXiv preprint arXiv:1909.05840.
Chenglei Si, Shuohang Wang, Min-Yen Kan, and
Jing Jiang. 2019a. What does BERT learn
from multiple-choice reading comprehension
datasets? arXiv preprint arXiv:1910.12391.
Chenglei Si, Shuohang Wang, Min-Yen Kan, and
Jing Jiang. 2019b. What does BERT Learn
from Multiple-Choice Reading Comprehension
Datasets? arXiv:1910.12391 [cs].
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and
Tie-Yan Liu. 2020. MPNet: Masked and Per-
muted Pre-training for Language Understanding.
arXiv:2004.09297 [cs].
Asa Cooper Stickland and Iain Murray. 2019.
BERT and PALs: Projected Attention Layers for
Efﬁcient Adaptation in Multi-Task Learning. In
International Conference on Machine Learning,
pages 5986–5995.
Emma Strubell, Ananya Ganesh, and Andrew Mc-
Callum. 2019. Energy and Policy Considera-
tions for Deep Learning in NLP. In ACL 2019.
Ta-Chun Su and Hsiang-Chih Cheng. 2019.
SesameBERT: Attention for Anywhere.
arXiv:1910.03176 [cs].
Saku Sugawara, Pontus Stenetorp, Kentaro Inui,
and Akiko Aizawa. 2020. Assessing the Bench-
marking Capacity of Machine Reading Compre-
hension Datasets. In AAAI.
Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu.
2019a. Patient Knowledge Distillation for BERT
Model Compression. In Proceedings of the 2019
Conference on Empirical Methods in Natural
Language Processing and the 9th International
Joint Conference on Natural Language Process-
ing (EMNLP-IJCNLP), pages 4314–4323.
Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng,
Xuyi Chen, Han Zhang, Xin Tian, Danxiang
Zhu, Hao Tian, and Hua Wu. 2019b. ERNIE:
Enhanced Representation through Knowledge
Integration. arXiv:1904.09223 [cs].
Yu Sun, Shuohuan Wang, Yukun Li, Shikun
Feng, Hao Tian, Hua Wu, and Haifeng Wang.
2019c. ERNIE 2.0: A Continual Pre-
Training Framework for Language Understand-
ing. arXiv:1907.12412 [cs].
Zhiqing Sun, Hongkun Yu, Xiaodan Song, Ren-
jie Liu, Yiming Yang, and Denny Zhou. 2020.
MobileBERT: Task-Agnostic Compression of
BERT for Resource Limited Devices.
Dhanasekar Sundararaman, Vivek Subramanian,
Guoyin Wang, Shijing Si, Dinghan Shen, Dong
Wang, and Lawrence Carin. 2019. Syntax-
Infused Transformer and BERT models for Ma-
chine Translation and Natural Language Under-
standing. arXiv:1911.06156 [cs, stat].
Alon Talmor, Yanai Elazar, Yoav Goldberg,
and Jonathan Berant. 2019. oLMpics – On
what Language Model Pre-Training Captures.
arXiv:1912.13283 [cs].
Hirotaka Tanaka, Hiroyuki Shinnou, Rui Cao, Jing
Bai, and Wen Ma. 2020. Document Classiﬁca-
tion by Word Embeddings of BERT. In Compu-
tational Linguistics, Communications in Com-
puter and Information Science, pages 145–154,
Singapore. Springer.
Raphael Tang, Yao Lu, Linqing Liu, Lili Mou,
Olga Vechtomova, and Jimmy Lin. 2019. Dis-
tilling Task-Speciﬁc Knowledge from BERT
into Simple Neural Networks. arXiv preprint
arXiv:1903.12136.
Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019a.
BERT Rediscovers the Classical NLP Pipeline.
In Proceedings of the 57th Annual Meeting of
the Association for Computational Linguistics,
pages 4593–4601.
Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang,
Adam Poliak, R. Thomas McCoy, Najoung Kim,
Benjamin Van Durme, Samuel R. Bowman, Di-
panjan Das, and Ellie Pavlick. 2019b. What do
you learn from context? Probing for sentence
structure in contextualized word representations.
In International Conference on Learning Repre-
sentations.
James Yi Tian, Alexander P Kreuzer, Pai-Hung
Chen, and Hans-Martin Will. 2019. WaL-
DORf: Wasteless Language-model Distillation