Kong, China. Association for Computational
Linguistics.
John Hewitt and Christopher D. Manning. 2019.
A Structural Probe for Finding Syntax in Word
Representations. In Proceedings of the 2019
Conference of the North American Chapter of
the Association for Computational Linguistics:
Human Language Technologies, Volume 1 (Long
and Short Papers), pages 4129–4138.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean.
2014. Distilling the Knowledge in a Neural Net-
work. In Deep Learning and Representation
Learning Workshop: NIPS 2014.
Benjamin Hoover, Hendrik Strobelt, and Sebastian
Gehrmann. 2019. exBERT: A Visual Analy-
sis Tool to Explore Learned Representations in
Transformers Models. arXiv:1910.05276 [cs].
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzeb-
ski, Bruna Morrone, Quentin de Laroussilhe, An-
drea Gesmundo, Mona Attariyan, and Sylvain
Gelly. 2019. Parameter-Efﬁcient Transfer Learn-
ing for NLP. arXiv:1902.00751 [cs, stat].
Phu Mon Htut, Jason Phang, Shikha Bordia, and
Samuel R Bowman. 2019. Do attention heads
in BERT track syntactic dependencies? arXiv
preprint arXiv:1911.12246.
Sarthak Jain and Byron C. Wallace. 2019. Atten-
tion is not Explanation. In Proceedings of the
2019 Conference of the North American Chapter
of the Association for Computational Linguis-
tics: Human Language Technologies, Volume 1
(Long and Short Papers), pages 3543–3556.
Ganesh Jawahar, Benoît Sagot, Djamé Seddah,
Samuel Unicomb, Gerardo Iñiguez, Márton Kar-
sai, Yannick Léo, Márton Karsai, Carlos Sar-
raute, Éric Fleury, et al. 2019. What does BERT
learn about the structure of language? In 57th
Annual Meeting of the Association for Computa-
tional Linguistics (ACL), Florence, Italy.
Haoming Jiang, Pengcheng He, Weizhu Chen, Xi-
aodong Liu, Jianfeng Gao, and Tuo Zhao. 2019a.
SMART: Robust and Efﬁcient Fine-Tuning for
Pre-trained Natural Language Models through
Principled Regularized Optimization. arXiv
preprint arXiv:1911.03437.
Zhengbao Jiang, Frank F. Xu, Jun Araki, and Gra-
ham Neubig. 2019b. How Can We Know What
Language Models Know? arXiv:1911.12543
[cs].
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang,
Xiao Chen, Linlin Li, Fang Wang, and Qun
Liu. 2019. TinyBERT: Distilling BERT for nat-
ural language understanding. arXiv preprint
arXiv:1909.10351.
Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter
Szolovits. 2020. Is BERT Really Robust? A
Strong Baseline for Natural Language Attack
on Text Classiﬁcation and Entailment. In AAAI
2020.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S.
Weld, Luke Zettlemoyer, and Omer Levy. 2020.
SpanBERT: Improving Pre-Training by Repre-
senting and Predicting Spans. Transactions of
the Association for Computational Linguistics,
8:64–77.
Wei-Tsung Kao, Tsung-Han Wu, Po-Han Chi,
Chun-Cheng Hsieh, and Hung-Yi Lee. 2020.
Further boosting BERT-based models by du-
plicating existing layers: Some intriguing
phenomena inside BERT. arXiv preprint
arXiv:2001.09309.
Taeuk Kim, Jihun Choi, Daniel Edmiston, and
Sang-goo Lee. 2020. Are pre-trained language
models aware of phrases? simple but strong
baselines for grammar induction. In ICLR 2020.
Goro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi,
and Kentaro Inui. 2020. Attention Module is
Not Only a Weight: Analyzing Transformers
with Vector Norms. arXiv:2004.10102 [cs].
Dan Kondratyuk and Milan Straka. 2019. 75 Lan-
guages, 1 Model: Parsing Universal Dependen-
cies Universally. In Proceedings of the 2019
Conference on Empirical Methods in Natural
Language Processing and the 9th International
Joint Conference on Natural Language Process-
ing (EMNLP-IJCNLP), pages 2779–2795, Hong
Kong, China. Association for Computational
Linguistics.
Lingpeng Kong, Cyprien de Masson d’Autume, Lei
Yu, Wang Ling, Zihang Dai, and Dani Yogatama.
2019. A mutual information maximization per-
spective of language representation learning. In