Pre-Training for Deep Language Understanding.
arXiv:1908.04577 [cs].
Wenhui Wang, Furu Wei, Li Dong, Hangbo
Bao, Nan Yang, and Ming Zhou. 2020b.
MiniLM: Deep Self-Attention Distillation for
Task-Agnostic Compression of Pre-Trained
Transformers. arXiv preprint arXiv:2002.10957.
Xiaozhi Wang, Tianyu Gao, Zhaocheng Zhu,
Zhiyuan Liu, Juanzi Li, and Jian Tang. 2020c.
KEPLER: A Uniﬁed Model for Knowledge Em-
bedding and Pre-trained Language Representa-
tion. arXiv:1911.06136 [cs].
Yile Wang, Leyang Cui, and Yue Zhang. 2020d.
How Can BERT Help Lexical Semantics Tasks?
arXiv:1911.02929 [cs].
Zihan Wang, Stephen Mayhew, Dan Roth, et al.
2019b. Cross-Lingual Ability of Multilingual
BERT: An Empirical Study. arXiv preprint
arXiv:1912.07840.
Alex Warstadt and Samuel R. Bowman. 2020. Can
neural networks acquire a structural bias from
raw linguistic data? In Proceedings of the 42nd
Annual Virtual Meeting of the Cognitive Science
Society, Online.
Alex Warstadt, Yu Cao, Ioana Grosu, Wei Peng,
Hagen Blix, Yining Nie, Anna Alsop, Shikha
Bordia, Haokun Liu, Alicia Parrish, et al. 2019.
Investigating BERT’s Knowledge of Language:
Five Analysis Methods with NPIs. In Proceed-
ings of the 2019 Conference on Empirical Meth-
ods in Natural Language Processing and the
9th International Joint Conference on Natural
Language Processing (EMNLP-IJCNLP), pages
2870–2880.
Gregor Wiedemann, Steffen Remus, Avi Chawla,
and Chris Biemann. 2019. Does BERT Make
Any Sense? Interpretable Word Sense Dis-
ambiguation with Contextualized Embeddings.
arXiv preprint arXiv:1909.10430.
Sarah Wiegreffe and Yuval Pinter. 2019. Atten-
tion is not not Explanation. In Proceedings of
the 2019 Conference on Empirical Methods in
Natural Language Processing and the 9th In-
ternational Joint Conference on Natural Lan-
guage Processing (EMNLP-IJCNLP), pages 11–
20, Hong Kong, China. Association for Compu-
tational Linguistics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien
Chaumond, Clement Delangue, Anthony Moi,
Pierric Cistac, Tim Rault, Rémi Louf, Morgan
Funtowicz, and Jamie Brew. 2020. Hugging-
Face’s Transformers: State-of-the-Art Natural
Language Processing. arXiv:1910.03771 [cs].
Felix Wu, Angela Fan, Alexei Baevski, Yann
Dauphin, and Michael Auli. 2019a. Pay Less At-
tention with Lightweight and Dynamic Convolu-
tions. In International Conference on Learning
Representations.
Xing Wu, Shangwen Lv, Liangjun Zang, Jizhong
Han, and Songlin Hu. 2019b. Conditional BERT
Contextual Augmentation. In ICCS 2019: Com-
putational Science – ICCS 2019 , pages 84–95.
Springer.
Yonghui Wu, Mike Schuster, Zhifeng Chen,
Quoc V Le, Mohammad Norouzi, Wolfgang
Macherey, Maxim Krikun, Yuan Cao, Qin Gao,
Klaus Macherey, et al. 2016. Google’s Neural
Machine Translation System: Bridging the Gap
between Human and Machine Translation. arXiv
preprint arXiv:1609.08144.
Zhiyong Wu, Yun Chen, Ben Kao, and Qun Liu.
2020. Perturbed Masking: Parameter-free Prob-
ing for Analyzing and Interpreting BERT. In
Proceedings of the 58th Annual Meeting of
the Association for Computational Linguistics,
pages 4166–4176, Online. Association for Com-
putational Linguistics.
Canwen Xu, Wangchunshu Zhou, Tao Ge, Furu
Wei, and Ming Zhou. 2020. BERT-of-Theseus:
Compressing BERT by Progressive Module Re-
placing. arXiv preprint arXiv:2002.02925.
Junjie Yang and Hai Zhao. 2019. Deepening Hid-
den Representations from Pre-Trained Language
Models for Natural Language Understanding.
arXiv:1911.01940 [cs].
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime
Carbonell, Ruslan Salakhutdinov, and Quoc V .
Le. 2019. XLNet: Generalized Autoregres-
sive Pretraining for Language Understanding.
arXiv:1906.08237 [cs].
Pengcheng Yin, Graham Neubig, Wen-tau Yih, and
Sebastian Riedel. 2020. TaBERT: Pretraining
for Joint Understanding of Textual and Tabular