ing directions are the "amnesic probing" (Elazar
et al., 2020), identifying features important for pre-
diction for a given task (Arkhangelskaia and Dutta,
2019), and pruning the model to remove the non-
important components (V oita et al., 2019b; Michel
et al., 2019; Prasanna et al., 2020).
8 Conclusion
In a little over a year, BERT has become a ubiq-
uitous baseline in NLP experiments and inspired
numerous studies analyzing the model and propos-
ing various improvements. The stream of papers
seems to be accelerating rather than slowing down,
and we hope that this survey helps the community
to focus on the biggest unresolved questions.
9 Acknowledgements
We thank the anonymous reviewers for their valu-
able feedback. This work is funded in part by
the NSF award number IIS-1844740 to Anna
Rumshisky.