text: • Taking more layers into account: learning 
  a complementary representation of the infor- 
  mation in deep and output layers (Yang and 
  Zhao, 2019), using a weighted combination 
  of all layers instead of the ﬁnal one (Su and 
  Cheng, 2019; Kondratyuk and Straka, 2019), 
  and layer dropout (Kondratyuk and Straka, 
  2019). 
  
  • Two-stage ﬁne-tuning introduces an inter- 
  mediate supervised training stage between 
  pre-training and ﬁne-tuning (Phang et al., 
  2019; Garg et al., 2020; Arase and Tsujii, 
  2019; Pruksachatkun et al., 2020; Glavaš and 
  Vuli´c, 2020). Ben-David et al. (2020) propose 
  a pivot-based variant of MLM to ﬁne-tune 
  BERT for domain adaptation. 
  
  • Adversarial token perturbations improve 
  robustness of the model (Zhu et al., 2019). 
  • Adversarial regularization in combination 
  with Bregman Proximal Point Optimization 
  helps alleviate pre-trained knowledge forget- 
  ting and therefore prevents BERT from overﬁt- 
  ting to downstream tasks (Jiang et al., 2019a). 
  • Mixout regularization improves the stability 
  of BERT ﬁne-tuning even for a small number 
  of training examples (Lee et al., 2019). 