text: A big methodological challenge in the current 
  NLP is that the reported performance improve- 
  ments of new models may well be within varia- 
  tion induced by environment factors (Crane, 2018). 
  BERT is not an exception. Dodge et al. (2020) 
  report signiﬁcant variation for BERT ﬁne-tuned 
  on GLUE tasks due to both weight initialization 
  and training data order. They also propose early 
  stopping on the less-promising seeds. 
  
  In general, larger BERT models perform better 
  (Liu et al., 2019a; Roberts et al., 2020), but not 
  always: BERT-base outperformed BERT-large on 
  subject-verb agreement (Goldberg, 2019) and sen- 
  tence subject detection (Lin et al., 2019). Given 
  the complexity of language, and amounts of pre- 
  training data, it is not clear why BERT ends up with 
  redundant heads and layers. Clark et al. (2019) sug- 
  gest that one possible reason is the use of attention 
  dropouts, which causes some attention weights to 
  be zeroed-out during training. 
  
  Although we hope that the above observations 
  may be useful for the practitioners, this section 
  does not exhaust the current research on ﬁne-tuning 
  and its alternatives. For example, we do not cover 
  such topics as Siamese architectures, policy gradi- 
  ent training, automated curriculum learning, and 
  others. 