text: Pre-training is the most expensive part of train- 
  ing BERT, and it would be informative to know 
  how much beneﬁt it provides. On some tasks, a 
  randomly initialized and ﬁne-tuned BERT obtains 
  competitive or higher results than the pre-trained 
  BERT with the task classiﬁer and frozen weights 
  (Kovaleva et al., 2019). The consensus in the com- 
  munity is that pre-training does help in most situa- 
  tions, but the degree and its exact contribution re- 
  quires further investigation. Prasanna et al. (2020) 
  found that most weights of pre-trained BERT are 
  useful in ﬁne-tuning, although there are "better" 
  and "worse" subnetworks. One explanation is that 
  pre-trained weights help the ﬁne-tuned BERT ﬁnd 
  wider and ﬂatter areas with smaller generalization 
  error, which makes the model more robust to over- 
  ﬁtting (see Figure 5 from Hao et al. (2019)). 
  
  Given the large number and variety of proposed 
  modiﬁcations, one would wish to know how much 
  impact each of them has. However, due to the 
  overall trend towards large model sizes, systematic 
  ablations have become expensive. Most new mod- 
  els claim superiority on standard benchmarks, but 
  gains are often marginal, and estimates of model 
  stability and signiﬁcance testing are very rare. 
  
  With large models, even ﬁne-tuning becomes ex- 
  pensive, but Houlsby et al. (2019) show that it can 
  
  5Kondratyuk and Straka (2019) suggest that ﬁne-tuning on 
  Universal Dependencies does result in syntactically meaning- 
  ful attention patterns, but there was no quantitative evaluation. 