text: layers are more transferable (Liu et al., 2019a). In 
  ﬁne-tuning, it explains why the ﬁnal layers change 
  the most (Kovaleva et al., 2019), and why restoring 
  the weights of lower layers of ﬁne-tuned BERT to 
  their original values does not dramatically hurt the 
  model performance (Hao et al., 2019). 
  
  sistently better, but the gains varied by setting. 
  
  All in all, changes in the number of heads 
  and layers appear to perform different func- 
  tions. The issue of model depth must be related to 
  the information ﬂow from the most task-speciﬁc 
  layers closer to the classiﬁer (Liu et al., 2019a), 
  to the initial layers which appear to be the most 
  task-invariant (Hao et al., 2019), and where the 
  tokens resemble the input tokens the most (Brun- 
  ner et al., 2020) (see subsection 4.3). If that is the 
  case, a deeper model has more capacity to encode 
  information that is not task-speciﬁc. 
  
  Tenney et al. (2019a) suggest that while syntactic 
  information appears early in the model and can be 
  localized, semantics is spread across the entire 
  model, which explains why certain non-trivial ex- 
  amples get solved incorrectly at ﬁrst but correctly 
  at the later layers. This is rather to be expected: 
  semantics permeates all language, and linguists de- 
  bate whether meaningless structures can exist at 
  all (Goldberg, 2006, p.166-182). But this raises 
  the question of what stacking more Transformer 
  layers in BERT actually achieves in terms of the 
  spread of semantic knowledge, and whether that 
  is beneﬁcial. Tenney et al. compared BERT-base 
  and BERT-large, and found that the overall pattern 
  of cumulative score gains is the same, only more 
  spread out in the larger model. 