text: On the other head, many self-attention heads 
  in vanilla BERT seem to naturally learn the same 
  patterns (Kovaleva et al., 2019). This explains 
  why pruning them does not have too much impact. 
  The question that arises from this is how far we 
  could get with intentionally encouraging diverse 
  self-attention patterns: 
  theoretically, this would 
  mean increasing the amount of information in the 
  model with the same number of weights. Raganato 
  et al. (2020) show for Transformer-based machine 
  translation we can simply pre-set the patterns that 
  we already know the model would learn, instead of 
  learning them from scratch. 
  
  Note that Tenney et al. (2019a)’s experiments 
  concern sentence-level semantic relations; Cui et al. 
  (2020) report that the encoding of ConceptNet se- 
  mantic relations is the worst in the early layers and 
  increases towards the top. Jawahar et al. (2019) 
  place "surface features in lower layers, syntactic 
  features in middle layers and semantic features in 
  higher layers", but their conclusion is surprising, 
  given that only one semantic task in this study actu- 
  ally topped at the last layer, and three others peaked 
  around the middle and then considerably degraded 
  by the ﬁnal layers. 
  
  Vanilla BERT is symmetric and balanced in 
  terms of self-attention and feed-forward layers, but 
  it may not have to be. For the base Transformer, 
  Press et al. (2020) report beneﬁts from more self- 
  attention sublayers at the bottom and more feedfor- 
  ward sublayers at the top. 
  
  5.2  Improvements to the training regime 