text: Liu et al. (2019b) demonstrate the beneﬁts of 
  large-batch training: with 8k examples both the 
  language model perplexity and downstream task 
  performance are improved. They also publish their 
  recommendations for other parameters. You et al. 
  (2019) report that with a batch size of 32k BERT’s 
  training time can be signiﬁcantly reduced with no 
  degradation in performance. Zhou et al. (2019) ob- 
  serve that the normalization of the trained [CLS] 
  token stabilizes the training and slightly improves 
  performance on text classiﬁcation tasks. 
  
  5 Training BERT 
  
  This section reviews the proposals to optimize the 
  training and architecture of the original BERT. 
  
  To date, the most systematic study of BERT archi- 
  tecture was performed by Wang et al. (2019b), who 
  experimented with the number of layers, heads, and 
  model parameters, varying one option and freez- 
  ing the others. They concluded that the number 
  of heads was not as signiﬁcant as the number 
  of layers. That is consistent with the ﬁndings 
  of Voita et al. (2019b) and Michel et al. (2019) 
  (section 6), and also the observation by Liu et al. 
  (2019a) that the middle layers were the most trans- 
  ferable. Larger hidden representation size was con- 