5.2 Improvements to the training regime
Liu et al. (2019b) demonstrate the beneﬁts of
large-batch training: with 8k examples both the
language model perplexity and downstream task
performance are improved. They also publish their
recommendations for other parameters. You et al.
(2019) report that with a batch size of 32k BERT’s
training time can be signiﬁcantly reduced with no
degradation in performance. Zhou et al. (2019) ob-
serve that the normalization of the trained [CLS]
token stabilizes the training and slightly improves
performance on text classiﬁcation tasks.
Gong et al. (2019) note that, since self-attention
patterns in higher and lower layers are similar, the
model training can be done in a recursive man-
ner, where the shallower version is trained ﬁrst and
then the trained parameters are copied to deeper
layers. Such a "warm-start" can lead to a 25% faster
training without sacriﬁcing performance.