5.3 Pre-training BERT
The original BERT is a bidirectional Transformer
pre-trained on two tasks: next sentence prediction
(NSP) and masked language model (MLM) (sec-
tion 2). Multiple studies have come up with alter-
native training objectives to improve on BERT,
which could be categorized as follows:
• How to mask. Raffel et al. (2019) systemati-
cally experiment with corruption rate and cor-
rupted span length. Liu et al. (2019b) propose
diverse masks for training examples within
an epoch, while Baevski et al. (2019) mask
every token in a sequence instead of a random
selection. Clinchant et al. (2019) replace the
MASK token with [UNK] token, to help the
model learn a representation for unknowns
that could be useful for translation. Song et al.
(2020) maximize the amount of information
available to the model by conditioning on both
masked and unmasked tokens, and letting the
model see how many tokens are missing.
• What to mask. Masks can be applied to full
words instead of word-pieces (Devlin et al.,
2019; Cui et al., 2019). Similarly, we can
mask spans rather than single tokens (Joshi
et al., 2020), predicting how many are missing
(Lewis et al., 2019). Masking phrases and
named entities (Sun et al., 2019b) improves
representation of structured knowledge.
• Where to mask. Lample and Conneau (2019)
use arbitrary text streams instead of sentence
pairs and subsample frequent outputs similar
to Mikolov et al. (2013). Bao et al. (2020)
combine the standard autoencoding MLM
with partially autoregressive LM objective us-
ing special pseudo mask tokens.
• Alternatives to masking. Raffel et al. (2019)
experiment with replacing and dropping spans,
Lewis et al. (2019) explore deletion, inﬁlling,
sentence permutation and document rotation,
and Sun et al. (2019c) predict whether a to-
ken is capitalized and whether it occurs in
other segments of the same document. Yang
et al. (2019) train on different permutations
of word order in the input sequence, maximiz-
ing the probability of the original word order
(cf. the n-gram word order reconstruction task
(Wang et al., 2019a)). Clark et al. (2020) de-
tect tokens that were replaced by a generator
network rather than masked.
• NSP alternatives. Removing NSP does not
hurt or slightly improves performance (Liu
et al., 2019b; Joshi et al., 2020; Clinchant
et al., 2019). Wang et al. (2019a) and Cheng
et al. (2019) replace NSP with the task of
predicting both the next and the previous sen-
tences. Lan et al. (2020a) replace the negative
NSP examples by swapped sentences from
positive examples, rather than sentences from
different documents. ERNIE 2.0 includes sen-
tence reordering and sentence distance pre-
diction. Bai et al. (2020) replace both NSP
and token position embeddings by a combina-
tion of paragraph, sentence, and token index
embeddings. Li and Choi (2020) experiment
with utterance order prediction task for multi-
party dialogue (and also MLM at the level of
utterances and the whole dialogue).
• Other tasks. Sun et al. (2019c) propose si-
multaneous learning of 7 tasks, including dis-
course relation classiﬁcation and predicting
whether a segment is relevant for IR. Guu
et al. (2020) include a latent knowledge re-
triever in language model pretraining. Wang
et al. (2020c) combine MLM with knowledge
base completion objective. Glass et al. (2020)
replace MLM with span prediction task (as
in extractive question answering), where the
model is expected to provide the answer not
from its own weights, but from a different pas-
sage containing the correct answer (a relevant
search engine query snippet).
Another obvious source of improvement is pre-
training data. Several studies explored the ben-
eﬁts of increasing the corpus volume (Liu et al.,
2019b; Conneau et al., 2019; Baevski et al., 2019)
and longer training (Liu et al., 2019b). The data
also does not have to be raw text: there is a num-
ber efforts to incorporate explicit linguistic in-
formation, both syntactic (Sundararaman et al.,