text: • NSP alternatives. Removing NSP does not 
  hurt or slightly improves performance (Liu 
  et al., 2019b; Joshi et al., 2020; Clinchant 
  et al., 2019). Wang et al. (2019a) and Cheng 
  et al. (2019) replace NSP with the task of 
  predicting both the next and the previous sen- 
  tences. Lan et al. (2020a) replace the negative 
  NSP examples by swapped sentences from 
  positive examples, rather than sentences from 
  different documents. ERNIE 2.0 includes sen- 
  tence reordering and sentence distance pre- 
  diction. Bai et al. (2020) replace both NSP 
  and token position embeddings by a combina- 
  tion of paragraph, sentence, and token index 
  embeddings. Li and Choi (2020) experiment 
  with utterance order prediction task for multi- 
  party dialogue (and also MLM at the level of 
  utterances and the whole dialogue). 
  
  • How to mask. Raffel et al. (2019) systemati- 
  cally experiment with corruption rate and cor- 
  rupted span length. Liu et al. (2019b) propose 
  diverse masks for training examples within 
  an epoch, while Baevski et al. (2019) mask 
  every token in a sequence instead of a random 
  selection. Clinchant et al. (2019) replace the 
  MASK token with [UNK] token, to help the 
  model learn a representation for unknowns 
  that could be useful for translation. Song et al. 
  (2020) maximize the amount of information 
  available to the model by conditioning on both 
  masked and unmasked tokens, and letting the 
  model see how many tokens are missing. 
  • What to mask. Masks can be applied to full 
  words instead of word-pieces (Devlin et al., 
  2019; Cui et al., 2019). Similarly, we can 
  mask spans rather than single tokens (Joshi 
  et al., 2020), predicting how many are missing 
  (Lewis et al., 2019). Masking phrases and 
  named entities (Sun et al., 2019b) improves 
  representation of structured knowledge. 
  
  • Other tasks. Sun et al. (2019c) propose si- 
  multaneous learning of 7 tasks, including dis- 
  course relation classiﬁcation and predicting 
  whether a segment is relevant for IR. Guu 
  et al. (2020) include a latent knowledge re- 
  triever in language model pretraining. Wang 
  et al. (2020c) combine MLM with knowledge 
  base completion objective. Glass et al. (2020) 
  replace MLM with span prediction task (as 
  in extractive question answering), where the 
  model is expected to provide the answer not 
  from its own weights, but from a different pas- 
  sage containing the correct answer (a relevant 
  search engine query snippet). 
  
  • Where to mask. Lample and Conneau (2019) 
  use arbitrary text streams instead of sentence 
  pairs and subsample frequent outputs similar 
  to Mikolov et al. (2013). Bao et al. (2020) 
  combine the standard autoencoding MLM 
  with partially autoregressive LM objective us- 
  ing special pseudo mask tokens. 
  
  Another obvious source of improvement is pre- 
  training data. Several studies explored the ben- 
  eﬁts of increasing the corpus volume (Liu et al., 
  2019b; Conneau et al., 2019; Baevski et al., 2019) 
  and longer training (Liu et al., 2019b). The data 
  also does not have to be raw text: there is a num- 
  ber efforts to incorporate explicit linguistic in- 
  formation, both syntactic (Sundararaman et al., 
  2019) and semantic (Zhang et al., 2020). Wu et al. 
  (2019b) and Kumar et al. (2020) include the label 
  for a given sequence from an annotated task dataset. 
  Schick and Schütze (2020) separately learn repre- 
  sentations for rare words. 