text: • Alternatives to masking. Raffel et al. (2019) 
  experiment with replacing and dropping spans, 
  Lewis et al. (2019) explore deletion, inﬁlling, 
  sentence permutation and document rotation, 
  and Sun et al. (2019c) predict whether a to- 
  ken is capitalized and whether it occurs in 
  other segments of the same document. Yang 
  et al. (2019) train on different permutations 
  of word order in the input sequence, maximiz- 
  ing the probability of the original word order 
  (cf. the n-gram word order reconstruction task 
  (Wang et al., 2019a)). Clark et al. (2020) de- 
  tect tokens that were replaced by a generator 
  network rather than masked.  Although BERT is already actively used as a 
  source of world knowledge (see subsection 3.3), 
  there is also work on explicitly supplying struc- 
  One approach is entity- 
  tured knowledge. 
  enhanced models. For example, Peters et al. 
  (2019a); Zhang et al. (2019) include entity em- 