and longer training (Liu et al., 2019b). The data
also does not have to be raw text: there is a num-
ber efforts to incorporate explicit linguistic in-
formation, both syntactic (Sundararaman et al.,
2019) and semantic (Zhang et al., 2020). Wu et al.
(2019b) and Kumar et al. (2020) include the label
for a given sequence from an annotated task dataset.
Schick and Schütze (2020) separately learn repre-
sentations for rare words.
Although BERT is already actively used as a
source of world knowledge (see subsection 3.3),
there is also work on explicitly supplying struc-
tured knowledge . One approach is entity-
enhanced models. For example, Peters et al.
(2019a); Zhang et al. (2019) include entity em-