cessful at word sense disambiguation task. How-
ever, Mickus et al. (2019) note thatthe representa-
tions of the same word depend on the position
of the sentence in which it occurs , likely due to
the NSP objective. This is not desirable from the
linguistic point of view, and could be a promising
3V oita et al. (2019a) look at the evolution of token embed-
dings, showing that in the earlier Transformer layers, MLM
forces the acquisition of contextual information at the expense
of the token identity, which gets recreated in later layers.