this order and the order of components in a typical
NLP pipeline – from POS-tagging to dependency
parsing to semantic role labeling. Jawahar et al.
(2019) also report that the lower layers were more
useful for chunking, while middle layers were more
useful for parsing. At the same time, the probing
experiments by Liu et al. (2019a) ﬁnd the opposite:
both POS-tagging and chunking were performed
best at the middle layers, in both BERT-base and
BERT-large. However, all three studies use differ-
ent suites of probing tasks.
The ﬁnal layers of BERT are the most task-
speciﬁc. In pre-training, this means speciﬁcity to
the MLM task, which explains why the middle
4These BERT results are also compatible with ﬁndings by
Vig and Belinkov (2019), who report the highest attention to
tokens in dependency relations in the middle layers of GPT-2.