siﬁers trained to perform the bidirectional lan-
guage modeling task, given just the CWR s as in-
put. We notice that higher layers in recurrent mod-
els consistently achieve lower perplexities. Inter-
estingly, we see that layers 1 and 2 in the 4-layer
ELMo model have very similar performance—this
warrants further exploration. On the other hand,
the layers of the ELMo (transformer) model do not
exhibit such a monotonic increase. While the top-
most layer is best (which we expected, since this
is the vector originally fed into a softmax classiﬁer
during pretraining), the middle layers show vary-
ing performance. Across all models, the represen-
tations that are better-suited for language model-
ing are also those that exhibit worse probing task
performance (Figure 3), indicating that contextu-
alizer layers trade off between encoding general
and task-speciﬁc features.
These results also reveal a difference in the
layerwise behavior of LSTMs and transformers;
moving up the LSTM layers yields more task-
speciﬁc representations, but the same does not
hold for transformers. Better understanding the
differences between transformers and LSTMs is
an active area of research (Chen et al., 2018; Tang
et al., 2018), and we leave further exploration of
these observations to future work.
These observations motivate the gradual un-
freezing method of Howard and Ruder (2018),
where the model layers are progressively unfrozen
(starting from the ﬁnal layer) during the ﬁne-
tuning process. Given our observation that higher-
level LSTM layers are less general (and more pre-
training task-speciﬁc), they likely have to be ﬁne-
tuned a bit more in order to make them appropri-
ately task speciﬁc. Meanwhile, the base layer of
the LSTM already learns highly transferable fea-
tures, and may not beneﬁt from ﬁne-tuning.
6 Transferring Between Tasks
Successful pretrained contextualizers have used
self-supervised tasks such as bidirectional lan-
guage modeling (Peters et al., 2018a) and next sen-
tence prediction ( Devlin et al. , 2018), which en-
able the use of large, unannotated text corpora.
However, contextualizers can also be pretrained
on explicitly supervised objectives, as done in
pretrained sentence embedding methods ( Con-
neau et al. , 2017). To better understand how
the choice of pretraining task affects the linguis-
tic knowledge within and transferability of CWR s,
we compare pretraining on a range of different
explicitly-supervised tasks with bidirectional lan-
guage model pretraining.
Figure 4: BERT layer transferability (columns corre-
spond to probing tasks, Liu et al. (2019a).
of hierarchical sentence structure, as detected by
the probing tasks of predicting the token index, the
main auxiliary verb and the sentence subject.
There is a wide consensus in studies with differ-
ent tasks, datasets and methodologies that syntac-
tic information is most prominent in the middle
layers of BERT.4 Hewitt and Manning (2019) had
the most success reconstructing syntactic tree depth
from the middle BERT layers (6-9 for base-BERT,
14-19 for BERT-large). Goldberg (2019) reports
the best subject-verb agreement around layers 8-
9, and the performance on syntactic probing tasks
used by Jawahar et al. (2019) also seems to peak
around the middle of the model. The prominence
of syntactic information in the middle BERT layers
is related to Liu et al. (2019a)’s observation that the
middle layers of Transformers are best-performing
overall and the most transferable across tasks (see
Figure 4).
There is conﬂicting evidence about syntactic
chunks. Tenney et al. (2019a) conclude that "the
basic syntactic information appears earlier in the
network while high-level semantic features appear
at the higher layers", drawing parallels between
this order and the order of components in a typical
NLP pipeline – from POS-tagging to dependency
parsing to semantic role labeling. Jawahar et al.
(2019) also report that the lower layers were more