More recently, Kobayashi et al. (2020) showed
that the norms of attention-weighted input vec-
tors, which yield a more intuitive interpretation
of self-attention, reduce the attention to special to-
kens. However, even when the attention weights
are normed, it is still not the case that most heads
that do the "heavy lifting" are even potentially in-
terpretable (Prasanna et al., 2020).
One methodological choice in in many studies
of attention is to focus on inter-word attention and
simply exclude special tokens (e.g. Lin et al. (2019)
and Htut et al. (2019)). However, if attention to
special tokens actually matters at inference time,
drawing conclusions purely from inter-word atten-
tion patterns does not seem warranted.
The functions of special tokens are not yet well
understood. [CLS] is typically viewed as an ag-
gregated sentence-level representation (although
all token representations also contain at least some
sentence-level information, as discussed in subsec-
tion 4.1); in that case, we may not see e.g. full
syntactic trees in inter-word attention because part
of that information is actually packed in [CLS].
Clark et al. (2019) experiment with encoding
Wikipedia paragraphs with base BERT to consider
speciﬁcally the attention to special tokens, noting
that heads in early layers attend more to [CLS],
in middle layers to [SEP], and in ﬁnal layers to
periods and commas. They hypothesize that its
function might be one of "no-op", a signal to ig-
nore the head if its pattern is not applicable to the
current case. As a result, for example, [SEP]
gets increased attention starting in layer 5, but its
importance for prediction drops. However, after
ﬁne-tuning both [SEP] and [CLS] get a lot of
attention, depending on the task (Kovaleva et al.,
2019). Interestingly, BERT also pays a lot of at-
tention to punctuation, which Clark et al. (2019)
explain by the fact that periods and commas are
simply almost as frequent as the special tokens, and
so the model might learn to rely on them for the
same reasons.
4.3 BERT layers
The ﬁrst layer of BERT receives as input a combina-
tion of token, segment, and positional embeddings.
It stands to reason that the lower layers have
the most information about linear word order.
Lin et al. (2019) report a decrease in the knowledge
of linear word order around layer 4 in BERT-base.
This is accompanied by an increased knowledge
(a) ELMo (original)
Layer 0
Layer 2
(b) ELMo (4-layer)
Layer 0
Layer 4
(c) ELMo (transformer)
Layer 0
Layer 6
(d) OpenAI transformer
Layer 0
Layer 12
(e) BERT (base, cased)
Layer 0
Layer 12
(f) BERT (large, cased)
Layer 0
Layer 24
Lower Performance Higher Performance
Figure 3: A visualization of layerwise patterns in task
performance. Each column represents a probing task,
and each row represents a contextualizer layer.
textualizers. Furthermore, the ELMo-based mod-
els facilitate a controlled comparison—they only
differ in the contextualizer architecture used.
We evaluate how well CWR features perform
the pretraining task—bidirectional language mod-
eling. Speciﬁcally, we take the pretrained repre-
sentations for each layer and relearn the language
model softmax classiﬁers used to predict the next
and previous token. The ELMo models are trained
on the Billion Word Benchmark, so we retrain
the softmax classiﬁer on similar data to mitigate
any possible effects from domain shift. We split
the held-out portion of the Billion Word Bench-
mark into train (80%, 6.2M tokens) and evalua-
tion (20%, 1.6M tokens) sets and use this data to
retrain and evaluate the softmax classiﬁers. We
expect that biLM perplexity will be lower when
training the softmax classiﬁers on representations
from layers that capture more information about
the pretraining task.
5.2 Results and Discussion
Figure 4 presents the performance of softmax clas-
siﬁers trained to perform the bidirectional lan-
guage modeling task, given just the CWR s as in-
put. We notice that higher layers in recurrent mod-