text: The functions of special tokens are not yet well 
  understood. [CLS] is typically viewed as an ag- 
  gregated sentence-level representation (although 
  all token representations also contain at least some 
  sentence-level information, as discussed in subsec- 
  tion 4.1); in that case, we may not see e.g. full 
  syntactic trees in inter-word attention because part 
  of that information is actually packed in [CLS]. 
  Clark et al. (2019) experiment with encoding 
  Wikipedia paragraphs with base BERT to consider 
  speciﬁcally the attention to special tokens, noting 
  that heads in early layers attend more to [CLS], 
  in middle layers to [SEP], and in ﬁnal layers to 
  periods and commas. They hypothesize that its 
  function might be one of "no-op", a signal to ig- 
  nore the head if its pattern is not applicable to the 
  current case. As a result, for example, [SEP] 
  gets increased attention starting in layer 5, but its 
  importance for prediction drops. However, after 
  ﬁne-tuning both [SEP] and [CLS] get a lot of 
  attention, depending on the task (Kovaleva et al., 
  2019). Interestingly, BERT also pays a lot of at- 
  tention to punctuation, which Clark et al. (2019) 
  explain by the fact that periods and commas are 
  simply almost as frequent as the special tokens, and 
  so the model might learn to rely on them for the 
  same reasons. 