text: avenue for future work. 
  
  The above discussion concerns token embed- 
  dings, but BERT is typically used as a sentence or 
  text encoder. The standard way to generate sen- 
  tence or text representations for classiﬁcation is 
  to use the [CLS] token, but alternatives are also 
  being discussed, including concatenation of token 
  representations (Tanaka et al., 2020), normalized 
  mean (Tanaka et al., 2020), and layer activations 
  (Ma et al., 2019). See Toshniwal et al. (2020) for a 
  systematic comparison of several methods across 
  tasks and sentence encoders. 
  
  Lin et al. (2019) present evidence that atten- 
  tion weights are weak indicators of subject- 
  verb agreement and reﬂexive anaphora. Instead 
  of serving as strong pointers between tokens that 
  should be related, BERT’s self-attention weights 
  were close to a uniform attention baseline, but there 
  was some sensitivity to different types of distrac- 
  tors coherent with psycholinguistic data. This is 
  consistent with conclusions by Ettinger (2019). 
  4.2 Self-attention heads 