text: then check which of them survive the pruning, ﬁnd- 
  ing that the syntactic and positional heads are the 
  last ones to go. For BERT, Prasanna et al. (2020) 
  go in the opposite direction: pruning on the basis of 
  importance scores, and interpreting the remaining 
  "good" subnetwork. With respect to self-attention 
  heads speciﬁcally, it does not seem to be the case 
  that only the heads that potentially encode non- 
  trivial linguistic patterns survive the pruning. 
  
  the data, we have no reason to expect BERT to not 
  learn it. But harder datasets that cannot be resolved 
  with shallow heuristics are unlikely to emerge if 
  their development is not as valued as modeling 
  work. 
  
  Benchmarks for the full range of linguistic 
  competence. While the language models seem to 
  acquire a great deal of knowledge about language, 
  we do not currently have comprehensive stress tests 
  for different aspects of linguistic knowledge. A 
  step in this direction is the "Checklist" behavioral 
  testing (Ribeiro et al., 2020), the best paper at ACL 
  2020. Ideally, such tests would measure not only 
  errors, but also sensitivity (Ettinger, 2019). 