text: Out-of-the-box BERT is surprisingly brittle to 
  named entity replacements: e.g. replacing names 
  in the coreference task changes 85% of predictions 
  (Balasubramanian et al., 2020). This suggests that 
  the model does not actually form a generic idea of 
  named entities, although its F1 scores on NER prob- 
  ing tasks are high (Tenney et al., 2019a). Broscheit 
  (2019) ﬁnd that ﬁne-tuning BERT on Wikipedia 
  entity linking "teaches" it additional entity knowl- 
  edge, which would suggest that it did not absorb all 
  the relevant entity information during pre-training 
  on Wikipedia. 
  
  3.4 Limitations 
  
  Multiple probing studies in section 3 and section 4 
  report that BERT possesses a surprising amount of 
  syntactic, semantic, and world knowledge. How- 
  ever, Tenney et al. (2019a) remarks, “the fact that 
  a linguistic pattern is not observed by our probing 
  classiﬁer does not guarantee that it is not there, and 
  the observation of a pattern does not tell us how it 
  is used." There is also the issue of how complex a 
  probe should be allowed to be (Liu et al., 2019a). If 
  a more complex probe recovers more information, 
  to what extent are we still relying on the original 
  model? 