text: The original BERT paper [39] showed the effectiveness of the 
  architecture and the pretraining technique by evaluating on the 
  General Language Understanding Evaluation (GLUE) benchmark 
  [138], the Stanford Question Answering Datasets (SQuAD 1.1 and 
  2.0) [108], and the Situations With Adversarial Generations bench- 
  mark (SWAG) [155], all datasets designed to test language under- 
  standing and/or commonsense reasoning. BERT posted state of 
  the art results on all of these tasks, and the authors conclude by 
  saying that “unsupervised pre-training is an integral part of many 
  language understanding systems.” [39, p.4179]. Even before [39] 
  was published, BERT was picked up by the NLP community and 
  applied with great success to a wide variety of tasks [e.g. 2, 149]. 
  However, no actual language understanding is taking place in 
  LM-driven approaches to these tasks, as can be shown by careful 
  manipulation of the test data to remove spurious cues the systems 
  are leveraging [21, 93]. Furthermore, as Bender and Koller [14] 
  argue from a theoretical perspective, languages are systems of 
  signs [37], i.e. pairings of form and meaning. But the training data 
  for LMs is only form; they do not have access to meaning. Therefore, 
  claims about model abilities must be carefully characterized. 