text: As far as how syntax is represented, it seems 
  that syntactic structure is not directly encoded 
  in self-attention weights. Htut et al. (2019) were 
  unable to extract full parse trees from BERT heads 
  even with the gold annotations for the root. Jawahar 
  et al. (2019) include a brief illustration of a depen- 
  dency tree extracted directly from self-attention 
  weights, but provide no quantitative evaluation. 
  
  The ﬁll-in-the-gap probes of MLM showed that 
  BERT takes subject-predicate agreement into 
  account when performing the cloze task (Gold- 
  berg, 2019; van Schijndel et al., 2019), even for 
  meaningless sentences and sentences with distrac- 
  tor clauses between the subject and the verb (Gold- 
  berg, 2019). A study of negative polarity items 
  (NPIs) by Warstadt et al. (2019) showed that BERT 
  is better able to detect the presence of NPIs (e.g. 
  "ever") and the words that allow their use (e.g. 
  "whether") than scope violations. 
  
  The above claims of syntactic knowledge are be- 
  lied by the evidence that BERT does not "under- 
  stand" negation and is insensitive to malformed 
  input. In particular, its predictions were not al- 
  tered2 even with shufﬂed word order, truncated 
  sentences, removed subjects and objects (Ettinger, 
  2019). This could mean that either BERT’s syn- 
  tactic knowledge is incomplete, or it does not 
  need to rely on it for solving its tasks. The latter 
  seems more likely, since Glavaš and Vuli´c (2020) 