text: 3.1 Syntactic knowledge 
  
  Lin et al. (2019) showed that BERT represen- 
  tations are hierarchical rather than linear, i.e. 
  there is something akin to syntactic tree structure 
  in addition to the word order information. Ten- 
  ney et al. (2019b) and Liu et al. (2019a) also 
  showed that BERT embeddings encode informa- 
  tion about parts of speech, syntactic chunks 
  and roles. Enough syntactic information seems 
  to be captured in the token embeddings themselves 
  to recover syntactic trees (Vilares et al., 2020; Kim 
  et al., 2020; Rosa and Mareˇcek, 2019), although 
  probing classiﬁers could not recover the labels of 
  distant parent nodes in the syntactic tree (Liu et al., 
  2019a). Warstadt and Bowman (2020) report evi- 
  dence of hierarchical structure in three out of four 
  probing tasks. 
  
  Figure 1: Parameter-free probe for syntactic knowledge: 
  words sharing syntactic subtrees have larger impact on 
  each other in the MLM prediction (Wu et al., 2020) 
  
  parameter-free approach based on measuring the 
  impact that one word has on predicting another 
  word within a sequence in the MLM task (Figure 1). 
  They concluded that BERT "naturally" learns 
  some syntactic information, although it is not 
  very similar to linguistic annotated resources. 