text: There is conﬂicting evidence about syntactic 
  chunks. Tenney et al. (2019a) conclude that "the 
  basic syntactic information appears earlier in the 
  network while high-level semantic features appear 
  at the higher layers", drawing parallels between 
  this order and the order of components in a typical 
  NLP pipeline – from POS-tagging to dependency 
  parsing to semantic role labeling. Jawahar et al. 
  (2019) also report that the lower layers were more 
  useful for chunking, while middle layers were more 
  useful for parsing. At the same time, the probing 
  experiments by Liu et al. (2019a) ﬁnd the opposite: 
  both POS-tagging and chunking were performed 
  best at the middle layers, in both BERT-base and 
  BERT-large. However, all three studies use differ- 
  ent suites of probing tasks. 
  
  4.3 BERT layers 
  
  The ﬁrst layer of BERT receives as input a combina- 
  tion of token, segment, and positional embeddings. 
  It stands to reason that the lower layers have 
  the most information about linear word order. 
  Lin et al. (2019) report a decrease in the knowledge 
  of linear word order around layer 4 in BERT-base. 
  This is accompanied by an increased knowledge 
  
  The ﬁnal layers of BERT are the most task- 
  speciﬁc. In pre-training, this means speciﬁcity to 
  the MLM task, which explains why the middle 