text: 3.2 Semantic knowledge 
  
  To date, more studies have been devoted to BERT’s 
  knowledge of syntactic rather than semantic phe- 
  nomena. However, we do have evidence from an 
  MLM probing study that BERT has some knowl- 
  edge of semantic roles (Ettinger, 2019). BERT 
  even displays some preference for the incorrect 
  ﬁllers for semantic roles that are semantically re- 
  lated to the correct ones, as opposed to those that 
  are unrelated (e.g. "to tip a chef" is better than "to 
  tip a robin", but worse than "to tip a waiter"). 
  
  Figure 2: BERT world knowledge (Petroni et al., 2019) 
  
  blanks (e.g. "Cats like to chase [___]"). Petroni 
  et al. (2019) showed that, for some relation types, 
  vanilla BERT is competitive with methods rely- 
  ing on knowledge bases (Figure 2), and Roberts 
  et al. (2020) show the same for open-domain QA 
  using T5 model (Raffel et al., 2019). Davison et al. 
  (2019) suggest that it generalizes better to unseen 
  data. In order to retrieve BERT’s knowledge, we 
  need good template sentences, and there is work 
  on their automatic extraction and augmentation 
  (Bouraoui et al., 2019; Jiang et al., 2019b). 
  Tenney et al. (2019b) showed that BERT en- 
  codes information about entity types, relations, 
  semantic roles, and proto-roles, since this infor- 
  mation can be detected with probing classiﬁers. 