text: But this is not to say that there is no room for 
  improvement. Ethayarajh (2019) measure how 
  similar the embeddings for identical words are in 
  every layer, reporting that later BERT layers pro- 
  duce more context-speciﬁc representations3. They 
  also ﬁnd that BERT embeddings occupy a narrow 
  cone in the vector space, and this effect increases 
  from the earlier to later layers. That is, two ran- 
  dom words will on average have a much higher 
  cosine similarity than expected if embeddings 
  were directionally uniform (isotropic). Since 
  isotropy was shown to be beneﬁcial for static word 
  embeddings (Mu and Viswanath, 2018), this might 
  be a fruitful direction to explore for BERT. 
  
  Another direction is information-theoretic prob- 
  ing. Pimentel et al. (2020) operationalize prob- 
  ing as estimating mutual information between the 
  learned representation and a given linguistic prop- 
  erty, which highlights that the focus should be not 
  on the amount of information contained in a rep- 
  resentation, but rather on how easily it can be ex- 
  tracted from it. Voita and Titov (2020) quantify 
  the amount of effort needed to extract information 
  from a given representation as minimum descrip- 
  tion length needed to communicate both the probe 
  size and the amount of data required for it to do 
  well on a task. 