Turning to semantics, using visualizations of the activations created by different pieces of text, we show suggestive evidence that BERT distinguishes word senses at a very fine level.  We apply attention probes to the task of identifying the existence and type of dependency relation between two words. 
 Question: How were the feature representations evaluated?
attention probes using visualizations of the activations created by different pieces of text