We measure how contextual a word representation is using three different metrics: self-similarity, intra-sentence similarity, and maximum explainable variance. 
 Question: What experiments are proposed to test that upper layers produce context-specific embeddings?
They measure self-similarity, intra-sentence similarity and maximum explainable variance of the embeddings in the upper layers.