title: How to get cosine similarity of word embedding from BERT model
body: <p>I was interesting in how to get the similarity of word embedding in different sentences from BERT model (actually, that means words have different meanings in different scenarios).</p>
  <p>For example:</p>
  <pre><code>sent1 = 'I like living in New York.'
  sent2 = 'New York is a prosperous city.'
  </code></pre>
  <p>I want to get the cos(New York, New York)'s value from sent1 and sent2, even if the phrase 'New York' is same, but it appears in different sentence. I got some intuition from <a href="https://discuss.huggingface.co/t/generate-raw-word-embeddings-using-transformer-models-like-bert-for-downstream-process/2958/2" rel="nofollow noreferrer">https://discuss.huggingface.co/t/generate-raw-word-embeddings-using-transformer-models-like-bert-for-downstream-process/2958/2</a></p>
  <p>But I still do not know which layer's embedding I need to extract and how to caculate the cos similarity for my above example.</p>
  <p>Thanks in advance for any suggestions!</p>