title: Sentence embedding using T5
body: <p>I would like to use state-of-the-art LM T5 to get sentence embedding vector.
  I found this repository <a href="https://github.com/UKPLab/sentence-transformers" rel="nofollow noreferrer">https://github.com/UKPLab/sentence-transformers</a>
  As I know, in BERT I should take the first token as [CLS] token, and it will be the sentence embedding.
  In this repository I see the same behaviour on T5 model:</p>
  <pre><code>cls_tokens = output_tokens[:, 0, :]  # CLS token is first token
  </code></pre>
  <p>Does this behaviour correct? I have taken encoder from T5 and encoded two phrases with it:</p>
  <pre><code>&quot;I live in the kindergarden&quot;
  &quot;Yes, I live in the kindergarden&quot;
  </code></pre>
  <p>The cosine similarity between them was only &quot;0.2420&quot;.</p>
  <p>I just need to understand how sentence embedding works - should I train network to find similarity to reach correct results? Or I it is enough of base pretrained language model?</p>