<h2 dir="auto">Environment info</h2>
<ul dir="auto">
<li><code class="notranslate">transformers</code> version: 4.11.3</li>
<li>Platform: Linux-5.4.104+-x86_64-with-Ubuntu-18.04-bionic</li>
<li>Python version: 3.7.12</li>
<li>PyTorch version (GPU?): 1.9.0+cu111 (False)</li>
<li>Tensorflow version (GPU?): 2.6.0 (False)</li>
<li>Flax version (CPU?/GPU?/TPU?): not installed (NA)</li>
<li>Jax version: not installed</li>
<li>JaxLib version: not installed</li>
<li>Using GPU in script?: no</li>
<li>Using distributed or parallel set-up in script?: no</li>
</ul>
<h2 dir="auto">Information</h2>
<p dir="auto">I'm using <code class="notranslate">t5-base</code> model (yet my test shows the same result for <code class="notranslate">t5-small</code>).</p>
<p dir="auto">To implement some knowledge-distillation task and "classical" sequence-to-sequence tasks - I'm trying to use <code class="notranslate">batch_size x seq_length x vocab_size</code> array as labels for both kinds of tasks (soft labels in KD case, one-hot hard labels I'm seq2seq case).</p>
<p dir="auto">So I need to convert tokenizer output from <code class="notranslate">batch_size x seq_length</code> to <code class="notranslate">batch_size x seq_length x vocab_size</code> one-hot array to pass it to my custom loss later.<br>
Yet I found out that I can't just use <code class="notranslate">T5Tokenizer(...).vocab_size</code> to build one-hot matrix - <code class="notranslate">T5ForConditionalGeneration(...).config.vocab_size</code> have different value.<br>
So when I'm trying to build a one-hot vector based on tokenizer vocab size - I'm getting dimension mismatch errors.</p>
<p dir="auto">The next code shows me different vocabulary sizes when I'm trying to access vocab size through <code class="notranslate">T5Tokenizer</code> and related <code class="notranslate">T5ForConditionalGeneration</code> config:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="from transformers import T5ForConditionalGeneration, T5Tokenizer

T5_MODEL = &quot;t5-base&quot;
print(&quot;tokenizer&quot;, T5Tokenizer.from_pretrained(T5_MODEL).vocab_size)
print(&quot;model&quot;, T5ForConditionalGeneration.from_pretrained(T5_MODEL).config.vocab_size)"><pre class="notranslate"><code class="notranslate">from transformers import T5ForConditionalGeneration, T5Tokenizer

T5_MODEL = "t5-base"
print("tokenizer", T5Tokenizer.from_pretrained(T5_MODEL).vocab_size)
print("model", T5ForConditionalGeneration.from_pretrained(T5_MODEL).config.vocab_size)
</code></pre></div>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="tokenizer 32100
model 32128"><pre class="notranslate"><code class="notranslate">tokenizer 32100
model 32128
</code></pre></div>
<h2 dir="auto">Expected behavior</h2>
<p dir="auto">I expected the tokenizer &amp; model to have the same vocabulary size.</p>
<h1 dir="auto"><g-emoji class="g-emoji" alias="question" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/2753.png">❓</g-emoji> Questions &amp; Help</h1>
<p dir="auto">Pretrained <code class="notranslate">T5Tokenizer </code> has vocab size of 32100 (32000 tokens plus 100 extra_ids) but the shared embedding layer of <code class="notranslate">T5ForConditionalGeneration</code> has size of (32128, 768). I checked the google-research implementation of T5 and also found that they have vocab size of 32100 also.</p>
<p dir="auto">Where did the extra 28 embeddings come from and how can we map it to the tokenizer?</p>
<h2 dir="auto">To reproduce</h2>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="from transformers import (
    T5Tokenizer, 
    T5ForConditionalGeneration,
)

tokenizer_pretrained = T5Tokenizer.from_pretrained('t5-base')
model_pretrained = T5ForConditionalGeneration.from_pretrained('t5-base')
len(tokenizer_pretrained.get_vocab()), model_pretrained.state_dict()['shared.weight'].shape"><pre class="notranslate"><code class="notranslate">from transformers import (
    T5Tokenizer, 
    T5ForConditionalGeneration,
)

tokenizer_pretrained = T5Tokenizer.from_pretrained('t5-base')
model_pretrained = T5ForConditionalGeneration.from_pretrained('t5-base')
len(tokenizer_pretrained.get_vocab()), model_pretrained.state_dict()['shared.weight'].shape
</code></pre></div>
<p dir="auto">Output:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="(32100, torch.Size([32128, 768]))"><pre class="notranslate"><code class="notranslate">(32100, torch.Size([32128, 768]))
</code></pre></div>