<h2 dir="auto">Environment info</h2>

<ul dir="auto">
<li><code class="notranslate">transformers</code> version: Installed from git</li>
</ul>
<h2 dir="auto">Issue</h2>
<p dir="auto">The <code class="notranslate">len(AutoTokenizer.from_pretrained("t5-base"))</code> is <code class="notranslate">32100</code> but the <code class="notranslate">T5ForConditionalGeneration.from_pretrained("t5-base").config.vocab_size</code>  is <code class="notranslate">32128</code>. Seems to be a similar issue to that of : <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="531034498" data-permission-text="Title is private" data-url="https://github.com/huggingface/transformers/issues/2020" data-hovercard-type="issue" data-hovercard-url="/huggingface/transformers/issues/2020/hovercard" href="https://github.com/huggingface/transformers/issues/2020">#2020</a></p>
<h1 dir="auto"><g-emoji class="g-emoji" alias="question" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/2753.png">❓</g-emoji> Questions &amp; Help</h1>
<p dir="auto">Pretrained <code class="notranslate">T5Tokenizer </code> has vocab size of 32100 (32000 tokens plus 100 extra_ids) but the shared embedding layer of <code class="notranslate">T5ForConditionalGeneration</code> has size of (32128, 768). I checked the google-research implementation of T5 and also found that they have vocab size of 32100 also.</p>
<p dir="auto">Where did the extra 28 embeddings come from and how can we map it to the tokenizer?</p>
<h2 dir="auto">To reproduce</h2>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="from transformers import (
    T5Tokenizer, 
    T5ForConditionalGeneration,
)

tokenizer_pretrained = T5Tokenizer.from_pretrained('t5-base')
model_pretrained = T5ForConditionalGeneration.from_pretrained('t5-base')
len(tokenizer_pretrained.get_vocab()), model_pretrained.state_dict()['shared.weight'].shape"><pre class="notranslate"><code class="notranslate">from transformers import (
    T5Tokenizer, 
    T5ForConditionalGeneration,
)

tokenizer_pretrained = T5Tokenizer.from_pretrained('t5-base')
model_pretrained = T5ForConditionalGeneration.from_pretrained('t5-base')
len(tokenizer_pretrained.get_vocab()), model_pretrained.state_dict()['shared.weight'].shape
</code></pre></div>
<p dir="auto">Output:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="(32100, torch.Size([32128, 768]))"><pre class="notranslate"><code class="notranslate">(32100, torch.Size([32128, 768]))
</code></pre></div>