<h2 dir="auto">Environment info</h2>

<ul dir="auto">
<li><code class="notranslate">transformers</code> version: 4.1.1</li>
<li>Python version: 3.8.5</li>
<li>PyTorch version (GPU?): 1.7.1</li>
<li>tokenizers: 0.9.4</li>
<li>Using GPU in script?: No</li>
<li>Using distributed or parallel set-up in script?: No</li>
</ul>
<h2 dir="auto">Information</h2>
<p dir="auto">Hi <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/patrickvonplaten/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/patrickvonplaten">@patrickvonplaten</a><br>
I am trying to train a "t5-base" model and I directly use from_pretrained tokenizer, config and model. However, I found the vocabulary size given by the tokenizer and config is different (see to reproduce). Does this is expected?</p>
<p dir="auto">If I use the model <code class="notranslate">T5ForConditionalGeneration.from_pretrained('t5-base', config=config)</code> to do predictions, this will result in the last dimension of lm_logits is different from <code class="notranslate">tokenizer.vocab_size</code>.</p>
<h2 dir="auto">To reproduce</h2>
<p dir="auto">Steps to reproduce the behavior:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="&gt;&gt;&gt; from transformers import T5Tokenizer, T5Config
&gt;&gt;&gt; tokenizer = T5Tokenizer.from_pretrained(&quot;t5-base&quot;)
&gt;&gt;&gt; config = T5Config.from_pretrained(&quot;t5-base&quot;)
&gt;&gt;&gt; print(tokenizer.vocab_size)
32100
&gt;&gt;&gt; print(config.vocab_size)
32128"><pre class="notranslate"><code class="notranslate">&gt;&gt;&gt; from transformers import T5Tokenizer, T5Config
&gt;&gt;&gt; tokenizer = T5Tokenizer.from_pretrained("t5-base")
&gt;&gt;&gt; config = T5Config.from_pretrained("t5-base")
&gt;&gt;&gt; print(tokenizer.vocab_size)
32100
&gt;&gt;&gt; print(config.vocab_size)
32128
</code></pre></div>
<h2 dir="auto">Expected behavior</h2>

<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="&gt;&gt;&gt; print(tokenizer.vocab_size)
32128
&gt;&gt;&gt; print(config.vocab_size)
32128"><pre class="notranslate"><code class="notranslate">&gt;&gt;&gt; print(tokenizer.vocab_size)
32128
&gt;&gt;&gt; print(config.vocab_size)
32128
</code></pre></div>
<h1 dir="auto"><g-emoji class="g-emoji" alias="question" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/2753.png">❓</g-emoji> Questions &amp; Help</h1>
<p dir="auto">Pretrained <code class="notranslate">T5Tokenizer </code> has vocab size of 32100 (32000 tokens plus 100 extra_ids) but the shared embedding layer of <code class="notranslate">T5ForConditionalGeneration</code> has size of (32128, 768). I checked the google-research implementation of T5 and also found that they have vocab size of 32100 also.</p>
<p dir="auto">Where did the extra 28 embeddings come from and how can we map it to the tokenizer?</p>
<h2 dir="auto">To reproduce</h2>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="from transformers import (
    T5Tokenizer, 
    T5ForConditionalGeneration,
)

tokenizer_pretrained = T5Tokenizer.from_pretrained('t5-base')
model_pretrained = T5ForConditionalGeneration.from_pretrained('t5-base')
len(tokenizer_pretrained.get_vocab()), model_pretrained.state_dict()['shared.weight'].shape"><pre class="notranslate"><code class="notranslate">from transformers import (
    T5Tokenizer, 
    T5ForConditionalGeneration,
)

tokenizer_pretrained = T5Tokenizer.from_pretrained('t5-base')
model_pretrained = T5ForConditionalGeneration.from_pretrained('t5-base')
len(tokenizer_pretrained.get_vocab()), model_pretrained.state_dict()['shared.weight'].shape
</code></pre></div>
<p dir="auto">Output:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="(32100, torch.Size([32128, 768]))"><pre class="notranslate"><code class="notranslate">(32100, torch.Size([32128, 768]))
</code></pre></div>