<h2 dir="auto">Environment info</h2>

<ul dir="auto">
<li><code class="notranslate">transformers</code> version: 4.1.1</li>
<li>Platform: ubuntu 18.04</li>
<li>Python version: 3.8.5</li>
<li>PyTorch version (GPU?): 1.7.1</li>
</ul>
<h3 dir="auto">Who can help</h3>
<p dir="auto"><a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/patrickvonplaten/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/patrickvonplaten">@patrickvonplaten</a></p>

<h2 dir="auto">To reproduce</h2>
<p dir="auto">Steps to reproduce the behavior:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="from transformers import AutoModelForSeq2SeqLM
from transformers import AutoTokenizer

mt5s = ['google/mt5-base', 'google/mt5-small', 'google/mt5-large', 'google/mt5-xl', 'google/mt5-xxl']

for mt5 in mt5s:
    model = AutoModelForSeq2SeqLM.from_pretrained(mt5)
    tokenizer = AutoTokenizer.from_pretrained(mt5)

    print()
    print(mt5)
    print(f&quot;tokenizer vocab: {tokenizer.vocab_size}, model vocab: {model.config.vocab_size}&quot;)"><pre class="notranslate"><span class="pl-k">from</span> <span class="pl-s1">transformers</span> <span class="pl-k">import</span> <span class="pl-v">AutoModelForSeq2SeqLM</span>
<span class="pl-k">from</span> <span class="pl-s1">transformers</span> <span class="pl-k">import</span> <span class="pl-v">AutoTokenizer</span>

<span class="pl-s1">mt5s</span> <span class="pl-c1">=</span> [<span class="pl-s">'google/mt5-base'</span>, <span class="pl-s">'google/mt5-small'</span>, <span class="pl-s">'google/mt5-large'</span>, <span class="pl-s">'google/mt5-xl'</span>, <span class="pl-s">'google/mt5-xxl'</span>]

<span class="pl-k">for</span> <span class="pl-s1">mt5</span> <span class="pl-c1">in</span> <span class="pl-s1">mt5s</span>:
    <span class="pl-s1">model</span> <span class="pl-c1">=</span> <span class="pl-v">AutoModelForSeq2SeqLM</span>.<span class="pl-en">from_pretrained</span>(<span class="pl-s1">mt5</span>)
    <span class="pl-s1">tokenizer</span> <span class="pl-c1">=</span> <span class="pl-v">AutoTokenizer</span>.<span class="pl-en">from_pretrained</span>(<span class="pl-s1">mt5</span>)

    <span class="pl-en">print</span>()
    <span class="pl-en">print</span>(<span class="pl-s1">mt5</span>)
    <span class="pl-en">print</span>(<span class="pl-s">f"tokenizer vocab: <span class="pl-s1"><span class="pl-kos">{</span><span class="pl-s1">tokenizer</span>.<span class="pl-s1">vocab_size</span><span class="pl-kos">}</span></span>, model vocab: <span class="pl-s1"><span class="pl-kos">{</span><span class="pl-s1">model</span>.<span class="pl-s1">config</span>.<span class="pl-s1">vocab_size</span><span class="pl-kos">}</span></span>"</span>)</pre></div>
<p dir="auto">This is problematic in case when one addes some (special) tokens to tokenizer and resizes the token embedding of the model with <code class="notranslate">model.resize_token_embedding(len(tokenizer))</code></p>

<h2 dir="auto">Expected behavior</h2>
<p dir="auto">vocab_size for model and tokenizer should be the same?</p>
<h1 dir="auto"><g-emoji class="g-emoji" alias="question" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/2753.png">❓</g-emoji> Questions &amp; Help</h1>
<p dir="auto">Pretrained <code class="notranslate">T5Tokenizer </code> has vocab size of 32100 (32000 tokens plus 100 extra_ids) but the shared embedding layer of <code class="notranslate">T5ForConditionalGeneration</code> has size of (32128, 768). I checked the google-research implementation of T5 and also found that they have vocab size of 32100 also.</p>
<p dir="auto">Where did the extra 28 embeddings come from and how can we map it to the tokenizer?</p>
<h2 dir="auto">To reproduce</h2>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="from transformers import (
    T5Tokenizer, 
    T5ForConditionalGeneration,
)

tokenizer_pretrained = T5Tokenizer.from_pretrained('t5-base')
model_pretrained = T5ForConditionalGeneration.from_pretrained('t5-base')
len(tokenizer_pretrained.get_vocab()), model_pretrained.state_dict()['shared.weight'].shape"><pre class="notranslate"><code class="notranslate">from transformers import (
    T5Tokenizer, 
    T5ForConditionalGeneration,
)

tokenizer_pretrained = T5Tokenizer.from_pretrained('t5-base')
model_pretrained = T5ForConditionalGeneration.from_pretrained('t5-base')
len(tokenizer_pretrained.get_vocab()), model_pretrained.state_dict()['shared.weight'].shape
</code></pre></div>
<p dir="auto">Output:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="(32100, torch.Size([32128, 768]))"><pre class="notranslate"><code class="notranslate">(32100, torch.Size([32128, 768]))
</code></pre></div>