<h2 dir="auto">Environment info</h2>

<ul dir="auto">
<li><code class="notranslate">transformers</code> version: 4.0.0 --rc1</li>
<li>Platform: Colab</li>
<li>Python version: 3.6.9</li>
<li>PyTorch version (GPU?): TESLA V4</li>
<li>Tensorflow version (GPU?):  2.3.0</li>
<li>Using GPU in script?: YES</li>
<li>Using distributed or parallel set-up in script?: NO</li>
</ul>
<h3 dir="auto">Who can help</h3>

<h2 dir="auto">Information</h2>
<p dir="auto">Model I am using (Bert, XLNet ...): T5-1.1</p>
<p dir="auto">The problem arises when using:</p>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> the official example scripts: (give details below)</li>
<li>[X ] my own modified scripts: (give details below)</li>
</ul>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="from transformers import T5TokenizerFast, T5ForConditionalGeneration
TOKENIZER_NAME = 't5-base'
MODEL_NAME = 'google/t5-v1_1-base'
tokenizer = T5TokenizerFast.from_pretrained(MODEL_NAME)
special_tokens_dict = {'additional_special_tokens': ['&lt;ORG&gt;','&lt;PERSON&gt;']}
num_added_toks = tokenizer.add_special_tokens(special_tokens_dict)
model = T5ForConditionalGeneration.from_pretrained('google/t5-v1_1-base', return_dict=True)
model.resize_token_embeddings(len(tokenizer))
model.to(&quot;cuda&quot;)"><pre class="notranslate"><span class="pl-k">from</span> <span class="pl-s1">transformers</span> <span class="pl-k">import</span> <span class="pl-v">T5TokenizerFast</span>, <span class="pl-v">T5ForConditionalGeneration</span>
<span class="pl-v">TOKENIZER_NAME</span> <span class="pl-c1">=</span> <span class="pl-s">'t5-base'</span>
<span class="pl-v">MODEL_NAME</span> <span class="pl-c1">=</span> <span class="pl-s">'google/t5-v1_1-base'</span>
<span class="pl-s1">tokenizer</span> <span class="pl-c1">=</span> <span class="pl-v">T5TokenizerFast</span>.<span class="pl-en">from_pretrained</span>(<span class="pl-v">MODEL_NAME</span>)
<span class="pl-s1">special_tokens_dict</span> <span class="pl-c1">=</span> {<span class="pl-s">'additional_special_tokens'</span>: [<span class="pl-s">'&lt;ORG&gt;'</span>,<span class="pl-s">'&lt;PERSON&gt;'</span>]}
<span class="pl-s1">num_added_toks</span> <span class="pl-c1">=</span> <span class="pl-s1">tokenizer</span>.<span class="pl-en">add_special_tokens</span>(<span class="pl-s1">special_tokens_dict</span>)
<span class="pl-s1">model</span> <span class="pl-c1">=</span> <span class="pl-v">T5ForConditionalGeneration</span>.<span class="pl-en">from_pretrained</span>(<span class="pl-s">'google/t5-v1_1-base'</span>, <span class="pl-s1">return_dict</span><span class="pl-c1">=</span><span class="pl-c1">True</span>)
<span class="pl-s1">model</span>.<span class="pl-en">resize_token_embeddings</span>(<span class="pl-en">len</span>(<span class="pl-s1">tokenizer</span>))
<span class="pl-s1">model</span>.<span class="pl-en">to</span>(<span class="pl-s">"cuda"</span>)</pre></div>
<p dir="auto">The tasks I am working on is:</p>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> an official GLUE/SQUaD task: (give the name)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> my own task or dataset: (give details below)<br>
Custom dataset</li>
</ul>
<h2 dir="auto">To reproduce</h2>
<p dir="auto">Steps to reproduce the behavior:</p>
<ol dir="auto">
<li>Attempting to add Entity tokens to T5 1.1, upon loading from pretrained the following error occurs:<br>
<code class="notranslate">size mismatch for lm_head.weight: copying a param with shape torch.Size([32128, 768]) from checkpoint, the shape in current model is torch.Size([32102, 768]).</code></li>
</ol>
<p dir="auto">I am assuming the addition of the special tokens did not get propagated to the lm head size.</p>
<p dir="auto">I would expect the LM Head to be resized in addition to the standard layers.</p>
<p dir="auto">Many Thanks,<br>
Chris</p>
<h2 dir="auto">Environment info</h2>
<p dir="auto">Google colab</p>
<h3 dir="auto">Who can help</h3>
<p dir="auto">T5: <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/patrickvonplaten/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/patrickvonplaten">@patrickvonplaten</a></p>
<h2 dir="auto">Information</h2>
<p dir="auto">I'm noticing something strange with T5. The model embedding size and the tokenizer size does not match.<br>
When I try to resize the model to have a smaller embedding this crashes CUDA. This is probably two bugs - one for the size mismatch, and one for shortening the embedding causing a crash.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained(&quot;t5-base&quot;)
print (len(tokenizer))
model = AutoModel.from_pretrained(&quot;t5-base&quot;)
print (model.shared)
model.resize_token_embeddings(len(tokenizer))
model.to('cuda')"><pre class="notranslate"><code class="notranslate">from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("t5-base")
print (len(tokenizer))
model = AutoModel.from_pretrained("t5-base")
print (model.shared)
model.resize_token_embeddings(len(tokenizer))
model.to('cuda')
</code></pre></div>
<h2 dir="auto">Expected behavior</h2>
<p dir="auto">Expected behaviour is regular loading of the model into cuda.</p>
<h2 dir="auto">What I got instead was:<br>
32100<br>
Some weights of T5Model were not initialized from the model checkpoint at t5-base and are newly initialized: ['encoder.embed_tokens.weight', 'decoder.embed_tokens.weight']<br>
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.<br>
Embedding(32128, 768)</h2>
<p dir="auto">RuntimeError                              Traceback (most recent call last)<br>
 in ()<br>
5 print (model.shared)<br>
6 model.resize_token_embeddings(len(tokenizer))<br>
----&gt; 7 model.to('cuda')</p>
<p dir="auto">3 frames<br>
/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in convert(t)<br>
608             if convert_to_format is not None and t.dim() == 4:<br>
609                 return t.to(device, dtype if t.is_floating_point() else None, non_blocking, memory_format=convert_to_format)<br>
--&gt; 610             return t.to(device, dtype if t.is_floating_point() else None, non_blocking)<br>
611<br>
612         return self._apply(convert)</p>
<p dir="auto">RuntimeError: CUDA error: device-side assert triggered</p>