<h2 dir="auto">Environment info</h2>
<ul dir="auto">
<li><code class="notranslate">transformers</code> version: 4.0.1</li>
<li>Platform: Linux-4.15.0-132-generic-x86_64-with-glibc2.10</li>
<li>Python version: 3.8.5</li>
<li>PyTorch version (GPU?): 1.7.1+cu110 (True)</li>
<li>Tensorflow version (GPU?): not installed (NA)</li>
<li>Using GPU in script?: </li>
<li>Using distributed or parallel set-up in script?: no</li>
</ul>
<p dir="auto">Note also: cookiecutter dependency is not included in pip install transformers so transformers-cli env initially fails</p>
<h3 dir="auto">Who can help</h3>
<p dir="auto">(T5) <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/patrickvonplaten/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/patrickvonplaten">@patrickvonplaten</a>, <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/patil-suraj/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/patil-suraj">@patil-suraj</a>, <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/sshleifer/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/sshleifer">@sshleifer</a></p>
<p dir="auto">When using <code class="notranslate">T5ForConditionalGeneration.from_pretrained('t5-base')</code>, I get the following warning at load:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Some weights of the model checkpoint at t5-large were not used when initializing T5ForConditionalGeneration: ['decoder.block.0.layer.1.EncDecAttention.relative_attention_bias.weight']
- This IS expected if you are initializing T5ForConditionalGeneration from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining m
odel).                                                                                                                 
- This IS NOT expected if you are initializing T5ForConditionalGeneration from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification mode
l). "><pre class="notranslate"><code class="notranslate">Some weights of the model checkpoint at t5-large were not used when initializing T5ForConditionalGeneration: ['decoder.block.0.layer.1.EncDecAttention.relative_attention_bias.weight']
- This IS expected if you are initializing T5ForConditionalGeneration from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining m
odel).                                                                                                                 
- This IS NOT expected if you are initializing T5ForConditionalGeneration from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification mode
l). 
</code></pre></div>
<p dir="auto">If I load from a checkpoint that I create (i.e. local file), I get the same message.</p>
<p dir="auto">But I think that all weights are, in fact, identical:</p>
<ul dir="auto">
<li>evaluation code on the model I finetune before saving AND</li>
<li>evaluation code on the model I finetune, save, and then reload<br>
are identical.</li>
</ul>
<p dir="auto">This suggests that <em>all</em> weights are identical, since performance is identical. This contradicts the warning message.</p>
<p dir="auto">Questions:</p>
<ol dir="auto">
<li>Are some weights actually not being loaded? If so, how could I observe identical behavior on metrics. Or is this warning wrong?</li>
<li>If this warning is correct, how can I force the model to fully load the model exactly as I saved it.</li>
<li>Is there any other difference (randomly initialized head, randomly initialized weights) between the t5 that is pretrained and the T5ForConditionalGeneration?</li>
</ol>
<h2 dir="auto">Environment info</h2>
<ul dir="auto">
<li><code class="notranslate">transformers</code> version: 4.0.0</li>
<li>Platform: Linux 18.04</li>
<li>Python version: 3.6.9</li>
<li>PyTorch version (GPU?): 1.70 (True)</li>
<li>Tensorflow version (GPU?): N/A</li>
<li>Using GPU in script?: Yes</li>
<li>Using distributed or parallel set-up in script?: No</li>
</ul>
<h3 dir="auto">Who can help</h3>
<p dir="auto"><a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/patrickvonplaten/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/patrickvonplaten">@patrickvonplaten</a></p>
<h2 dir="auto">Information</h2>
<p dir="auto">Model I am using: T5ForConditionalGeneration, specifically the "allenai/unifiedqa-t5-large" checkpoint from the model hub</p>
<p dir="auto">The problem arises when I try to load the checkpoint following standard loading procedures under <code class="notranslate">transformers==4.0.0</code>. The same doesn't happen in version <code class="notranslate">3.5.0</code>.</p>
<h2 dir="auto">To reproduce</h2>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="from transformers import AutoTokenizer, T5ForConditionalGeneration
model_name_or_path = &quot;allenai/unifiedqa-t5-large&quot;
tokenizer = AutoTokenizer.from_pretrained(model_name_or_path)
# Warnings in version 4.0, not in 3.5.0 an preceding ones
model = T5ForConditionalGeneration.from_pretrained(model_name_or_path)"><pre class="notranslate"><span class="pl-k">from</span> <span class="pl-s1">transformers</span> <span class="pl-k">import</span> <span class="pl-v">AutoTokenizer</span>, <span class="pl-v">T5ForConditionalGeneration</span>
<span class="pl-s1">model_name_or_path</span> <span class="pl-c1">=</span> <span class="pl-s">"allenai/unifiedqa-t5-large"</span>
<span class="pl-s1">tokenizer</span> <span class="pl-c1">=</span> <span class="pl-v">AutoTokenizer</span>.<span class="pl-en">from_pretrained</span>(<span class="pl-s1">model_name_or_path</span>)
<span class="pl-c"># Warnings in version 4.0, not in 3.5.0 an preceding ones</span>
<span class="pl-s1">model</span> <span class="pl-c1">=</span> <span class="pl-v">T5ForConditionalGeneration</span>.<span class="pl-en">from_pretrained</span>(<span class="pl-s1">model_name_or_path</span>)</pre></div>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Some weights of the model checkpoint at allenai/unifiedqa-t5-large were not used when initializing T5ForConditionalGeneration: ['decoder.blo
ck.0.layer.1.EncDecAttention.relative_attention_bias.weight']                                                                               
- This IS expected if you are initializing T5ForConditionalGeneration from the checkpoint of a model trained on another task or with another
 architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).                                    
- This IS NOT expected if you are initializing T5ForConditionalGeneration from the checkpoint of a model that you expect to be exactly ident
ical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model)."><pre class="notranslate"><code class="notranslate">Some weights of the model checkpoint at allenai/unifiedqa-t5-large were not used when initializing T5ForConditionalGeneration: ['decoder.blo
ck.0.layer.1.EncDecAttention.relative_attention_bias.weight']                                                                               
- This IS expected if you are initializing T5ForConditionalGeneration from the checkpoint of a model trained on another task or with another
 architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).                                    
- This IS NOT expected if you are initializing T5ForConditionalGeneration from the checkpoint of a model that you expect to be exactly ident
ical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
</code></pre></div>
<h2 dir="auto">Expected behavior</h2>
<p dir="auto">A consistent behavior across versions, either always or never raising the warning at loading time.</p>