<h2 dir="auto"><g-emoji class="g-emoji" alias="bug" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/1f41b.png">🐛</g-emoji> Bug</h2>

<p dir="auto">Model I am using (Bert, XLNet....): Bert</p>
<p dir="auto">Language I am using the model on (English, Chinese....): English</p>
<p dir="auto">The problem arise when using:</p>
<ul dir="auto">
<li>the official example scripts: modeling_bert.py</li>
</ul>
<p dir="auto">The tasks I am working on is:</p>
<ul dir="auto">
<li>my own task or dataset: fine-tuning Bert with added new tokens to vocabulary</li>
</ul>
<h2 dir="auto">To Reproduce</h2>
<p dir="auto">Steps to reproduce the behavior:</p>
<p dir="auto">Running "run_lm_finetuning.py" with added tokens to vocabulary.</p>

<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="new_vocab_list = ['token_1', 'token_2', 'token_3']
tokenizer.add_tokens(new_vocab_list)
logger.info(&quot;vocabulary size after adding: &quot; + str(len(tokenizer)))
model.resize_token_embeddings(len(tokenizer))
logger.info(&quot;size of model.cls.predictions.bias: &quot; + str(len(model.cls.predictions.bias)))"><pre class="notranslate"><code class="notranslate">new_vocab_list = ['token_1', 'token_2', 'token_3']
tokenizer.add_tokens(new_vocab_list)
logger.info("vocabulary size after adding: " + str(len(tokenizer)))
model.resize_token_embeddings(len(tokenizer))
logger.info("size of model.cls.predictions.bias: " + str(len(model.cls.predictions.bias)))
</code></pre></div>
<h2 dir="auto">Expected behavior</h2>

<ul dir="auto">
<li>The result should be:<br>
vocabulary size after adding: 31119<br>
size of model.cls.predictions.bias: 31119</li>
<li>But actually the result is:<br>
vocabulary size after adding: 31119<br>
size of model.cls.predictions.bias: 31116</li>
</ul>
<h2 dir="auto">Environment</h2>
<ul dir="auto">
<li>OS: Ubuntu</li>
<li>Python version: 3.6</li>
<li>PyTorch version: 1.3.1</li>
<li>PyTorch Transformers version (or branch): 2.2.1</li>
<li>Using GPU: yes</li>
<li>Distributed or parallel setup: no</li>
</ul>
<h2 dir="auto">Additional context</h2>

<p dir="auto">I have found the problem to be: for BERT model, the class "BertLMPredictionHead" has two separate attributes "decoder" and "bias". When adding new tokens, the code "model.resize_token_embeddings(len(tokenizer))" only updates the size of "decoder" and its bias if it has (this bias is different from the "BertLMPredictionHead.bias"). The attribute "BertLMPredictionHead.bias" is not updated and therefore, causes the error.</p>
<p dir="auto">I have added the updating-bias code in my "modeling_bert.py". And if you want, I can merge my branch to your code. However, if I misunderstand something, please notice me too.</p>
<p dir="auto">Thank you very much for your code base.</p>
<h2 dir="auto">🐛 Bug</h2>

<p dir="auto">Model I am using : <strong>BertForMaskedLM</strong></p>
<p dir="auto">Language I am using the model on (English, Chinese....): <strong>multilingual</strong></p>
<p dir="auto">The problem arise when using:</p>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> the official example scripts: <strong>BertForMaskedLM</strong></li>
</ul>
<p dir="auto">The tasks I am working on is:</p>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> my own task or dataset: <strong>Finetuning with newly added tokens</strong>.</li>
</ul>
<h2 dir="auto">To Reproduce</h2>
<p dir="auto">Steps to reproduce the behavior:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="# Get model
model = BertForMaskedLM.from_pretrained(&quot;bert-base-multilingual-cased&quot;)

# define input
input = torch.tensor([[1,2,3,4,5]])

# before experiment
print(f&quot;BERT num embeddings before\t: {model.bert.embeddings.word_embeddings.num_embeddings}&quot;)
print(f&quot;LM Decoder num embedding before\t: {model.cls.predictions.decoder.out_features}&quot;)
print(f&quot;MLM Loss\t\t\t: {model(input_ids=input, masked_lm_labels=input)[0]}&quot;)

# change embedding size for experiment
model.resize_token_embeddings(119547 + 5)
print(f&quot;\nBERT num embeddings after\t: {model.bert.embeddings.word_embeddings.num_embeddings}&quot;)
print(f&quot;LM Decoder num embedding after\t: {model.cls.predictions.decoder.out_features}&quot;)
# print failuire
try:
  print(f&quot;MLM Loss\t\t\t: {model(input_ids=input, masked_lm_labels=input)[0]}&quot;)
except RuntimeError:
  print(&quot; ---- Forward Pass failed ---- &quot;)"><pre class="notranslate"><span class="pl-c"># Get model</span>
<span class="pl-s1">model</span> <span class="pl-c1">=</span> <span class="pl-v">BertForMaskedLM</span>.<span class="pl-en">from_pretrained</span>(<span class="pl-s">"bert-base-multilingual-cased"</span>)

<span class="pl-c"># define input</span>
<span class="pl-s1">input</span> <span class="pl-c1">=</span> <span class="pl-s1">torch</span>.<span class="pl-en">tensor</span>([[<span class="pl-c1">1</span>,<span class="pl-c1">2</span>,<span class="pl-c1">3</span>,<span class="pl-c1">4</span>,<span class="pl-c1">5</span>]])

<span class="pl-c"># before experiment</span>
<span class="pl-en">print</span>(<span class="pl-s">f"BERT num embeddings before<span class="pl-cce">\t</span>: <span class="pl-s1"><span class="pl-kos">{</span><span class="pl-s1">model</span>.<span class="pl-s1">bert</span>.<span class="pl-s1">embeddings</span>.<span class="pl-s1">word_embeddings</span>.<span class="pl-s1">num_embeddings</span><span class="pl-kos">}</span></span>"</span>)
<span class="pl-en">print</span>(<span class="pl-s">f"LM Decoder num embedding before<span class="pl-cce">\t</span>: <span class="pl-s1"><span class="pl-kos">{</span><span class="pl-s1">model</span>.<span class="pl-s1">cls</span>.<span class="pl-s1">predictions</span>.<span class="pl-s1">decoder</span>.<span class="pl-s1">out_features</span><span class="pl-kos">}</span></span>"</span>)
<span class="pl-en">print</span>(<span class="pl-s">f"MLM Loss<span class="pl-cce">\t</span><span class="pl-cce">\t</span><span class="pl-cce">\t</span>: <span class="pl-s1"><span class="pl-kos">{</span><span class="pl-en">model</span>(<span class="pl-s1">input_ids</span><span class="pl-c1">=</span><span class="pl-s1">input</span>, <span class="pl-s1">masked_lm_labels</span><span class="pl-c1">=</span><span class="pl-s1">input</span>)[<span class="pl-c1">0</span>]<span class="pl-kos">}</span></span>"</span>)

<span class="pl-c"># change embedding size for experiment</span>
<span class="pl-s1">model</span>.<span class="pl-en">resize_token_embeddings</span>(<span class="pl-c1">119547</span> <span class="pl-c1">+</span> <span class="pl-c1">5</span>)
<span class="pl-en">print</span>(<span class="pl-s">f"<span class="pl-cce">\n</span>BERT num embeddings after<span class="pl-cce">\t</span>: <span class="pl-s1"><span class="pl-kos">{</span><span class="pl-s1">model</span>.<span class="pl-s1">bert</span>.<span class="pl-s1">embeddings</span>.<span class="pl-s1">word_embeddings</span>.<span class="pl-s1">num_embeddings</span><span class="pl-kos">}</span></span>"</span>)
<span class="pl-en">print</span>(<span class="pl-s">f"LM Decoder num embedding after<span class="pl-cce">\t</span>: <span class="pl-s1"><span class="pl-kos">{</span><span class="pl-s1">model</span>.<span class="pl-s1">cls</span>.<span class="pl-s1">predictions</span>.<span class="pl-s1">decoder</span>.<span class="pl-s1">out_features</span><span class="pl-kos">}</span></span>"</span>)
<span class="pl-c"># print failuire</span>
<span class="pl-k">try</span>:
  <span class="pl-en">print</span>(<span class="pl-s">f"MLM Loss<span class="pl-cce">\t</span><span class="pl-cce">\t</span><span class="pl-cce">\t</span>: <span class="pl-s1"><span class="pl-kos">{</span><span class="pl-en">model</span>(<span class="pl-s1">input_ids</span><span class="pl-c1">=</span><span class="pl-s1">input</span>, <span class="pl-s1">masked_lm_labels</span><span class="pl-c1">=</span><span class="pl-s1">input</span>)[<span class="pl-c1">0</span>]<span class="pl-kos">}</span></span>"</span>)
<span class="pl-k">except</span> <span class="pl-v">RuntimeError</span>:
  <span class="pl-en">print</span>(<span class="pl-s">" ---- Forward Pass failed ---- "</span>)</pre></div>
<p dir="auto">This is the output that I see</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="BERT num embeddings before	: 119547
LM Decoder num embedding before	: 119547
MLM Loss			: 18.269268035888672

BERT num embeddings after	: 119552
LM Decoder num embedding after	: 119547
 ---- Forward Pass failed ---- "><pre class="notranslate"><code class="notranslate">BERT num embeddings before	: 119547
LM Decoder num embedding before	: 119547
MLM Loss			: 18.269268035888672

BERT num embeddings after	: 119552
LM Decoder num embedding after	: 119547
 ---- Forward Pass failed ---- 
</code></pre></div>
<p dir="auto">If you see the LM decoder did not change the embedding size, hence won't ever predict the new tokens.</p>
<p dir="auto">The expectation would be that <code class="notranslate">resize_token_embeddings</code> handles the FC inside LM decoder.</p>
<p dir="auto">I tried also using BertConfig</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="config = BertConfig.from_pretrained(&quot;bert-base-multilingual-cased&quot;)
config.vocab_size = 119547 + 5
model = BertForMaskedLM.from_pretrained(&quot;bert-base-multilingual-cased&quot;, config=config)"><pre class="notranslate"><span class="pl-s1">config</span> <span class="pl-c1">=</span> <span class="pl-v">BertConfig</span>.<span class="pl-en">from_pretrained</span>(<span class="pl-s">"bert-base-multilingual-cased"</span>)
<span class="pl-s1">config</span>.<span class="pl-s1">vocab_size</span> <span class="pl-c1">=</span> <span class="pl-c1">119547</span> <span class="pl-c1">+</span> <span class="pl-c1">5</span>
<span class="pl-s1">model</span> <span class="pl-c1">=</span> <span class="pl-v">BertForMaskedLM</span>.<span class="pl-en">from_pretrained</span>(<span class="pl-s">"bert-base-multilingual-cased"</span>, <span class="pl-s1">config</span><span class="pl-c1">=</span><span class="pl-s1">config</span>)</pre></div>
<p dir="auto">Which naturally results in error while loading <code class="notranslate">state_dict</code></p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="RuntimeError: Error(s) in loading state_dict for BertForMaskedLM:
	size mismatch for bert.embeddings.word_embeddings.weight: copying a param with shape torch.Size([119547, 768]) from checkpoint, the shape in current model is torch.Size([119554, 768]).
	size mismatch for cls.predictions.bias: copying a param with shape torch.Size([119547]) from checkpoint, the shape in current model is torch.Size([119554]).
	size mismatch for cls.predictions.decoder.weight: copying a param with shape torch.Size([119547, 768]) from checkpoint, the shape in current model is torch.Size([119554, 768])."><pre class="notranslate"><code class="notranslate">RuntimeError: Error(s) in loading state_dict for BertForMaskedLM:
	size mismatch for bert.embeddings.word_embeddings.weight: copying a param with shape torch.Size([119547, 768]) from checkpoint, the shape in current model is torch.Size([119554, 768]).
	size mismatch for cls.predictions.bias: copying a param with shape torch.Size([119547]) from checkpoint, the shape in current model is torch.Size([119554]).
	size mismatch for cls.predictions.decoder.weight: copying a param with shape torch.Size([119547, 768]) from checkpoint, the shape in current model is torch.Size([119554, 768]).
</code></pre></div>