<h3 dir="auto">System Info</h3>
<ul dir="auto">
<li>transformers version: <strong>4.30.2</strong></li>
<li>Python version: <strong>3.10.11</strong></li>
<li>System: Ubuntu 22.04.2 LTS</li>
</ul>
<h3 dir="auto">Who can help?</h3>
<p dir="auto"><a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/ArthurZucker/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/ArthurZucker">@ArthurZucker</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/younesbelkada/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/younesbelkada">@younesbelkada</a></p>
<h3 dir="auto">Information</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> The official example scripts</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> My own modified scripts</li>
</ul>
<h3 dir="auto">Tasks</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> An officially supported task in the <code class="notranslate">examples</code> folder (such as GLUE/SQuAD, ...)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> My own task or dataset (give details below)</li>
</ul>
<h3 dir="auto">Reproduction</h3>
<p dir="auto">For the following code,</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="model_name = 'TheBloke/guanaco-7B-HF'
model = AutoModel.from_pretrained(model_name, torch_dtype=torch.bfloat16, trust_remote_code=True)
model = model.to('cuda')

inputs = tokenizer(['hello'], max_length=100, truncation=True, return_tensors=&quot;pt&quot;).to(model.device)
outputs = model(**inputs)"><pre class="notranslate"><span class="pl-s1">model_name</span> <span class="pl-c1">=</span> <span class="pl-s">'TheBloke/guanaco-7B-HF'</span>
<span class="pl-s1">model</span> <span class="pl-c1">=</span> <span class="pl-v">AutoModel</span>.<span class="pl-en">from_pretrained</span>(<span class="pl-s1">model_name</span>, <span class="pl-s1">torch_dtype</span><span class="pl-c1">=</span><span class="pl-s1">torch</span>.<span class="pl-s1">bfloat16</span>, <span class="pl-s1">trust_remote_code</span><span class="pl-c1">=</span><span class="pl-c1">True</span>)
<span class="pl-s1">model</span> <span class="pl-c1">=</span> <span class="pl-s1">model</span>.<span class="pl-en">to</span>(<span class="pl-s">'cuda'</span>)

<span class="pl-s1">inputs</span> <span class="pl-c1">=</span> <span class="pl-en">tokenizer</span>([<span class="pl-s">'hello'</span>], <span class="pl-s1">max_length</span><span class="pl-c1">=</span><span class="pl-c1">100</span>, <span class="pl-s1">truncation</span><span class="pl-c1">=</span><span class="pl-c1">True</span>, <span class="pl-s1">return_tensors</span><span class="pl-c1">=</span><span class="pl-s">"pt"</span>).<span class="pl-en">to</span>(<span class="pl-s1">model</span>.<span class="pl-s1">device</span>)
<span class="pl-s1">outputs</span> <span class="pl-c1">=</span> <span class="pl-en">model</span>(<span class="pl-c1">**</span><span class="pl-s1">inputs</span>)</pre></div>
<p dir="auto">I got following error messages:</p>
<div class="highlight highlight-source-js notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[34], line 2
      1 inputs = tokenizer(['hello'], max_length=100, truncation=True, return_tensors=&quot;pt&quot;).to(model.device)
----&gt; 2 outputs = model(**inputs)

File ~/miniconda/envs/vocab/lib/python3.10/site-packages/torch/nn/modules/module.py:1501, in Module._call_impl(self, *args, **kwargs)
   1496 # If we don't have any hooks, we want to skip the rest of the logic in
   1497 # this function, and just call forward.
   1498 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
   1499         or _global_backward_pre_hooks or _global_backward_hooks
   1500         or _global_forward_hooks or _global_forward_pre_hooks):
-&gt; 1501     return forward_call(*args, **kwargs)
   1502 # Do not call functions when jit is used
   1503 full_backward_hooks, non_full_backward_hooks = [], []

TypeError: LlamaModel.forward() got an unexpected keyword argument 'token_type_ids'"><pre class="notranslate"><span class="pl-c1">--</span><span class="pl-c1">--</span><span class="pl-c1">--</span><span class="pl-c1">--</span><span class="pl-c1">--</span><span class="pl-c1">--</span><span class="pl-c1">--</span><span class="pl-c1">--</span><span class="pl-c1">--</span><span class="pl-c1">--</span><span class="pl-c1">--</span><span class="pl-c1">--</span><span class="pl-c1">--</span><span class="pl-c1">--</span><span class="pl-c1">--</span><span class="pl-c1">--</span><span class="pl-c1">--</span><span class="pl-c1">--</span><span class="pl-c1">--</span><span class="pl-c1">--</span><span class="pl-c1">--</span><span class="pl-c1">--</span><span class="pl-c1">--</span><span class="pl-c1">--</span><span class="pl-c1">--</span><span class="pl-c1">--</span><span class="pl-c1">--</span><span class="pl-c1">--</span><span class="pl-c1">--</span><span class="pl-c1">--</span><span class="pl-c1">--</span><span class="pl-c1">--</span><span class="pl-c1">--</span><span class="pl-c1">--</span><span class="pl-c1">--</span><span class="pl-c1">--</span><span class="pl-c1">--</span><span class="pl-c1">-</span>
<span class="pl-v">TypeError</span>                                 <span class="pl-v">Traceback</span> <span class="pl-kos">(</span><span class="pl-s1">most</span> <span class="pl-s1">recent</span> <span class="pl-s1">call</span> <span class="pl-s1">last</span><span class="pl-kos">)</span>
<span class="pl-v">Cell</span> <span class="pl-v">In</span><span class="pl-kos">[</span><span class="pl-c1">34</span><span class="pl-kos">]</span><span class="pl-kos">,</span> <span class="pl-s1">line</span> <span class="pl-c1">2</span>
      <span class="pl-c1">1</span><span class="pl-kos"></span> <span class="pl-s1">inputs</span> <span class="pl-c1">=</span> <span class="pl-en">tokenizer</span><span class="pl-kos">(</span><span class="pl-kos">[</span><span class="pl-s">'hello'</span><span class="pl-kos">]</span><span class="pl-kos">,</span> <span class="pl-s1">max_length</span><span class="pl-c1">=</span><span class="pl-c1">100</span><span class="pl-kos">,</span> <span class="pl-s1">truncation</span><span class="pl-c1">=</span><span class="pl-v">True</span><span class="pl-kos">,</span> <span class="pl-s1">return_tensors</span><span class="pl-c1">=</span><span class="pl-s">"pt"</span><span class="pl-kos">)</span><span class="pl-kos">.</span><span class="pl-en">to</span><span class="pl-kos">(</span><span class="pl-s1">model</span><span class="pl-kos">.</span><span class="pl-c1">device</span><span class="pl-kos">)</span>
<span class="pl-c1">--</span><span class="pl-c1">--</span><span class="pl-c1">&gt;</span> <span class="pl-c1">2</span> <span class="pl-s1">outputs</span> <span class="pl-c1">=</span> <span class="pl-en">model</span><span class="pl-kos">(</span><span class="pl-s1"></span><span class="pl-c1">**</span><span class="pl-s1">inputs</span><span class="pl-kos">)</span>

<span class="pl-v">File</span><span class="pl-kos"></span> <span class="pl-c1">~</span><span class="pl-pds"><span class="pl-c1">/</span>miniconda<span class="pl-c1">/</span>envs</span><span class="pl-c1">/</span><span class="pl-s1">vocab</span><span class="pl-c1">/</span><span class="pl-s1">lib</span><span class="pl-c1">/</span><span class="pl-s1">python3</span><span class="pl-c1">.10</span><span class="pl-c1">/</span><span class="pl-s1">site</span><span class="pl-c1">-</span><span class="pl-s1">packages</span><span class="pl-c1">/</span><span class="pl-s1">torch</span><span class="pl-c1">/</span><span class="pl-s1">nn</span><span class="pl-c1">/</span><span class="pl-s1">modules</span><span class="pl-c1">/</span><span class="pl-smi">module</span><span class="pl-kos">.</span><span class="pl-c1">py</span>:<span class="pl-c1">1501</span><span class="pl-kos">,</span> <span class="pl-s1">in</span> <span class="pl-v">Module</span><span class="pl-kos">.</span><span class="pl-en">_call_impl</span><span class="pl-kos">(</span><span class="pl-s1">self</span><span class="pl-kos">,</span> <span class="pl-c1">*</span><span class="pl-s1">args</span><span class="pl-kos">,</span> <span class="pl-c1">**</span><span class="pl-s1">kwargs</span><span class="pl-kos">)</span>
   <span class="pl-c1">1496</span> # <span class="pl-v">If</span> <span class="pl-s1">we</span> <span class="pl-s1">don</span><span class="pl-s">'t have any hooks, we want to skip the rest of the logic in</span>
<span class="pl-s">   1497 # this function, and just call forward.</span>
<span class="pl-s">   1498 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks</span>
<span class="pl-s">   1499         or _global_backward_pre_hooks or _global_backward_hooks</span>
<span class="pl-s">   1500         or _global_forward_hooks or _global_forward_pre_hooks):</span>
<span class="pl-s">-&gt; 1501     return forward_call(*args, **kwargs)</span>
<span class="pl-s">   1502 # Do not call functions when jit is used</span>
<span class="pl-s">   1503 full_backward_hooks, non_full_backward_hooks = [], []</span>
<span class="pl-s"></span>
<span class="pl-s">TypeError: LlamaModel.forward() got an unexpected keyword argument '</span><span class="pl-s1">token_type_ids</span>'</pre></div>
<p dir="auto">The <code class="notranslate">inputs</code> value is following:</p>
<div class="highlight highlight-source-js notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{'input_ids': tensor([[    1, 22172]], device='cuda:0'), 'token_type_ids': tensor([[0, 0]], device='cuda:0'), 'attention_mask': tensor([[1, 1]], device='cuda:0')}"><pre class="notranslate"><span class="pl-kos">{</span><span class="pl-s">'input_ids'</span>: <span class="pl-en">tensor</span><span class="pl-kos">(</span><span class="pl-kos">[</span><span class="pl-kos">[</span>    <span class="pl-c1">1</span><span class="pl-kos">,</span> <span class="pl-c1">22172</span><span class="pl-kos">]</span><span class="pl-kos">]</span><span class="pl-kos">,</span> <span class="pl-s1">device</span><span class="pl-c1">=</span><span class="pl-s">'cuda:0'</span><span class="pl-kos">)</span><span class="pl-kos">,</span> <span class="pl-s">'token_type_ids'</span>: <span class="pl-en">tensor</span><span class="pl-kos">(</span><span class="pl-kos">[</span><span class="pl-kos">[</span><span class="pl-c1">0</span><span class="pl-kos">,</span> <span class="pl-c1">0</span><span class="pl-kos">]</span><span class="pl-kos">]</span><span class="pl-kos">,</span> <span class="pl-s1">device</span><span class="pl-c1">=</span><span class="pl-s">'cuda:0'</span><span class="pl-kos">)</span><span class="pl-kos">,</span> <span class="pl-s">'attention_mask'</span>: <span class="pl-en">tensor</span><span class="pl-kos">(</span><span class="pl-kos">[</span><span class="pl-kos">[</span><span class="pl-c1">1</span><span class="pl-kos">,</span> <span class="pl-c1">1</span><span class="pl-kos">]</span><span class="pl-kos">]</span><span class="pl-kos">,</span> <span class="pl-s1">device</span><span class="pl-c1">=</span><span class="pl-s">'cuda:0'</span><span class="pl-kos">)</span><span class="pl-kos">}</span></pre></div>
<p dir="auto">There is <code class="notranslate">token_type_ids</code> in the value returned from <code class="notranslate">tokenizer</code>, however, <code class="notranslate">LlamaModel.forward()</code> don't accept the arguments.</p>
<p dir="auto">And I compared <code class="notranslate">LlamaTokenizer</code> and <code class="notranslate">LlamaTokenizerFast</code>, I found they behave differently.</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="from transformers import LlamaTokenizer, LlamaTokenizerFast
print(f&quot;LlamaTokenizer: {LlamaTokenizer.from_pretrained(model_name)('hello')}&quot;)
print(f&quot;LlamaTokenizerFast: {LlamaTokenizerFast.from_pretrained(model_name)('hello')}&quot;)"><pre class="notranslate"><span class="pl-k">from</span> <span class="pl-s1">transformers</span> <span class="pl-k">import</span> <span class="pl-v">LlamaTokenizer</span>, <span class="pl-v">LlamaTokenizerFast</span>
<span class="pl-en">print</span>(<span class="pl-s">f"LlamaTokenizer: <span class="pl-s1"><span class="pl-kos">{</span><span class="pl-v">LlamaTokenizer</span>.<span class="pl-en">from_pretrained</span>(<span class="pl-s1">model_name</span>)(<span class="pl-s">'hello'</span>)<span class="pl-kos">}</span></span>"</span>)
<span class="pl-en">print</span>(<span class="pl-s">f"LlamaTokenizerFast: <span class="pl-s1"><span class="pl-kos">{</span><span class="pl-v">LlamaTokenizerFast</span>.<span class="pl-en">from_pretrained</span>(<span class="pl-s1">model_name</span>)(<span class="pl-s">'hello'</span>)<span class="pl-kos">}</span></span>"</span>)</pre></div>
<p dir="auto">the results are:</p>
<div class="highlight highlight-source-js notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="LlamaTokenizer: {'input_ids': [1, 22172], 'attention_mask': [1, 1]}
LlamaTokenizerFast: {'input_ids': [1, 22172], 'token_type_ids': [0, 0], 'attention_mask': [1, 1]}"><pre class="notranslate">LlamaTokenizer: <span class="pl-kos">{</span><span class="pl-s">'input_ids'</span>: <span class="pl-kos">[</span><span class="pl-c1">1</span><span class="pl-kos">,</span> <span class="pl-c1">22172</span><span class="pl-kos">]</span><span class="pl-kos">,</span> <span class="pl-s">'attention_mask'</span>: <span class="pl-kos">[</span><span class="pl-c1">1</span><span class="pl-kos">,</span> <span class="pl-c1">1</span><span class="pl-kos">]</span><span class="pl-kos">}</span>
LlamaTokenizerFast: <span class="pl-kos">{</span><span class="pl-s">'input_ids'</span>: <span class="pl-kos">[</span><span class="pl-c1">1</span><span class="pl-kos">,</span> <span class="pl-c1">22172</span><span class="pl-kos">]</span><span class="pl-kos">,</span> <span class="pl-s">'token_type_ids'</span>: <span class="pl-kos">[</span><span class="pl-c1">0</span><span class="pl-kos">,</span> <span class="pl-c1">0</span><span class="pl-kos">]</span><span class="pl-kos">,</span> <span class="pl-s">'attention_mask'</span>: <span class="pl-kos">[</span><span class="pl-c1">1</span><span class="pl-kos">,</span> <span class="pl-c1">1</span><span class="pl-kos">]</span><span class="pl-kos">}</span></pre></div>
<h3 dir="auto">Expected behavior</h3>
<p dir="auto">Should <code class="notranslate">LlamaTokenizerFast</code> remove the <code class="notranslate">token_type_ids</code> in the returned value? or should <code class="notranslate">LlamaModel.forward()</code> accept the <code class="notranslate">token_type_ids</code> in the function arguments list? Thanks.</p>
<h3 dir="auto">System Info</h3>
<p dir="auto">platform==Ubuntu18.04<br>
python==3.10<br>
transformers==4.29.2</p>
<h3 dir="auto">Who can help?</h3>
<p dir="auto"><a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/ArthurZucker/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/ArthurZucker">@ArthurZucker</a></p>
<h3 dir="auto">Information</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> The official example scripts</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> My own modified scripts</li>
</ul>
<h3 dir="auto">Tasks</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> An officially supported task in the <code class="notranslate">examples</code> folder (such as GLUE/SQuAD, ...)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> My own task or dataset (give details below)</li>
</ul>
<h3 dir="auto">Reproduction</h3>
<p dir="auto"><code class="notranslate">&lt;/s&gt;</code> is the special token of LLaMATokenizer(Fast), it is expected that <code class="notranslate">&lt;/s&gt;</code> can be recognized as a single token when encoding the text. However, it can be shown that the two tokenizers behave differently:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="&gt;&gt;&gt; t1 = transformers.AutoTokenizer.from_pretrained(&quot;huggyllama/llama-7b&quot;, use_fast=True)
&gt;&gt;&gt; t2 = transformers.AutoTokenizer.from_pretrained(&quot;huggyllama/llama-7b&quot;, use_fast=False)
&gt;&gt;&gt; text = &quot;I love you.&lt;/s&gt;&quot;
&gt;&gt;&gt; t1(text)
&gt;&gt;&gt; {'input_ids': [1, 306, 5360, 366, 21106, 29879, 29958], 'token_type_ids': [0, 0, 0, 0, 0, 0, 0], 'attention_mask': [1, 1, 1, 1, 1, 1, 1]}
&gt;&gt;&gt; t2(text)
&gt;&gt;&gt; {'input_ids': [1, 306, 5360, 366, 29889, 2], 'attention_mask': [1, 1, 1, 1, 1, 1]}"><pre class="notranslate"><span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-s1">t1</span> <span class="pl-c1">=</span> <span class="pl-s1">transformers</span>.<span class="pl-v">AutoTokenizer</span>.<span class="pl-en">from_pretrained</span>(<span class="pl-s">"huggyllama/llama-7b"</span>, <span class="pl-s1">use_fast</span><span class="pl-c1">=</span><span class="pl-c1">True</span>)
<span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-s1">t2</span> <span class="pl-c1">=</span> <span class="pl-s1">transformers</span>.<span class="pl-v">AutoTokenizer</span>.<span class="pl-en">from_pretrained</span>(<span class="pl-s">"huggyllama/llama-7b"</span>, <span class="pl-s1">use_fast</span><span class="pl-c1">=</span><span class="pl-c1">False</span>)
<span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-s1">text</span> <span class="pl-c1">=</span> <span class="pl-s">"I love you.&lt;/s&gt;"</span>
<span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-en">t1</span>(<span class="pl-s1">text</span>)
<span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> {<span class="pl-s">'input_ids'</span>: [<span class="pl-c1">1</span>, <span class="pl-c1">306</span>, <span class="pl-c1">5360</span>, <span class="pl-c1">366</span>, <span class="pl-c1">21106</span>, <span class="pl-c1">29879</span>, <span class="pl-c1">29958</span>], <span class="pl-s">'token_type_ids'</span>: [<span class="pl-c1">0</span>, <span class="pl-c1">0</span>, <span class="pl-c1">0</span>, <span class="pl-c1">0</span>, <span class="pl-c1">0</span>, <span class="pl-c1">0</span>, <span class="pl-c1">0</span>], <span class="pl-s">'attention_mask'</span>: [<span class="pl-c1">1</span>, <span class="pl-c1">1</span>, <span class="pl-c1">1</span>, <span class="pl-c1">1</span>, <span class="pl-c1">1</span>, <span class="pl-c1">1</span>, <span class="pl-c1">1</span>]}
<span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-en">t2</span>(<span class="pl-s1">text</span>)
<span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> {<span class="pl-s">'input_ids'</span>: [<span class="pl-c1">1</span>, <span class="pl-c1">306</span>, <span class="pl-c1">5360</span>, <span class="pl-c1">366</span>, <span class="pl-c1">29889</span>, <span class="pl-c1">2</span>], <span class="pl-s">'attention_mask'</span>: [<span class="pl-c1">1</span>, <span class="pl-c1">1</span>, <span class="pl-c1">1</span>, <span class="pl-c1">1</span>, <span class="pl-c1">1</span>, <span class="pl-c1">1</span>]}</pre></div>
<p dir="auto">also, LLaMATokenizerFast returns <code class="notranslate">token_type_ids</code> but LLaMATokenizer does not.</p>
<h3 dir="auto">Expected behavior</h3>
<p dir="auto">LLaMATokenizerFast to be consistent with LLaMATokenzier.</p>