<p dir="auto">Slow tokenizers:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="&gt;&gt;&gt; import transformers
&gt;&gt;&gt; t_slow = transformers.AutoTokenizer.from_pretrained(&quot;roberta-base&quot;, use_fast=False)
&gt;&gt;&gt; t_slow.encode_plus(&quot;A &lt;mask&gt; sentence.&quot;)
{'input_ids': [0, 83, 50264, 3645, 4, 2],
 'token_type_ids': [0, 0, 0, 0, 0, 0],
 'attention_mask': [1, 1, 1, 1, 1, 1]}

&gt;&gt;&gt; t_slow.convert_ids_to_tokens([0, 83, 50264, 3645, 4, 2])
['&lt;s&gt;', 'ĠA', '&lt;mask&gt;', 'Ġsentence', '.', '&lt;/s&gt;']"><pre class="notranslate"><code class="notranslate">&gt;&gt;&gt; import transformers
&gt;&gt;&gt; t_slow = transformers.AutoTokenizer.from_pretrained("roberta-base", use_fast=False)
&gt;&gt;&gt; t_slow.encode_plus("A &lt;mask&gt; sentence.")
{'input_ids': [0, 83, 50264, 3645, 4, 2],
 'token_type_ids': [0, 0, 0, 0, 0, 0],
 'attention_mask': [1, 1, 1, 1, 1, 1]}

&gt;&gt;&gt; t_slow.convert_ids_to_tokens([0, 83, 50264, 3645, 4, 2])
['&lt;s&gt;', 'ĠA', '&lt;mask&gt;', 'Ġsentence', '.', '&lt;/s&gt;']
</code></pre></div>
<p dir="auto">Fast tokenizers:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="&gt;&gt;&gt; import transformers
&gt;&gt;&gt; t_fast = transformers.AutoTokenizer.from_pretrained(&quot;roberta-base&quot;, use_fast=True)
&gt;&gt;&gt; t_fast.encode_plus(&quot;A &lt;mask&gt; sentence.&quot;)
{'input_ids': [0, 250, 1437, 50264, 3645, 4, 2],
 'token_type_ids': [0, 0, 0, 0, 0, 0, 0],
 'attention_mask': [1, 1, 1, 1, 1, 1, 1]}

&gt;&gt;&gt; t_fast.convert_ids_to_tokens([0, 250, 1437, 50264, 3645, 4, 2])
['&lt;s&gt;', 'A', 'Ġ', '&lt;mask&gt;', 'Ġsentence', '.', '&lt;/s&gt;'] "><pre class="notranslate"><code class="notranslate">&gt;&gt;&gt; import transformers
&gt;&gt;&gt; t_fast = transformers.AutoTokenizer.from_pretrained("roberta-base", use_fast=True)
&gt;&gt;&gt; t_fast.encode_plus("A &lt;mask&gt; sentence.")
{'input_ids': [0, 250, 1437, 50264, 3645, 4, 2],
 'token_type_ids': [0, 0, 0, 0, 0, 0, 0],
 'attention_mask': [1, 1, 1, 1, 1, 1, 1]}

&gt;&gt;&gt; t_fast.convert_ids_to_tokens([0, 250, 1437, 50264, 3645, 4, 2])
['&lt;s&gt;', 'A', 'Ġ', '&lt;mask&gt;', 'Ġsentence', '.', '&lt;/s&gt;'] 
</code></pre></div>
<h1 dir="auto"><g-emoji class="g-emoji" alias="bug" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/1f41b.png">🐛</g-emoji> Bug</h1>
<h2 dir="auto">Information</h2>
<p dir="auto">Model I am using (Bert, XLNet ...): Bert</p>
<p dir="auto">Language I am using the model on (English, Chinese ...): English</p>
<p dir="auto">The problem arises when using:</p>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> the official example scripts: (give details below)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> my own modified scripts: (give details below)</li>
</ul>
<p dir="auto">The tasks I am working on is:</p>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> an official GLUE/SQUaD task: (give the name)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> my own task or dataset: (give details below)</li>
</ul>
<h2 dir="auto">To reproduce</h2>
<p dir="auto">Steps to reproduce the behavior:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="from transformers import AutoTokenizer

pretrained_model_name = &quot;bert-base-cased&quot;

fast_tokenizer = AutoTokenizer.from_pretrained(pretrained_model_name)
slow_tokenizer = AutoTokenizer.from_pretrained(pretrained_model_name, use_fast=False)

text = &quot;hello&quot;

assert fast_tokenizer.encode(text, add_special_tokens=False) == slow_tokenizer.encode(text, add_special_tokens=False)"><pre class="notranslate"><span class="pl-k">from</span> <span class="pl-s1">transformers</span> <span class="pl-k">import</span> <span class="pl-v">AutoTokenizer</span>

<span class="pl-s1">pretrained_model_name</span> <span class="pl-c1">=</span> <span class="pl-s">"bert-base-cased"</span>

<span class="pl-s1">fast_tokenizer</span> <span class="pl-c1">=</span> <span class="pl-v">AutoTokenizer</span>.<span class="pl-en">from_pretrained</span>(<span class="pl-s1">pretrained_model_name</span>)
<span class="pl-s1">slow_tokenizer</span> <span class="pl-c1">=</span> <span class="pl-v">AutoTokenizer</span>.<span class="pl-en">from_pretrained</span>(<span class="pl-s1">pretrained_model_name</span>, <span class="pl-s1">use_fast</span><span class="pl-c1">=</span><span class="pl-c1">False</span>)

<span class="pl-s1">text</span> <span class="pl-c1">=</span> <span class="pl-s">"hello"</span>

<span class="pl-k">assert</span> <span class="pl-s1">fast_tokenizer</span>.<span class="pl-en">encode</span>(<span class="pl-s1">text</span>, <span class="pl-s1">add_special_tokens</span><span class="pl-c1">=</span><span class="pl-c1">False</span>) <span class="pl-c1">==</span> <span class="pl-s1">slow_tokenizer</span>.<span class="pl-en">encode</span>(<span class="pl-s1">text</span>, <span class="pl-s1">add_special_tokens</span><span class="pl-c1">=</span><span class="pl-c1">False</span>)</pre></div>
<h2 dir="auto">Expected behavior</h2>
<p dir="auto">The fast tokenizers shouldn't add the special tokens if <code class="notranslate">add_special_tokens</code> is equal to <code class="notranslate">False</code>.</p>
<h2 dir="auto">Environment info</h2>

<ul dir="auto">
<li><code class="notranslate">transformers</code> version: 2.5.0</li>
<li>Platform: Linux-4.15.0-76-generic-x86_64-with-debian-buster-sid</li>
<li>Python version: 3.7.4</li>
<li>PyTorch version (GPU?): 1.4.0 (True)</li>
<li>Tensorflow version (GPU?): 2.0.0 (False)</li>
<li>Using GPU in script?: No</li>
<li>Using distributed or parallel set-up in script?: No</li>
</ul>