<p dir="auto">Example:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="&gt;&gt;&gt; import transformers
&gt;&gt;&gt; t_fast = transformers.AutoTokenizer.from_pretrained(&quot;bert-base-cased&quot;, use_fast=True, add_special_tokens=False)
&gt;&gt;&gt; sentence = &quot;A, naïve [MASK] AllenNLP sentence.&quot;
&gt;&gt;&gt; tokenized = t_fast.encode_plus(sentence, add_special_tokens=False, return_offsets_mapping=True)
&gt;&gt;&gt; for start, end in tokenized['offset_mapping']:
...     print(repr(sentence[start:end]))
'A'
','
'naïve'
' [MASK'
' Alle'
'nN'
'L'
' sentenc'
'e'"><pre class="notranslate"><code class="notranslate">&gt;&gt;&gt; import transformers
&gt;&gt;&gt; t_fast = transformers.AutoTokenizer.from_pretrained("bert-base-cased", use_fast=True, add_special_tokens=False)
&gt;&gt;&gt; sentence = "A, naïve [MASK] AllenNLP sentence."
&gt;&gt;&gt; tokenized = t_fast.encode_plus(sentence, add_special_tokens=False, return_offsets_mapping=True)
&gt;&gt;&gt; for start, end in tokenized['offset_mapping']:
...     print(repr(sentence[start:end]))
'A'
','
'naïve'
' [MASK'
' Alle'
'nN'
'L'
' sentenc'
'e'
</code></pre></div>
<p dir="auto">As you can see, after the word "naïve", the offsets go off the rails.</p>
<h1 dir="auto"><g-emoji class="g-emoji" alias="bug" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/1f41b.png">🐛</g-emoji> Bug</h1>
<h2 dir="auto">Information</h2>
<p dir="auto">Model I am using (Bert, XLNet ...): Bert (could happen with other ones, don't know)</p>
<p dir="auto">Language I am using the model on (English, Chinese ...): English</p>
<p dir="auto">The problem arises when using:</p>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> the official example scripts: (give details below)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> my own modified scripts: (give details below)</li>
</ul>
<p dir="auto">The tasks I am working on is:</p>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> an official GLUE/SQUaD task: (give the name)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> my own task or dataset: (give details below)</li>
</ul>
<h2 dir="auto">To reproduce</h2>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="from transformers import AutoTokenizer

pretrained_model_name = &quot;bert-base-cased&quot;

fast_tokenizer = AutoTokenizer.from_pretrained(pretrained_model_name)
slow_tokenizer = AutoTokenizer.from_pretrained(pretrained_model_name, use_fast=False)

text = &quot;naïve&quot;

assert fast_tokenizer.encode(text) == slow_tokenizer.encode(text)"><pre class="notranslate"><span class="pl-k">from</span> <span class="pl-s1">transformers</span> <span class="pl-k">import</span> <span class="pl-v">AutoTokenizer</span>

<span class="pl-s1">pretrained_model_name</span> <span class="pl-c1">=</span> <span class="pl-s">"bert-base-cased"</span>

<span class="pl-s1">fast_tokenizer</span> <span class="pl-c1">=</span> <span class="pl-v">AutoTokenizer</span>.<span class="pl-en">from_pretrained</span>(<span class="pl-s1">pretrained_model_name</span>)
<span class="pl-s1">slow_tokenizer</span> <span class="pl-c1">=</span> <span class="pl-v">AutoTokenizer</span>.<span class="pl-en">from_pretrained</span>(<span class="pl-s1">pretrained_model_name</span>, <span class="pl-s1">use_fast</span><span class="pl-c1">=</span><span class="pl-c1">False</span>)

<span class="pl-s1">text</span> <span class="pl-c1">=</span> <span class="pl-s">"naïve"</span>

<span class="pl-k">assert</span> <span class="pl-s1">fast_tokenizer</span>.<span class="pl-en">encode</span>(<span class="pl-s1">text</span>) <span class="pl-c1">==</span> <span class="pl-s1">slow_tokenizer</span>.<span class="pl-en">encode</span>(<span class="pl-s1">text</span>)</pre></div>
<p dir="auto">With the slow, it only strips accents if lowercase is enabled (maybe a bug?):</p>
<p dir="auto"></p><div class="Box Box--condensed my-2">
  <div class="Box-header f6">
    <p class="mb-0 text-bold">
      <a href="https://github.com/huggingface/transformers/blob/e67676424191e5935362e5fe7e04b5c317d706a9/src/transformers/tokenization_bert.py#L346">transformers/src/transformers/tokenization_bert.py</a>
    </p>
    <p class="mb-0 color-fg-muted">
         Line 346
      in
      <a data-pjax="true" class="commit-tease-sha" href="/huggingface/transformers/commit/e67676424191e5935362e5fe7e04b5c317d706a9">e676764</a>
    </p>
  </div>
  <div itemprop="text" class="Box-body p-0 blob-wrapper blob-wrapper-embedded data">
    <table class="highlight tab-size mb-0 js-file-line-container" data-tab-size="8" data-paste-markdown-skip="">

        <tbody><tr class="border-0">
          <td id="L346" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="346"></td>
          <td id="LC346" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line"> <span class="pl-s1">token</span> <span class="pl-c1">=</span> <span class="pl-s1">self</span>.<span class="pl-en">_run_strip_accents</span>(<span class="pl-s1">token</span>) </td>
        </tr>
    </tbody></table>
  </div>
</div>
<p></p>
<p dir="auto">With the fast one, it'd never strip accents:</p>
<p dir="auto"><a href="https://github.com/huggingface/tokenizers/blob/python-v0.5.0/bindings/python/tokenizers/implementations/bert_wordpiece.py#L23">https://github.com/huggingface/tokenizers/blob/python-v0.5.0/bindings/python/tokenizers/implementations/bert_wordpiece.py#L23</a></p>
<p dir="auto"></p><div class="Box Box--condensed my-2">
  <div class="Box-header f6">
    <p class="mb-0 text-bold">
      <a href="https://github.com/huggingface/transformers/blob/e67676424191e5935362e5fe7e04b5c317d706a9/src/transformers/tokenization_bert.py#L557-L565">transformers/src/transformers/tokenization_bert.py</a>
    </p>
    <p class="mb-0 color-fg-muted">
        Lines 557 to 565
      in
      <a data-pjax="true" class="commit-tease-sha" href="/huggingface/transformers/commit/e67676424191e5935362e5fe7e04b5c317d706a9">e676764</a>
    </p>
  </div>
  <div itemprop="text" class="Box-body p-0 blob-wrapper blob-wrapper-embedded data">
    <table class="highlight tab-size mb-0 js-file-line-container" data-tab-size="8" data-paste-markdown-skip="">

        <tbody><tr class="border-0">
          <td id="L557" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="557"></td>
          <td id="LC557" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line"> <span class="pl-v">BertWordPieceTokenizer</span>( </td>
        </tr>

        <tr class="border-0">
          <td id="L558" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="558"></td>
          <td id="LC558" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">     <span class="pl-s1">vocab_file</span><span class="pl-c1">=</span><span class="pl-s1">vocab_file</span>, </td>
        </tr>

        <tr class="border-0">
          <td id="L559" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="559"></td>
          <td id="LC559" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">     <span class="pl-s1">add_special_tokens</span><span class="pl-c1">=</span><span class="pl-s1">add_special_tokens</span>, </td>
        </tr>

        <tr class="border-0">
          <td id="L560" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="560"></td>
          <td id="LC560" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">     <span class="pl-s1">unk_token</span><span class="pl-c1">=</span><span class="pl-s1">unk_token</span>, </td>
        </tr>

        <tr class="border-0">
          <td id="L561" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="561"></td>
          <td id="LC561" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">     <span class="pl-s1">sep_token</span><span class="pl-c1">=</span><span class="pl-s1">sep_token</span>, </td>
        </tr>

        <tr class="border-0">
          <td id="L562" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="562"></td>
          <td id="LC562" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">     <span class="pl-s1">cls_token</span><span class="pl-c1">=</span><span class="pl-s1">cls_token</span>, </td>
        </tr>

        <tr class="border-0">
          <td id="L563" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="563"></td>
          <td id="LC563" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">     <span class="pl-s1">handle_chinese_chars</span><span class="pl-c1">=</span><span class="pl-s1">tokenize_chinese_chars</span>, </td>
        </tr>

        <tr class="border-0">
          <td id="L564" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="564"></td>
          <td id="LC564" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">     <span class="pl-s1">lowercase</span><span class="pl-c1">=</span><span class="pl-s1">do_lower_case</span>, </td>
        </tr>

        <tr class="border-0">
          <td id="L565" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="565"></td>
          <td id="LC565" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line"> ), </td>
        </tr>
    </tbody></table>
  </div>
</div>
<p></p>
<p dir="auto">I'd be cool to have that flag also, in both tokenizers.</p>
<p dir="auto">Finally, this warning seems odd for the simple code from above:</p>
<div class="highlight highlight-text-python-console notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="&gt;&gt;&gt; assert fast_tokenizer.encode(text) == slow_tokenizer.encode(text)
Disabled padding because no padding token set (pad_token: [PAD], pad_token_id: 0).
To remove this error, you can add a new pad token and then resize model embedding:
	tokenizer.pad_token = '&lt;PAD&gt;'
	model.resize_token_embeddings(len(tokenizer))
Traceback (most recent call last):
  File &quot;&lt;stdin&gt;&quot;, line 1, in &lt;module&gt;
AssertionError"><pre class="notranslate">&gt;&gt;&gt; <span class="pl-k">assert</span> fast_tokenizer.encode(text) <span class="pl-k">==</span> slow_tokenizer.encode(text)
Disabled padding because no padding token set (pad_token: [PAD], pad_token_id: 0).
To remove this error, you can add a new pad token and then resize model embedding:
	tokenizer.pad_token = '&lt;PAD&gt;'
	model.resize_token_embeddings(len(tokenizer))
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
AssertionError</pre></div>
<p dir="auto">Maybe here the <code class="notranslate">if pad_to_max_length</code> should be nesting the rest of the if?</p>
<p dir="auto"></p><div class="Box Box--condensed my-2">
  <div class="Box-header f6">
    <p class="mb-0 text-bold">
      <a href="https://github.com/huggingface/transformers/blob/e67676424191e5935362e5fe7e04b5c317d706a9/src/transformers/tokenization_utils.py#L80-L95">transformers/src/transformers/tokenization_utils.py</a>
    </p>
    <p class="mb-0 color-fg-muted">
        Lines 80 to 95
      in
      <a data-pjax="true" class="commit-tease-sha" href="/huggingface/transformers/commit/e67676424191e5935362e5fe7e04b5c317d706a9">e676764</a>
    </p>
  </div>
  <div itemprop="text" class="Box-body p-0 blob-wrapper blob-wrapper-embedded data">
    <table class="highlight tab-size mb-0 js-file-line-container" data-tab-size="8" data-paste-markdown-skip="">

        <tbody><tr class="border-0">
          <td id="L80" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="80"></td>
          <td id="LC80" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line"> <span class="pl-k">if</span> <span class="pl-s1">pad_to_max_length</span> <span class="pl-c1">and</span> (<span class="pl-s1">pad_token</span> <span class="pl-c1">and</span> <span class="pl-s1">pad_token_id</span> <span class="pl-c1">&gt;=</span> <span class="pl-c1">0</span>): </td>
        </tr>

        <tr class="border-0">
          <td id="L81" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="81"></td>
          <td id="LC81" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">     <span class="pl-s1">tokenizer</span>.<span class="pl-en">enable_padding</span>( </td>
        </tr>

        <tr class="border-0">
          <td id="L82" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="82"></td>
          <td id="LC82" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">         <span class="pl-s1">max_length</span><span class="pl-c1">=</span><span class="pl-c1">None</span>, </td>
        </tr>

        <tr class="border-0">
          <td id="L83" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="83"></td>
          <td id="LC83" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">         <span class="pl-s1">direction</span><span class="pl-c1">=</span><span class="pl-s1">padding_side</span>, </td>
        </tr>

        <tr class="border-0">
          <td id="L84" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="84"></td>
          <td id="LC84" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">         <span class="pl-s1">pad_id</span><span class="pl-c1">=</span><span class="pl-s1">pad_token_id</span>, </td>
        </tr>

        <tr class="border-0">
          <td id="L85" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="85"></td>
          <td id="LC85" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">         <span class="pl-s1">pad_type_id</span><span class="pl-c1">=</span><span class="pl-s1">pad_token_type_id</span>, </td>
        </tr>

        <tr class="border-0">
          <td id="L86" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="86"></td>
          <td id="LC86" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">         <span class="pl-s1">pad_token</span><span class="pl-c1">=</span><span class="pl-s1">pad_token</span>, </td>
        </tr>

        <tr class="border-0">
          <td id="L87" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="87"></td>
          <td id="LC87" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">     ) </td>
        </tr>

        <tr class="border-0">
          <td id="L88" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="88"></td>
          <td id="LC88" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line"> <span class="pl-k">else</span>: </td>
        </tr>

        <tr class="border-0">
          <td id="L89" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="89"></td>
          <td id="LC89" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">     <span class="pl-s1">logger</span>.<span class="pl-en">warning</span>( </td>
        </tr>

        <tr class="border-0">
          <td id="L90" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="90"></td>
          <td id="LC90" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">         <span class="pl-s">"Disabled padding because no padding token set (pad_token: {}, pad_token_id: {}).<span class="pl-cce">\n</span>"</span> </td>
        </tr>

        <tr class="border-0">
          <td id="L91" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="91"></td>
          <td id="LC91" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">         <span class="pl-s">"To remove this error, you can add a new pad token and then resize model embedding:<span class="pl-cce">\n</span>"</span> </td>
        </tr>

        <tr class="border-0">
          <td id="L92" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="92"></td>
          <td id="LC92" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">         <span class="pl-s">"<span class="pl-cce">\t</span>tokenizer.pad_token = '&lt;PAD&gt;'<span class="pl-cce">\n</span><span class="pl-cce">\t</span>model.resize_token_embeddings(len(tokenizer))"</span>.<span class="pl-en">format</span>( </td>
        </tr>

        <tr class="border-0">
          <td id="L93" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="93"></td>
          <td id="LC93" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">             <span class="pl-s1">pad_token</span>, <span class="pl-s1">pad_token_id</span> </td>
        </tr>

        <tr class="border-0">
          <td id="L94" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="94"></td>
          <td id="LC94" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">         ) </td>
        </tr>

        <tr class="border-0">
          <td id="L95" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="95"></td>
          <td id="LC95" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">     ) </td>
        </tr>
    </tbody></table>
  </div>
</div>
<p></p>
<p dir="auto">Didn't check in the other transformer models.</p>
<h2 dir="auto">Expected behavior</h2>
<ol dir="auto">
<li>The 2 tokenizer outputs (slow and fast) should be the same.</li>
<li>The tokenizers should allow you to choose if to strip accents or not.</li>
<li>That warning shouldn't appear, IMHO.</li>
</ol>
<h2 dir="auto">Environment info</h2>
<ul dir="auto">
<li><code class="notranslate">transformers</code> version: 2.5.0</li>
<li>Platform: Linux-4.15.0-76-generic-x86_64-with-debian-buster-sid</li>
<li>Python version: 3.7.4</li>
<li>PyTorch version (GPU?): 1.4.0 (True)</li>
<li>Tensorflow version (GPU?): 2.0.0 (False)</li>
<li>Using GPU in script?: No</li>
<li>Using distributed or parallel set-up in script?: No</li>
</ul>