<h2 dir="auto"><g-emoji class="g-emoji" alias="question" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/2753.png">❓</g-emoji> Questions &amp; Help</h2>

<p dir="auto">When using En-Fr XLMModel in transformers library,<br>
I found that result from tokenizing a whole sentence is different from tokenizing words in the sentence and then concatenate.</p>
<p dir="auto">My configuration is as below<br>
<strong>(XLMModel, XLMTokenizer, XLMConfig, 'xlm-mlm-enfr-1024')</strong></p>
<p dir="auto">The result is as below<br>
<a target="_blank" rel="noopener noreferrer nofollow" href="https://user-images.githubusercontent.com/12389429/71879066-84a93000-3170-11ea-91fe-25227ebe478e.png"><img src="https://user-images.githubusercontent.com/12389429/71879066-84a93000-3170-11ea-91fe-25227ebe478e.png" alt="image" style="max-width: 100%;"></a></p>
<p dir="auto">The ultimate goal is to 'detokenize' tokenized sentence which is<br>
['I', 'love', 'swim', '##ing'] -&gt; ['I', 'love', 'swimming']<br>
In order to do this, I have to know the raw token's index for each tokenized tokens.</p>
<p dir="auto">It would be great if anyone can help with this problem.</p>
<h2 dir="auto">🐛 Bug</h2>
<p dir="auto">Model I am using: <strong>Bert</strong><br>
Language I am using the model on: <strong>English</strong></p>
<h2 dir="auto">To Reproduce</h2>
<p dir="auto">Call <code class="notranslate">bertTokenizer.tokenize("text", return_tokens_mapped_to_origin=True)</code><br>
Result:</p>
<blockquote>
<p dir="auto">TypeError: _tokenize() got an unexpected keyword argument 'return_tokens_mapped_to_origin'</p>
</blockquote>
<h2 dir="auto">Expected behavior</h2>
<p dir="auto">The official documentation mentions a "return_tokens_mapped_to_origin" optional parameter that when set to True should return the index of each token in the initial given text.<br>
<a href="https://huggingface.co/transformers/main_classes/tokenizer.html?highlight=return_tokens_mapped_to_origin#transformers.PreTrainedTokenizer.tokenize" rel="nofollow">https://huggingface.co/transformers/main_classes/tokenizer.html?highlight=return_tokens_mapped_to_origin#transformers.PreTrainedTokenizer.tokenize</a></p>
<h2 dir="auto">Environment</h2>
<ul dir="auto">
<li>OS: macOS Mojave</li>
<li>Python version: 3.7</li>
<li>PyTorch version: 1.3.0</li>
<li>PyTorch Transformers version (or branch): 2.2.1</li>
<li>Using GPU ? No</li>
</ul>
<h2 dir="auto">Additional context</h2>
<p dir="auto">In the source code this parameter is never used outside of the doc comment, neither in the base class nor in its implementations.</p>