<h2 dir="auto">Environment info</h2>
<ul dir="auto">
<li><code class="notranslate">transformers</code> version: 4.5.0</li>
<li>Platform: Windows 10 (replicated on Ubuntu 18.04 as well)</li>
<li>Python version: 3.6.8</li>
<li>PyTorch version (GPU?): n/a</li>
<li>Tensorflow version (GPU?): n/a</li>
<li>Using GPU in script?: no</li>
<li>Using distributed or parallel set-up in script?: no</li>
</ul>
<p dir="auto">Full package list:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="certifi==2020.12.5
chardet==4.0.0
click==7.1.2
dataclasses==0.8
filelock==3.0.12
idna==2.10
importlib-metadata==3.10.0
joblib==1.0.1
numpy==1.19.5
packaging==20.9
protobuf==3.15.7
pyparsing==2.4.7
regex==2021.4.4
requests==2.25.1
sacremoses==0.0.44
sentencepiece==0.1.95
six==1.15.0
tokenizers==0.10.2
tqdm==4.60.0
transformers==4.5.0
typing-extensions==3.7.4.3
urllib3==1.26.4
zipp==3.4.1"><pre class="notranslate"><code class="notranslate">certifi==2020.12.5
chardet==4.0.0
click==7.1.2
dataclasses==0.8
filelock==3.0.12
idna==2.10
importlib-metadata==3.10.0
joblib==1.0.1
numpy==1.19.5
packaging==20.9
protobuf==3.15.7
pyparsing==2.4.7
regex==2021.4.4
requests==2.25.1
sacremoses==0.0.44
sentencepiece==0.1.95
six==1.15.0
tokenizers==0.10.2
tqdm==4.60.0
transformers==4.5.0
typing-extensions==3.7.4.3
urllib3==1.26.4
zipp==3.4.1
</code></pre></div>
<h3 dir="auto">Who can help</h3>
<p dir="auto"><a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/LysandreJik/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/LysandreJik">@LysandreJik</a></p>
<h2 dir="auto">Information</h2>
<p dir="auto">Model I am using (Bert, XLNet ...): n/a</p>
<p dir="auto">The problem arises when using:</p>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> the official example scripts: (give details below)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> my own modified scripts: (give details below)</li>
</ul>
<p dir="auto">The tasks I am working on is:</p>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> an official GLUE/SQUaD task: (give the name)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> my own task or dataset: (give details below)</li>
</ul>
<h2 dir="auto">To reproduce</h2>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="&gt;&gt;&gt; tokenizer = XLMRobertaTokenizerFast.from_pretrained('xlm-roberta-base')
&gt;&gt;&gt; tokenizer.save_pretrained('test')
('test\\tokenizer_config.json', 'test\\special_tokens_map.json', 'test\\sentencepiece.bpe.model', 'test\\added_tokens.json')
&gt;&gt;&gt; tokenizer_loaded = XLMRobertaTokenizerFast.from_pretrained('test')
&gt;&gt;&gt; tokenizer('hello world', return_offsets_mapping=True)
{'input_ids': [0, 33600, 31, 8999, 2], 'attention_mask': [1, 1, 1, 1, 1], 'offset_mapping': [(0, 0), (0, 4), (4, 5), (6, 11), (0, 0)]}
&gt;&gt;&gt; tokenizer_loaded('hello world', return_offsets_mapping=True)
{'input_ids': [0, 33600, 31, 8999, 2], 'attention_mask': [1, 1, 1, 1, 1], 'offset_mapping': [(0, 0), (0, 4), (4, 5), (5, 11), (0, 0)]}"><pre class="notranslate"><span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-s1">tokenizer</span> <span class="pl-c1">=</span> <span class="pl-v">XLMRobertaTokenizerFast</span>.<span class="pl-en">from_pretrained</span>(<span class="pl-s">'xlm-roberta-base'</span>)
<span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-s1">tokenizer</span>.<span class="pl-en">save_pretrained</span>(<span class="pl-s">'test'</span>)
(<span class="pl-s">'test<span class="pl-cce">\\</span>tokenizer_config.json'</span>, <span class="pl-s">'test<span class="pl-cce">\\</span>special_tokens_map.json'</span>, <span class="pl-s">'test<span class="pl-cce">\\</span>sentencepiece.bpe.model'</span>, <span class="pl-s">'test<span class="pl-cce">\\</span>added_tokens.json'</span>)
<span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-s1">tokenizer_loaded</span> <span class="pl-c1">=</span> <span class="pl-v">XLMRobertaTokenizerFast</span>.<span class="pl-en">from_pretrained</span>(<span class="pl-s">'test'</span>)
<span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-en">tokenizer</span>(<span class="pl-s">'hello world'</span>, <span class="pl-s1">return_offsets_mapping</span><span class="pl-c1">=</span><span class="pl-c1">True</span>)
{<span class="pl-s">'input_ids'</span>: [<span class="pl-c1">0</span>, <span class="pl-c1">33600</span>, <span class="pl-c1">31</span>, <span class="pl-c1">8999</span>, <span class="pl-c1">2</span>], <span class="pl-s">'attention_mask'</span>: [<span class="pl-c1">1</span>, <span class="pl-c1">1</span>, <span class="pl-c1">1</span>, <span class="pl-c1">1</span>, <span class="pl-c1">1</span>], <span class="pl-s">'offset_mapping'</span>: [(<span class="pl-c1">0</span>, <span class="pl-c1">0</span>), (<span class="pl-c1">0</span>, <span class="pl-c1">4</span>), (<span class="pl-c1">4</span>, <span class="pl-c1">5</span>), (<span class="pl-c1">6</span>, <span class="pl-c1">11</span>), (<span class="pl-c1">0</span>, <span class="pl-c1">0</span>)]}
<span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-en">tokenizer_loaded</span>(<span class="pl-s">'hello world'</span>, <span class="pl-s1">return_offsets_mapping</span><span class="pl-c1">=</span><span class="pl-c1">True</span>)
{<span class="pl-s">'input_ids'</span>: [<span class="pl-c1">0</span>, <span class="pl-c1">33600</span>, <span class="pl-c1">31</span>, <span class="pl-c1">8999</span>, <span class="pl-c1">2</span>], <span class="pl-s">'attention_mask'</span>: [<span class="pl-c1">1</span>, <span class="pl-c1">1</span>, <span class="pl-c1">1</span>, <span class="pl-c1">1</span>, <span class="pl-c1">1</span>], <span class="pl-s">'offset_mapping'</span>: [(<span class="pl-c1">0</span>, <span class="pl-c1">0</span>), (<span class="pl-c1">0</span>, <span class="pl-c1">4</span>), (<span class="pl-c1">4</span>, <span class="pl-c1">5</span>), (<span class="pl-c1">5</span>, <span class="pl-c1">11</span>), (<span class="pl-c1">0</span>, <span class="pl-c1">0</span>)]}</pre></div>
<h2 dir="auto">Expected behavior</h2>
<p dir="auto">The loaded tokenizer should produce the same offset mapping as the original tokenizer. It seems like the tokenizer loaded from disk does not account for spaces in its offset mapping.</p>
<h2 dir="auto">Environment info</h2>
<ul dir="auto">
<li>transformers` version: 4.2.1</li>
<li>Platform: Linux-4.19.112+-x86_64-with-Ubuntu-18.04-bionic</li>
<li>Python version: 3.6.9</li>
<li>PyTorch version (GPU?): 1.7.0+cu101 (False)</li>
<li>Tensorflow version (GPU?): 2.4.0 (False)</li>
<li>Using GPU in script?: yes</li>
<li>Using distributed or parallel set-up in script?: no</li>
</ul>
<h3 dir="auto">Who can help</h3>
<p dir="auto"><a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/mfuntowicz/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/mfuntowicz">@mfuntowicz</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/stefan-it/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/stefan-it">@stefan-it</a></p>
<h2 dir="auto">Information</h2>
<p dir="auto">Model I am using is XLM-RoBERTa.<br>
The problem arises when using XLMRobertaTokenizerFast tokenizer.</p>
<p dir="auto">The tasks I am working on is token-classification. In order to align the labels with the sub-word units I have used the code snippet provided here: <a href="https://huggingface.co/transformers/custom_datasets.html" rel="nofollow">https://huggingface.co/transformers/custom_datasets.html</a> [ Fine-tuning with custom datasets/Token Classification with W-NUT Emerging Entities ].</p>
<p dir="auto">When trying to align the labels with the encodings, it throws: "ValueError: NumPy boolean array indexing assignment cannot assign X input values to the Y output values where the mask is true."</p>
<p dir="auto">This behavior is due to tokenizing punctuation. Moreover comma ( ' , ' ) gets tokenized into '<strong>' and ',' ( having offset values (0,1) ) Similar behavior happens with dot. However, some other punctuation marks are producing only one token (i.g.  ' : ' -&gt; '</strong>:').<br>
In addition, the offset_mapping value for ':' is different in different sentences resulting either in (0,0) or (0,3) tuple. The problem is that padding tokens have offset tuple with values (0,0) which are excluded from alignment, but in this case I have to preserve the punctuation since it is POS tagging problem.</p>
<h2 dir="auto">To reproduce</h2>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="print(&quot;Token: {}  Offset_mapping: {}&quot;.format(train_encodings[338].tokens[67], train_encodings[338].offsets[67]))
# Token: ▁...  Offset_mapping: (0, 0)
print(&quot;Token: {}  Offset_mapping: {}&quot;.format(train_encodings[20].tokens[2], train_encodings[20].offsets[2]))
# Token: ▁...  Offset_mapping: (0, 3)
"><pre class="notranslate"><code class="notranslate">print("Token: {}  Offset_mapping: {}".format(train_encodings[338].tokens[67], train_encodings[338].offsets[67]))
# Token: ▁...  Offset_mapping: (0, 0)
print("Token: {}  Offset_mapping: {}".format(train_encodings[20].tokens[2], train_encodings[20].offsets[2]))
# Token: ▁...  Offset_mapping: (0, 3)

</code></pre></div>
<p dir="auto">Moreover, although I fixed this issue by writing my own masks, I found new issue: the blank space which denotes start of the word is tokenized as separate token instead of being together with the starting sub-token.</p>
<h2 dir="auto">To reproduce</h2>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="tokenizer = XLMRobertaTokenizerFast.from_pretrained(&quot;xlm-roberta-base&quot;)
model= XLMRobertaForTokenClassification.from_pretrained(&quot;xlm-roberta-base&quot;)
s = &quot;Je često kritizirao vladu .&quot;
print(tokenizer.tokenize(s)) 
# output: ['▁Je', '▁često', '▁krit', 'izira', 'o', '▁', 'vlad', 'u', '▁', '.']"><pre class="notranslate"><code class="notranslate">tokenizer = XLMRobertaTokenizerFast.from_pretrained("xlm-roberta-base")
model= XLMRobertaForTokenClassification.from_pretrained("xlm-roberta-base")
s = "Je često kritizirao vladu ."
print(tokenizer.tokenize(s)) 
# output: ['▁Je', '▁često', '▁krit', 'izira', 'o', '▁', 'vlad', 'u', '▁', '.']
</code></pre></div>
<h2 dir="auto">Expected behavior</h2>
<ol dir="auto">
<li>Punctuation marks should be consistently tokenized and having offset values different from padding tokens.</li>
<li>The first sub-word token should be with preceding blank space everywhere.</li>
</ol>