<h3 dir="auto">System Info</h3>
<ul dir="auto">
<li><code class="notranslate">transformers</code> version: 4.25.1</li>
<li>Platform: Linux-5.15.0-58-generic-x86_64-with-glibc2.35</li>
<li>Python version: 3.9.13</li>
<li>Huggingface_hub version: 0.10.0</li>
<li>PyTorch version (GPU?): 1.12.1.post201 (False)</li>
<li>Tensorflow version (GPU?): not installed (NA)</li>
<li>Flax version (CPU?/GPU?/TPU?): 0.6.3 (cpu)</li>
<li>Jax version: 0.4.1</li>
<li>JaxLib version: 0.4.1</li>
<li>Using GPU in script?: No</li>
<li>Using distributed or parallel set-up in script?: No</li>
</ul>
<h3 dir="auto">Who can help?</h3>
<p dir="auto"><a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/rooa/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/rooa">@rooa</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/patrickvonplaten/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/patrickvonplaten">@patrickvonplaten</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/patil-suraj/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/patil-suraj">@patil-suraj</a></p>
<h3 dir="auto">Information</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> The official example scripts</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> My own modified scripts</li>
</ul>
<h3 dir="auto">Tasks</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> An officially supported task in the <code class="notranslate">examples</code> folder (such as GLUE/SQuAD, ...)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> My own task or dataset (give details below)</li>
</ul>
<h3 dir="auto">Reproduction</h3>
<p dir="auto">The CodeGen tokenizer seems to remove the newline symbol in certain scenarios.<br>
In particular, <code class="notranslate">decode(encode(text))</code> does not always equal the original <code class="notranslate">text</code>.<br>
The following is the smallest example that reproduces the error but other text examples will have this issue as well.</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="from transformers import CodeGenTokenizer

# other checkpoints in the CodeGen series have the same issue
tokenizer = CodeGenTokenizer.from_pretrained(&quot;Salesforce/codegen-350M-multi&quot;)

# new line (10), space (32), space (32)
text = &quot;\n  &quot;
print([ord(c) for c in text])
# output: [10, 32, 32]

encoded = tokenizer.encode(text)
print(encoded)
# output: [50286]

decoded = tokenizer.decode(encoded)
print([ord(c) for c in decoded])
# actual: [32, 32]
# expected: [10, 32, 32]"><pre class="notranslate"><span class="pl-k">from</span> <span class="pl-s1">transformers</span> <span class="pl-k">import</span> <span class="pl-v">CodeGenTokenizer</span>

<span class="pl-c"># other checkpoints in the CodeGen series have the same issue</span>
<span class="pl-s1">tokenizer</span> <span class="pl-c1">=</span> <span class="pl-v">CodeGenTokenizer</span>.<span class="pl-en">from_pretrained</span>(<span class="pl-s">"Salesforce/codegen-350M-multi"</span>)

<span class="pl-c"># new line (10), space (32), space (32)</span>
<span class="pl-s1">text</span> <span class="pl-c1">=</span> <span class="pl-s">"<span class="pl-cce">\n</span>  "</span>
<span class="pl-en">print</span>([<span class="pl-en">ord</span>(<span class="pl-s1">c</span>) <span class="pl-k">for</span> <span class="pl-s1">c</span> <span class="pl-c1">in</span> <span class="pl-s1">text</span>])
<span class="pl-c"># output: [10, 32, 32]</span>

<span class="pl-s1">encoded</span> <span class="pl-c1">=</span> <span class="pl-s1">tokenizer</span>.<span class="pl-en">encode</span>(<span class="pl-s1">text</span>)
<span class="pl-en">print</span>(<span class="pl-s1">encoded</span>)
<span class="pl-c"># output: [50286]</span>

<span class="pl-s1">decoded</span> <span class="pl-c1">=</span> <span class="pl-s1">tokenizer</span>.<span class="pl-en">decode</span>(<span class="pl-s1">encoded</span>)
<span class="pl-en">print</span>([<span class="pl-en">ord</span>(<span class="pl-s1">c</span>) <span class="pl-k">for</span> <span class="pl-s1">c</span> <span class="pl-c1">in</span> <span class="pl-s1">decoded</span>])
<span class="pl-c"># actual: [32, 32]</span>
<span class="pl-c"># expected: [10, 32, 32]</span></pre></div>
<h3 dir="auto">Expected behavior</h3>
<p dir="auto">Expected: the decoded string is equal to the original string.<br>
Actual: the decoded string is missing the leading new line symbol.</p>
<h3 dir="auto">System Info</h3>
<ul dir="auto">
<li><code class="notranslate">transformers</code> version: 4.24.0</li>
<li>Platform: Linux-5.4.0-135-generic-x86_64-with-glibc2.31</li>
<li>Python version: 3.10.8</li>
<li>Huggingface_hub version: 0.11.1</li>
<li>PyTorch version (GPU?): 1.13.1 (True)</li>
<li>Tensorflow version (GPU?): not installed (NA)</li>
<li>Flax version (CPU?/GPU?/TPU?): not installed (NA)</li>
<li>Jax version: not installed</li>
<li>JaxLib version: not installed</li>
<li>Using GPU in script?: no</li>
<li>Using distributed or parallel set-up in script?: no</li>
</ul>
<h3 dir="auto">Who can help?</h3>
<p dir="auto"><a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/ArthurZucker/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/ArthurZucker">@ArthurZucker</a></p>
<h3 dir="auto">Information</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> The official example scripts</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> My own modified scripts</li>
</ul>
<h3 dir="auto">Tasks</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> An officially supported task in the <code class="notranslate">examples</code> folder (such as GLUE/SQuAD, ...)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> My own task or dataset (give details below)</li>
</ul>
<h3 dir="auto">Reproduction</h3>
<p dir="auto">Steps to reproduce the behavior:</p>
<ol dir="auto">
<li>load a <code class="notranslate">PreTrainedTokenizer</code> that contains <code class="notranslate">unique_no_split_tokens</code>, e.g. <code class="notranslate">EleutherAI/gpt-j-6B</code>.</li>
</ol>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="tokenizer = transformers.GPT2Tokenizer.from_pretrained('EleutherAI/gpt-j-6B')"><pre class="notranslate"><span class="pl-s1">tokenizer</span> <span class="pl-c1">=</span> <span class="pl-s1">transformers</span>.<span class="pl-v">GPT2Tokenizer</span>.<span class="pl-en">from_pretrained</span>(<span class="pl-s">'EleutherAI/gpt-j-6B'</span>)</pre></div>
<ol start="2" dir="auto">
<li>use the tokenizer to split a string that contains a <code class="notranslate">unique_no_split_tokens</code>, e.g. <code class="notranslate">" &lt;|extratoken_1|&gt; "</code>.</li>
</ol>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="print(tokenizer(&quot; &lt;|extratoken_1|&gt; &quot;).input_ids)"><pre class="notranslate"><span class="pl-en">print</span>(<span class="pl-en">tokenizer</span>(<span class="pl-s">" &lt;|extratoken_1|&gt; "</span>).<span class="pl-s1">input_ids</span>)</pre></div>
<h3 dir="auto">Expected behavior</h3>
<p dir="auto">The tokenizer splits the string into 3 tokens (<code class="notranslate">" "</code>, <code class="notranslate">"&lt;|extratoken_1|&gt;"</code> and <code class="notranslate">" "</code>), and gives their ids (<code class="notranslate">[220, 50257, 220]</code>). This is the behavior of <code class="notranslate">PreTrainedTokenizerFast</code>.</p>
<p dir="auto">But the actual behavior is that the <code class="notranslate">PreTrainedTokenizer</code> only gives the id of <code class="notranslate">"&lt;|extratoken_1|&gt;"</code>, i.e. <code class="notranslate">50257</code></p>