<h2 dir="auto">Environment info</h2>

<ul dir="auto">
<li><code class="notranslate">transformers</code> version: 3.4.0</li>
<li>Platform: macOS-10.15.7-x86_64-i386-64bit</li>
<li>Python version: 3.8.5</li>
<li>PyTorch version (GPU?): 1.7.0 (False)</li>
<li>Tensorflow version (GPU?): not installed (NA)</li>
<li>Using GPU in script?: N/a</li>
<li>Using distributed or parallel set-up in script? N/a</li>
</ul>
<h3 dir="auto">Who can help</h3>
<p dir="auto">examples/seq2seq: <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/sshleifer/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/sshleifer">@sshleifer</a><br>
--&gt;</p>
<h2 dir="auto">Information</h2>
<p dir="auto">Model I am using (Bert, XLNet ...): T5Tokenizer</p>
<p dir="auto">The problem arises when using:</p>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> the official example scripts: (give details below)</li>
<li>[X ] my own modified scripts: (give details below)</li>
</ul>
<p dir="auto">The tasks I am working on is:</p>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> an official GLUE/SQUaD task: (give the name)</li>
<li>[ X] my own task or dataset: (give details below)</li>
</ul>
<h2 dir="auto">To reproduce</h2>
<p dir="auto">Steps to reproduce the behavior:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="from transformers import T5Tokenizer
input = &quot;word  &lt;pad&gt; &lt;unk&gt; &lt;/s&gt; &lt;/s&gt;&quot;
t5tokenizer = T5Tokenizer.from_pretrained('t5-small')
tokenized = t5tokenizer.batch_encode_plus([input], max_length=10, padding=&quot;longest&quot;, return_tensors=&quot;pt&quot;).input_ids
print(t5tokenizer.batch_decode(tokenized, skip_special_tokens=False, clean_up_tokenization_spaces=False))"><pre class="notranslate"><span class="pl-k">from</span> <span class="pl-s1">transformers</span> <span class="pl-k">import</span> <span class="pl-v">T5Tokenizer</span>
<span class="pl-s1">input</span> <span class="pl-c1">=</span> <span class="pl-s">"word  &lt;pad&gt; &lt;unk&gt; &lt;/s&gt; &lt;/s&gt;"</span>
<span class="pl-s1">t5tokenizer</span> <span class="pl-c1">=</span> <span class="pl-v">T5Tokenizer</span>.<span class="pl-en">from_pretrained</span>(<span class="pl-s">'t5-small'</span>)
<span class="pl-s1">tokenized</span> <span class="pl-c1">=</span> <span class="pl-s1">t5tokenizer</span>.<span class="pl-en">batch_encode_plus</span>([<span class="pl-s1">input</span>], <span class="pl-s1">max_length</span><span class="pl-c1">=</span><span class="pl-c1">10</span>, <span class="pl-s1">padding</span><span class="pl-c1">=</span><span class="pl-s">"longest"</span>, <span class="pl-s1">return_tensors</span><span class="pl-c1">=</span><span class="pl-s">"pt"</span>).<span class="pl-s1">input_ids</span>
<span class="pl-en">print</span>(<span class="pl-s1">t5tokenizer</span>.<span class="pl-en">batch_decode</span>(<span class="pl-s1">tokenized</span>, <span class="pl-s1">skip_special_tokens</span><span class="pl-c1">=</span><span class="pl-c1">False</span>, <span class="pl-s1">clean_up_tokenization_spaces</span><span class="pl-c1">=</span><span class="pl-c1">False</span>))</pre></div>
<p dir="auto">IDs output: <code class="notranslate"> _word  &lt;pad&gt; &lt;unk&gt; &lt;/s&gt; &lt;/s&gt;</code><br>
decode output: <code class="notranslate">word  ⁇ </code></p>
<h2 dir="auto">Expected behavior</h2>
<p dir="auto">The tokens should be shown in the decoded output, but everything except for the unknown token is dropped (no pad or EOS).<br>
<code class="notranslate">convert_ids_to_tokens</code> followed by <code class="notranslate">convert_tokens_to_string</code> also drops the tokens.</p>
<p dir="auto">The docs recommend adding the special eos_token <code class="notranslate">&lt;\s&gt;</code> to the end of each string when encoding/decoding with <code class="notranslate">T5Tokenizer</code>. However, this (and the other special tokens e.g. <code class="notranslate">unk_token</code>, <code class="notranslate">pad_token</code> aren't assigned unique ids in the lookup vocabulary (they are mapped to <code class="notranslate">{0,1,2}</code>, which are indices for other common words in the vocab). In practice, I find my model fails to properly produce the <code class="notranslate">eos_token</code> since it is associated with blank spaces, so the model produces run-ons during generation</p>
<h2 dir="auto">To reproduce</h2>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="&gt;&gt;&gt; from transformers import T5Tokenizer
&gt;&gt;&gt; tokenizer = T5Tokenizer.from_pretrained('t5-base')
&gt;&gt;&gt; tokenizer.pad_token
'&lt;pad&gt;'
&gt;&gt;&gt; tokenizer.pad_token_id
0
&gt;&gt;&gt; tokenizer.eos_token
'&lt;/s&gt;'
&gt;&gt;&gt; tokenizer.eos_token_id
1
&gt;&gt;&gt; tokenizer.unk_token
'&lt;unk&gt;'
&gt;&gt;&gt; tokenizer.unk_token_id
2"><pre class="notranslate"><code class="notranslate">&gt;&gt;&gt; from transformers import T5Tokenizer
&gt;&gt;&gt; tokenizer = T5Tokenizer.from_pretrained('t5-base')
&gt;&gt;&gt; tokenizer.pad_token
'&lt;pad&gt;'
&gt;&gt;&gt; tokenizer.pad_token_id
0
&gt;&gt;&gt; tokenizer.eos_token
'&lt;/s&gt;'
&gt;&gt;&gt; tokenizer.eos_token_id
1
&gt;&gt;&gt; tokenizer.unk_token
'&lt;unk&gt;'
&gt;&gt;&gt; tokenizer.unk_token_id
2
</code></pre></div>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="&gt;&gt;&gt; tokenizer.decode([0])
''
&gt;&gt;&gt; tokenizer.decode([1])
''
&gt;&gt;&gt; tokenizer.decode([2])
' ⁇ '"><pre class="notranslate"><code class="notranslate">&gt;&gt;&gt; tokenizer.decode([0])
''
&gt;&gt;&gt; tokenizer.decode([1])
''
&gt;&gt;&gt; tokenizer.decode([2])
' ⁇ '
</code></pre></div>
<h2 dir="auto">Expected behavior</h2>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="&gt;&gt;&gt; tokenizer.decode([0])
'&lt;pad&gt;'
&gt;&gt;&gt; tokenizer.decode([1])
'&lt;/s&gt;'
&gt;&gt;&gt; tokenizer.decode([2])
'&lt;unk&gt;'"><pre class="notranslate"><code class="notranslate">&gt;&gt;&gt; tokenizer.decode([0])
'&lt;pad&gt;'
&gt;&gt;&gt; tokenizer.decode([1])
'&lt;/s&gt;'
&gt;&gt;&gt; tokenizer.decode([2])
'&lt;unk&gt;'
</code></pre></div>
<h2 dir="auto">Environment info</h2>
<ul dir="auto">
<li><code class="notranslate">transformers</code> version: 2.9.1</li>
</ul>