<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="  File &quot;C:\[path]\api\utilities\tokenizers.py&quot;, line 4, in &lt;module&gt;
    gpt2 = GPT2TokenizerFast.from_pretrained(&quot;gpt2&quot;)
  File &quot;C:\[path]\venv\lib\site-packages\transformers\tokenization_utils_base.py&quot;, line 1750, in from_pretrained
    commit_hash = extract_commit_hash(resolved_vocab_files[file_id], commit_hash)
  File &quot;C:\[path]\venv\lib\site-packages\transformers\utils\hub.py&quot;, line 225, in extract_commit_hash
    search = re.search(r&quot;snapshots/([^/]+)/&quot;, resolved_file)
  File &quot;C:\Program Files\Python310\lib\re.py&quot;, line 200, in search
    return _compile(pattern, flags).search(string)
TypeError: expected string or bytes-like object"><pre class="notranslate"><code class="notranslate">  File "C:\[path]\api\utilities\tokenizers.py", line 4, in &lt;module&gt;
    gpt2 = GPT2TokenizerFast.from_pretrained("gpt2")
  File "C:\[path]\venv\lib\site-packages\transformers\tokenization_utils_base.py", line 1750, in from_pretrained
    commit_hash = extract_commit_hash(resolved_vocab_files[file_id], commit_hash)
  File "C:\[path]\venv\lib\site-packages\transformers\utils\hub.py", line 225, in extract_commit_hash
    search = re.search(r"snapshots/([^/]+)/", resolved_file)
  File "C:\Program Files\Python310\lib\re.py", line 200, in search
    return _compile(pattern, flags).search(string)
TypeError: expected string or bytes-like object
</code></pre></div>
<p dir="auto">Just started getting this error today, any idea for a workaround or a good version to rollback to?</p>
<p dir="auto">Just some context, we use <code class="notranslate">TRANSFORMERS_OFFLINE=1</code> in the NeMo CI to ensure we load from the local cache. With the latest transformers version we noticed this bug in our CI!</p>
<h3 dir="auto">System Info</h3>
<ul dir="auto">
<li><code class="notranslate">transformers</code> version: 4.22.1</li>
<li>Platform: Linux-5.15.0-41-generic-x86_64-with-glibc2.31</li>
<li>Python version: 3.9.12</li>
<li>Huggingface_hub version: 0.9.1</li>
<li>PyTorch version (GPU?): 1.12.1+cu113 (True)</li>
<li>Tensorflow version (GPU?): not installed (NA)</li>
<li>Flax version (CPU?/GPU?/TPU?): not installed (NA)</li>
<li>Jax version: not installed</li>
<li>JaxLib version: not installed</li>
<li>Using GPU in script?: no</li>
<li>Using distributed or parallel set-up in script?: no</li>
</ul>
<h3 dir="auto">Who can help?</h3>
<p dir="auto"><a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/SaulLu/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/SaulLu">@SaulLu</a></p>
<h3 dir="auto">Information</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> The official example scripts</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> My own modified scripts</li>
</ul>
<h3 dir="auto">Tasks</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> An officially supported task in the <code class="notranslate">examples</code> folder (such as GLUE/SQuAD, ...)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> My own task or dataset (give details below)</li>
</ul>
<h3 dir="auto">Reproduction</h3>
<p dir="auto">Create this script <code class="notranslate">reprod.py</code>:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="from transformers import AutoTokenizer

AutoTokenizer.from_pretrained(pretrained_model_name_or_path='gpt2')"><pre class="notranslate"><span class="pl-k">from</span> <span class="pl-s1">transformers</span> <span class="pl-k">import</span> <span class="pl-v">AutoTokenizer</span>

<span class="pl-v">AutoTokenizer</span>.<span class="pl-en">from_pretrained</span>(<span class="pl-s1">pretrained_model_name_or_path</span><span class="pl-c1">=</span><span class="pl-s">'gpt2'</span>)</pre></div>
<p dir="auto">run:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="python reprod.py
TRANSFORMERS_OFFLINE=1 python reprod.py"><pre class="notranslate"><code class="notranslate">python reprod.py
TRANSFORMERS_OFFLINE=1 python reprod.py
</code></pre></div>
<p dir="auto">First one runs successfully, second one fails:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Traceback (most recent call last):
  File &quot;/home/snarenthiran/NeMo/reprod.py&quot;, line 3, in &lt;module&gt;
    AutoTokenizer.from_pretrained(pretrained_model_name_or_path='gpt2')
  File &quot;/home/snarenthiran/anaconda3/lib/python3.9/site-packages/transformers/models/auto/tokenization_auto.py&quot;, line 549, in from_pretrained
    tokenizer_config = get_tokenizer_config(pretrained_model_name_or_path, **kwargs)
  File &quot;/home/snarenthiran/anaconda3/lib/python3.9/site-packages/transformers/models/auto/tokenization_auto.py&quot;, line 418, in get_tokenizer_config
    commit_hash = extract_commit_hash(resolved_config_file, commit_hash)
  File &quot;/home/snarenthiran/anaconda3/lib/python3.9/site-packages/transformers/utils/hub.py&quot;, line 225, in extract_commit_hash
    search = re.search(r&quot;snapshots/([^/]+)/&quot;, resolved_file)
  File &quot;/home/snarenthiran/anaconda3/lib/python3.9/re.py&quot;, line 201, in search
    return _compile(pattern, flags).search(string)
TypeError: expected string or bytes-like object"><pre class="notranslate"><code class="notranslate">Traceback (most recent call last):
  File "/home/snarenthiran/NeMo/reprod.py", line 3, in &lt;module&gt;
    AutoTokenizer.from_pretrained(pretrained_model_name_or_path='gpt2')
  File "/home/snarenthiran/anaconda3/lib/python3.9/site-packages/transformers/models/auto/tokenization_auto.py", line 549, in from_pretrained
    tokenizer_config = get_tokenizer_config(pretrained_model_name_or_path, **kwargs)
  File "/home/snarenthiran/anaconda3/lib/python3.9/site-packages/transformers/models/auto/tokenization_auto.py", line 418, in get_tokenizer_config
    commit_hash = extract_commit_hash(resolved_config_file, commit_hash)
  File "/home/snarenthiran/anaconda3/lib/python3.9/site-packages/transformers/utils/hub.py", line 225, in extract_commit_hash
    search = re.search(r"snapshots/([^/]+)/", resolved_file)
  File "/home/snarenthiran/anaconda3/lib/python3.9/re.py", line 201, in search
    return _compile(pattern, flags).search(string)
TypeError: expected string or bytes-like object
</code></pre></div>
<h3 dir="auto">Expected behavior</h3>
<p dir="auto">To create the tokenizer from the local files.</p>