<h1 dir="auto"><g-emoji class="g-emoji" alias="bug" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/1f41b.png">🐛</g-emoji> Bug MarianTokenizer.from_pretrained() fails in Python 3.6.4 in Windows 10</h1>
<h2 dir="auto">Information</h2>
<p dir="auto">Occurs with using the example here: <a href="huggingface.co/transformers/model_doc/marian.html?highlight=marianmtmodel#transformers.MarianMTModel">https://huggingface.co/transformers/model_doc/marian.html?highlight=marianmtmodel#transformers.MarianMTModel</a></p>
<p dir="auto">Model I am using (Bert, XLNet ...): MarianMTModel</p>
<p dir="auto">Language I am using the model on (English, Chinese ...): English</p>
<p dir="auto">The problem arises when using:</p>
<ul class="contains-task-list">
<li>[X ] the official example scripts: (give details below)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> my own modified scripts: (give details below)</li>
</ul>
<p dir="auto">The tasks I am working on is:</p>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> an official GLUE/SQUaD task: (give the name)</li>
<li>[X ] my own task or dataset: (give details below)</li>
</ul>
<h2 dir="auto">To reproduce</h2>
<p dir="auto">Paste code from example and run:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="from transformers import MarianTokenizer, MarianMTModel
from typing import List
src = 'fr'  # source language
trg = 'en'  # target language
sample_text = &quot;où est l'arrêt de bus ?&quot;
mname = f'Helsinki-NLP/opus-mt-{src}-{trg}'

model = MarianMTModel.from_pretrained(mname)
tok = MarianTokenizer.from_pretrained(mname)
batch = tok.prepare_translation_batch(src_texts=[sample_text])  # don't need tgt_text for inference
gen = model.generate(**batch)  # for forward pass: model(**batch)
words: List[str] = tok.batch_decode(gen, skip_special_tokens=True)  # returns &quot;Where is the the bus stop ?&quot;
print(words)"><pre class="notranslate"><span class="pl-k">from</span> <span class="pl-s1">transformers</span> <span class="pl-k">import</span> <span class="pl-v">MarianTokenizer</span>, <span class="pl-v">MarianMTModel</span>
<span class="pl-k">from</span> <span class="pl-s1">typing</span> <span class="pl-k">import</span> <span class="pl-v">List</span>
<span class="pl-s1">src</span> <span class="pl-c1">=</span> <span class="pl-s">'fr'</span>  <span class="pl-c"># source language</span>
<span class="pl-s1">trg</span> <span class="pl-c1">=</span> <span class="pl-s">'en'</span>  <span class="pl-c"># target language</span>
<span class="pl-s1">sample_text</span> <span class="pl-c1">=</span> <span class="pl-s">"où est l'arrêt de bus ?"</span>
<span class="pl-s1">mname</span> <span class="pl-c1">=</span> <span class="pl-s">f'Helsinki-NLP/opus-mt-<span class="pl-s1"><span class="pl-kos">{</span><span class="pl-s1">src</span><span class="pl-kos">}</span></span>-<span class="pl-s1"><span class="pl-kos">{</span><span class="pl-s1">trg</span><span class="pl-kos">}</span></span>'</span>

<span class="pl-s1">model</span> <span class="pl-c1">=</span> <span class="pl-v">MarianMTModel</span>.<span class="pl-en">from_pretrained</span>(<span class="pl-s1">mname</span>)
<span class="pl-s1">tok</span> <span class="pl-c1">=</span> <span class="pl-v">MarianTokenizer</span>.<span class="pl-en">from_pretrained</span>(<span class="pl-s1">mname</span>)
<span class="pl-s1">batch</span> <span class="pl-c1">=</span> <span class="pl-s1">tok</span>.<span class="pl-en">prepare_translation_batch</span>(<span class="pl-s1">src_texts</span><span class="pl-c1">=</span>[<span class="pl-s1">sample_text</span>])  <span class="pl-c"># don't need tgt_text for inference</span>
<span class="pl-s1">gen</span> <span class="pl-c1">=</span> <span class="pl-s1">model</span>.<span class="pl-en">generate</span>(<span class="pl-c1">**</span><span class="pl-s1">batch</span>)  <span class="pl-c"># for forward pass: model(**batch)</span>
<span class="pl-s1">words</span>: <span class="pl-v">List</span>[<span class="pl-s1">str</span>] <span class="pl-c1">=</span> <span class="pl-s1">tok</span>.<span class="pl-en">batch_decode</span>(<span class="pl-s1">gen</span>, <span class="pl-s1">skip_special_tokens</span><span class="pl-c1">=</span><span class="pl-c1">True</span>)  <span class="pl-c"># returns "Where is the the bus stop ?"</span>
<span class="pl-en">print</span>(<span class="pl-s1">words</span>)</pre></div>
<p dir="auto">Steps to reproduce the behavior:</p>
<ol dir="auto">
<li>Run the example</li>
<li>Program terminates on <code class="notranslate">tok = MarianTokenizer.from_pretrained(mname)</code></li>
</ol>

<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="stdbuf was not found; communication with perl may hang due to stdio buffering.
Traceback (most recent call last):
  File &quot;C:\Program Files\Python\lib\site-packages\transformers\tokenization_utils.py&quot;, line 1055, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)
  File &quot;C:\Program Files\Python\lib\site-packages\transformers\tokenization_marian.py&quot;, line 89, in __init__
    self._setup_normalizer()
  File &quot;C:\Program Files\Python\lib\site-packages\transformers\tokenization_marian.py&quot;, line 95, in _setup_normalizer
    self.punc_normalizer = MosesPunctuationNormalizer(self.source_lang)
  File &quot;C:\Program Files\Python\lib\site-packages\mosestokenizer\punctnormalizer.py&quot;, line 47, in __init__
    super().__init__(argv)
  File &quot;C:\Program Files\Python\lib\site-packages\toolwrapper.py&quot;, line 64, in __init__
    self.start()
  File &quot;C:\Program Files\Python\lib\site-packages\toolwrapper.py&quot;, line 108, in start
    env=env,
  File &quot;C:\Program Files\Python\lib\subprocess.py&quot;, line 709, in __init__
    restore_signals, start_new_session)
  File &quot;C:\Program Files\Python\lib\subprocess.py&quot;, line 997, in _execute_child
    startupinfo)
FileNotFoundError: [WinError 2] The system cannot find the file specified

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File &quot;C:/Development/Research/COVID-19-Misinfo2/src/translate_test_2.py&quot;, line 9, in &lt;module&gt;
    tok = MarianTokenizer.from_pretrained(mname)
  File &quot;C:\Program Files\Python\lib\site-packages\transformers\tokenization_utils.py&quot;, line 902, in from_pretrained
    return cls._from_pretrained(*inputs, **kwargs)
  File &quot;C:\Program Files\Python\lib\site-packages\transformers\tokenization_utils.py&quot;, line 1058, in _from_pretrained
    &quot;Unable to load vocabulary from file. &quot;
OSError: Unable to load vocabulary from file. Please check that the provided vocabulary is accessible and not corrupted."><pre class="notranslate"><span class="pl-s1">stdbuf</span> <span class="pl-s1">was</span> <span class="pl-c1">not</span> <span class="pl-s1">found</span>; <span class="pl-s1">communication</span> <span class="pl-s1">with</span> <span class="pl-s1">perl</span> <span class="pl-s1">may</span> <span class="pl-s1">hang</span> <span class="pl-s1">due</span> <span class="pl-s1">to</span> <span class="pl-s1">stdio</span> <span class="pl-s1">buffering</span>.
<span class="pl-v">Traceback</span> (<span class="pl-s1">most</span> <span class="pl-s1">recent</span> <span class="pl-s1">call</span> <span class="pl-s1">last</span>):
  <span class="pl-v">File</span> <span class="pl-s">"C:\Program Files\Python\lib\site-packages<span class="pl-cce">\t</span>ransformers<span class="pl-cce">\t</span>okenization_utils.py"</span>, <span class="pl-s1">line</span> <span class="pl-c1">1055</span>, <span class="pl-s1">in</span> <span class="pl-s1">_from_pretrained</span>
    <span class="pl-s1">tokenizer</span> <span class="pl-c1">=</span> <span class="pl-en">cls</span>(<span class="pl-c1">*</span><span class="pl-s1">init_inputs</span>, <span class="pl-c1">**</span><span class="pl-s1">init_kwargs</span>)
  <span class="pl-v">File</span> <span class="pl-s">"C:\Program Files\Python\lib\site-packages<span class="pl-cce">\t</span>ransformers<span class="pl-cce">\t</span>okenization_marian.py"</span>, <span class="pl-s1">line</span> <span class="pl-c1">89</span>, <span class="pl-s1">in</span> <span class="pl-s1">__init__</span>
    <span class="pl-s1">self</span>.<span class="pl-en">_setup_normalizer</span>()
  <span class="pl-v">File</span> <span class="pl-s">"C:\Program Files\Python\lib\site-packages<span class="pl-cce">\t</span>ransformers<span class="pl-cce">\t</span>okenization_marian.py"</span>, <span class="pl-s1">line</span> <span class="pl-c1">95</span>, <span class="pl-s1">in</span> <span class="pl-s1">_setup_normalizer</span>
    <span class="pl-s1">self</span>.<span class="pl-s1">punc_normalizer</span> <span class="pl-c1">=</span> <span class="pl-v">MosesPunctuationNormalizer</span>(<span class="pl-s1">self</span>.<span class="pl-s1">source_lang</span>)
  <span class="pl-v">File</span> <span class="pl-s">"C:\Program Files\Python\lib\site-packages\mosestokenizer\punctnormalizer.py"</span>, <span class="pl-s1">line</span> <span class="pl-c1">47</span>, <span class="pl-s1">in</span> <span class="pl-s1">__init__</span>
    <span class="pl-en">super</span>().<span class="pl-en">__init__</span>(<span class="pl-s1">argv</span>)
  <span class="pl-v">File</span> <span class="pl-s">"C:\Program Files\Python\lib\site-packages<span class="pl-cce">\t</span>oolwrapper.py"</span>, <span class="pl-s1">line</span> <span class="pl-c1">64</span>, <span class="pl-s1">in</span> <span class="pl-s1">__init__</span>
    <span class="pl-s1">self</span>.<span class="pl-en">start</span>()
  <span class="pl-v">File</span> <span class="pl-s">"C:\Program Files\Python\lib\site-packages<span class="pl-cce">\t</span>oolwrapper.py"</span>, <span class="pl-s1">line</span> <span class="pl-c1">108</span>, <span class="pl-s1">in</span> <span class="pl-s1">start</span>
    <span class="pl-s1">env</span><span class="pl-c1">=</span><span class="pl-s1">env</span>,
  <span class="pl-v">File</span> <span class="pl-s">"C:\Program Files\Python\lib\subprocess.py"</span>, <span class="pl-s1">line</span> <span class="pl-c1">709</span>, <span class="pl-s1">in</span> <span class="pl-s1">__init__</span>
    <span class="pl-s1">restore_signals</span>, <span class="pl-s1">start_new_session</span>)
  <span class="pl-v">File</span> <span class="pl-s">"C:\Program Files\Python\lib\subprocess.py"</span>, <span class="pl-s1">line</span> <span class="pl-c1">997</span>, <span class="pl-s1">in</span> <span class="pl-s1">_execute_child</span>
    <span class="pl-s1">startupinfo</span>)
<span class="pl-v">FileNotFoundError</span>: [<span class="pl-v">WinError</span> <span class="pl-c1">2</span>] <span class="pl-v">The</span> <span class="pl-s1">system</span> <span class="pl-s1">cannot</span> <span class="pl-s1">find</span> <span class="pl-s1">the</span> <span class="pl-s1">file</span> <span class="pl-s1">specified</span>

<span class="pl-v">During</span> <span class="pl-s1">handling</span> <span class="pl-s1">of</span> <span class="pl-s1">the</span> <span class="pl-s1">above</span> <span class="pl-s1">exception</span>, <span class="pl-s1">another</span> <span class="pl-s1">exception</span> <span class="pl-s1">occurred</span>:

<span class="pl-v">Traceback</span> (<span class="pl-s1">most</span> <span class="pl-s1">recent</span> <span class="pl-s1">call</span> <span class="pl-s1">last</span>):
  <span class="pl-v">File</span> <span class="pl-s">"C:/Development/Research/COVID-19-Misinfo2/src/translate_test_2.py"</span>, <span class="pl-s1">line</span> <span class="pl-c1">9</span>, <span class="pl-s1">in</span> <span class="pl-c1">&lt;</span><span class="pl-s1">module</span><span class="pl-c1">&gt;</span>
    <span class="pl-s1">tok</span> <span class="pl-c1">=</span> <span class="pl-v">MarianTokenizer</span>.<span class="pl-en">from_pretrained</span>(<span class="pl-s1">mname</span>)
  <span class="pl-v">File</span> <span class="pl-s">"C:\Program Files\Python\lib\site-packages<span class="pl-cce">\t</span>ransformers<span class="pl-cce">\t</span>okenization_utils.py"</span>, <span class="pl-s1">line</span> <span class="pl-c1">902</span>, <span class="pl-s1">in</span> <span class="pl-s1">from_pretrained</span>
    <span class="pl-k">return</span> <span class="pl-s1">cls</span>.<span class="pl-en">_from_pretrained</span>(<span class="pl-c1">*</span><span class="pl-s1">inputs</span>, <span class="pl-c1">**</span><span class="pl-s1">kwargs</span>)
  <span class="pl-v">File</span> <span class="pl-s">"C:\Program Files\Python\lib\site-packages<span class="pl-cce">\t</span>ransformers<span class="pl-cce">\t</span>okenization_utils.py"</span>, <span class="pl-s1">line</span> <span class="pl-c1">1058</span>, <span class="pl-s1">in</span> <span class="pl-s1">_from_pretrained</span>
    <span class="pl-s">"Unable to load vocabulary from file. "</span>
<span class="pl-v">OSError</span>: <span class="pl-v">Unable</span> <span class="pl-s1">to</span> <span class="pl-s1">load</span> <span class="pl-s1">vocabulary</span> <span class="pl-k">from</span> <span class="pl-s1">file</span>. <span class="pl-v">Please</span> <span class="pl-s1">check</span> <span class="pl-s1">that</span> <span class="pl-s1">the</span> <span class="pl-s1">provided</span> <span class="pl-s1">vocabulary</span> <span class="pl-c1">is</span> <span class="pl-s1">accessible</span> <span class="pl-c1">and</span> <span class="pl-c1">not</span> <span class="pl-s1">corrupted</span>.</pre></div>
<h2 dir="auto">Expected behavior</h2>
<p dir="auto">prints ["Where is the the bus stop ?"]</p>

<h2 dir="auto">Environment info</h2>

<ul dir="auto">
<li><code class="notranslate">transformers</code> version: 2.9.1</li>
<li>Platform: Windows-10-10.0.18362-SP0</li>
<li>Python version: 3.6.4</li>
<li>PyTorch version (GPU?): 1.5.0+cu101 (True)</li>
<li>Tensorflow version (GPU?): 2.1.0 (True)</li>
<li>Using GPU in script?: yes</li>
<li>Using distributed or parallel set-up in script?: no</li>
</ul>
<h1 dir="auto"><g-emoji class="g-emoji" alias="rocket" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/1f680.png">🚀</g-emoji> Feature request</h1>
<p dir="auto">The HF approach is to unroll most of the code for the ease of understanding, but somehow this is not the case with docs.</p>
<p dir="auto">If possible, could we explicitly add documentation for inherited methods in the specific model documentation page?</p>
<p dir="auto">e.g. why <a href="https://huggingface.co/transformers/model_doc/t5.html" rel="nofollow">https://huggingface.co/transformers/model_doc/t5.html</a> doesn't have <code class="notranslate">T5ForConditionalGeneration.generate</code> documented - sure one eventually figures out it's <a href="https://huggingface.co/transformers/main_classes/model.html#transformers.generation_utils.GenerationMixin.generate" rel="nofollow">https://huggingface.co/transformers/main_classes/model.html#transformers.generation_utils.GenerationMixin.generate</a> but why make user's life so difficult when the docs are autogenerated anyway.</p>
<p dir="auto">In the worst case there could be an entry for <code class="notranslate">T5ForConditionalGeneration.generate</code> with the link to <a href="https://huggingface.co/transformers/main_classes/model.html#transformers.generation_utils.GenerationMixin.generate" rel="nofollow">https://huggingface.co/transformers/main_classes/model.html#transformers.generation_utils.GenerationMixin.generate</a>, but that is not ideal since one has to leave the main doc of the model, which makes it much harder to jump around its different parts.</p>
<p dir="auto">IMHO, it'd greatly improve user's experience to have all the functionality documentation supported by a model in one page of that model.</p>
<p dir="auto">And xrefs are super-useful too, e.g. <a href="https://huggingface.co/transformers/model_doc/t5.html#overview" rel="nofollow">t5 pre-amble</a> mentions  <code class="notranslate">T5ForConditionalGeneration.generate </code>but it's not linked to anywhere.</p>
<p dir="auto">Thank you!</p>
<p dir="auto"><a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/sgugger/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/sgugger">@sgugger</a>, <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/LysandreJik/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/LysandreJik">@LysandreJik</a></p>