title: Huggingface saving tokenizer
body: <p>I am trying to save the tokenizer in huggingface so that I can load it later from a container where I don't need access to the internet.</p>
  <pre class="lang-py prettyprint-override"><code>BASE_MODEL = &quot;distilbert-base-multilingual-cased&quot;
  tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL)
  tokenizer.save_vocabulary(&quot;./models/tokenizer/&quot;)
  tokenizer2 = AutoTokenizer.from_pretrained(&quot;./models/tokenizer/&quot;)
  </code></pre>
  <p>However, the last line is giving the error:</p>
  <pre class="lang-py prettyprint-override"><code>OSError: Can't load config for './models/tokenizer3/'. Make sure that:
  
  - './models/tokenizer3/' is a correct model identifier listed on 'https://huggingface.co/models'
  
  - or './models/tokenizer3/' is the correct path to a directory containing a config.json file
  </code></pre>
  <p><strong>transformers version: 3.1.0</strong></p>
  <p><a href="https://stackoverflow.com/questions/58417374/how-to-load-the-saved-tokenizer-from-pretrained-model-in-pytorch">How to load the saved tokenizer from pretrained model in Pytorch</a> didn't help unfortunately.</p>
  <h2>Edit 1</h2>
  <p>Thanks to @ashwin's answer below I tried <code>save_pretrained</code> instead, and I get the following error:</p>
  <pre class="lang-py prettyprint-override"><code>OSError: Can't load config for './models/tokenizer/'. Make sure that:
  
  - './models/tokenizer/' is a correct model identifier listed on 'https://huggingface.co/models'
  
  - or './models/tokenizer/' is the correct path to a directory containing a config.json file
  </code></pre>
  <p>the contents of the tokenizer folder is below:
  <a href="https://i.stack.imgur.com/hNYVy.png" rel="noreferrer"><img src="https://i.stack.imgur.com/hNYVy.png" alt="enter image description here" /></a></p>
  <p>I tried renaming <code>tokenizer_config.json</code> to <code>config.json</code> and then I got the error:</p>
  <pre class="lang-py prettyprint-override"><code>ValueError: Unrecognized model in ./models/tokenizer/. Should have a `model_type` key in its config.json, or contain one of the following strings in its name: retribert, t5, mobilebert, distilbert, albert, camembert, xlm-roberta, pegasus, marian, mbart, bart, reformer, longformer, roberta, flaubert, bert, openai-gpt, gpt2, transfo-xl, xlnet, xlm, ctrl, electra, encoder-decoder
  </code></pre>