title: BertWordPieceTokenizer vs BertTokenizer from HuggingFace
body: <p>I have the following pieces of code and trying to understand the difference between BertWordPieceTokenizer and BertTokenizer.</p>
  <h1><strong>BertWordPieceTokenizer (Rust based)</strong></h1>
  <pre><code>from tokenizers import BertWordPieceTokenizer
  
  sequence = &quot;Hello, y'all! How are you Tokenizer  ?&quot;
  tokenizer = BertWordPieceTokenizer(&quot;bert-base-uncased-vocab.txt&quot;)
  tokenized_sequence = tokenizer.encode(sequence)
  print(tokenized_sequence)
  &gt;&gt;&gt;Encoding(num_tokens=15, attributes=[ids, type_ids, tokens, offsets, attention_mask, special_tokens_mask, overflowing])
  
  print(tokenized_sequence.tokens)
  &gt;&gt;&gt;['[CLS]', 'hello', ',', 'y', &quot;'&quot;, 'all', '!', 'how', 'are', 'you', 'token', '##izer', '[UNK]', '?', '[SEP]']
  </code></pre>
  <h1>BertTokenizer</h1>
  <pre><code>from transformers import BertTokenizer
  tokenizer = BertTokenizer(&quot;bert-base-cased-vocab.txt&quot;)
  tokenized_sequence = tokenizer.encode(sequence)
  print(tokenized_sequence)
  #Output: [19082, 117, 194, 112, 1155, 106, 1293, 1132, 1128, 22559, 17260, 100, 136]
  </code></pre>
  <ol>
  <li>Why is encode working differently in both ? In BertWordPieceTokenizer it gives Encoding object while in BertTokenizer it gives the ids of the vocab.</li>
  <li>What is the Difference between BertWordPieceTokenizer and BertTokenizer fundamentally, because as I understand BertTokenizer also uses WordPiece under the hood.</li>
  </ol>
  <p>Thanks</p>