class_docstr: Construct a BPE tokenizer for HerBERT.
  
  Peculiarities:
  
  - uses BERT's pre-tokenizer: BaseTokenizer splits tokens on spaces, and also on punctuation. Each occurrence of a
    punctuation character will be treated separately.
  
  - Such pretokenized input is BPE subtokenized
  
  This tokenizer inherits from [`XLMTokenizer`] which contains most of the methods. Users should refer to the
  superclass for more information regarding methods.
code: def bpe(self, token):
      word = tuple(token[:-1]) + (token[-1] + "</w>",)
      if token in self.cache:
          return self.cache[token]
      pairs = get_pairs(word)
  
      if not pairs:
          return token + "</w>"
  
      while True:
          bigram = min(
              pairs,
              key=lambda pair: self.bpe_ranks.get(pair, float("inf")),
          )
          if bigram not in self.bpe_ranks:
              break
          first, second = bigram
          new_word = []
          i = 0
          while i < len(word):
              try:
                  j = word.index(first, i)
              except ValueError:
                  new_word.extend(word[i:])
                  break
              else:
                  new_word.extend(word[i:j])
                  i = j
  
              if (
                  word[i] == first
                  and i < len(word) - 1
                  and word[i + 1] == second
              ):
                  new_word.append(first + second)
                  i += 2
              else:
                  new_word.append(word[i])
                  i += 1
          new_word = tuple(new_word)
          word = new_word
          if len(word) == 1:
              break
          else:
              pairs = get_pairs(word)
      word = " ".join(word)
      if word == "\n  </w>":
          word = "\n</w>"
      self.cache[token] = word
      return word
  
imports: import json
  import os
  import re
  import unicodedata
  from typing import List, Optional, Tuple
  from ...tokenization_utils import PreTrainedTokenizer, _is_control, _is_punctuation, _is_whitespace
  from ...utils import logging
label_desc: Use image node_id 9 for calling the HerbertTokenizer obj's underlying member method code with example usage: obj.bpe(token) and returns: word, self, unknown