class_docstr: Base class for all slow tokenizers.
  
  Inherits from :class:`~transformers.tokenization_utils_base.PreTrainedTokenizerBase`.
  
  Handle all the shared methods for tokenization and special tokens as well as methods downloading/caching/loading
  pretrained tokenizers as well as adding tokens to the vocabulary.
  
  This class also contain the added tokens in a unified way on top of all tokenizers so we don't have to handle the
  specific vocabulary augmentation methods of the various underlying dictionary structures (BPE, sentencepiece...).
class_docstr_tok: ["Base","class","for","all","slow","tokenizers",".","Inherits","from",":","class",":","`","~transformers.tokenization_utils_base.PreTrainedTokenizerBase","`",".","Handle","all","the","shared","methods","for","tokenization","and","special","tokens","as","well","as","methods","downloading\/caching\/loading","pretrained","tokenizers","as","well","as","adding","tokens","to","the","vocabulary",".","This","class","also","contain","the","added","tokens","in","a","unified","way","on","top","of","all","tokenizers","so","we","do","n't","have","to","handle","the","specific","vocabulary","augmentation","methods","of","the","various","underlying","dictionary","structures","(","BPE",",","sentencepiece","...",")","."]
code: def __init__(self, **kwargs):
      super().__init__(**kwargs)
  
      # Added tokens - We store this for both slow and fast tokenizers
      # until the serialization of Fast tokenizers is updated
      self.added_tokens_encoder: Dict[str, int] = {}
      self.added_tokens_decoder: Dict[int, str] = {}
      self.unique_no_split_tokens: List[str] = []
      self.tokens_trie = Trie()
  
      self._decode_use_source_tokenizer = False
  
imports: import bisect
  import itertools
  import re
  import unicodedata
  from collections import OrderedDict
  from typing import Any, Dict, List, Optional, Tuple, Union, overload
  from .file_utils import PaddingStrategy, TensorType, add_end_docstrings
  from .tokenization_utils_base import ENCODE_KWARGS_DOCSTRING, ENCODE_PLUS_ADDITIONAL_KWARGS_DOCSTRING, INIT_TOKENIZER_DOCSTRING, AddedToken, BatchEncoding, EncodedInput, EncodedInputPair, PreTokenizedInput, PreTokenizedInputPair, PreTrainedTokenizerBase, TextInput, TextInputPair, TruncationStrategy
  from fastNLP.core.log import logger
label_desc: Use image node_id 1 to create a new PreTrainedTokenizer object from inherited base classes: PreTrainedTokenizerBase with example: obj = PreTrainedTokenizer()