class_docstr: Base class for all slow tokenizers.
  
  Inherits from [`~tokenization_utils_base.PreTrainedTokenizerBase`].
  
  Handle all the shared methods for tokenization and special tokens as well as methods downloading/caching/loading
  pretrained tokenizers as well as adding tokens to the vocabulary.
  
  This class also contain the added tokens in a unified way on top of all tokenizers so we don't have to handle the
  specific vocabulary augmentation methods of the various underlying dictionary structures (BPE, sentencepiece...).
class_docstr_tok: ["Base","class","for","all","slow","tokenizers",".","Inherits","from","[","`","~tokenization_utils_base.PreTrainedTokenizerBase","`","]",".","Handle","all","the","shared","methods","for","tokenization","and","special","tokens","as","well","as","methods","downloading\/caching\/loading","pretrained","tokenizers","as","well","as","adding","tokens","to","the","vocabulary",".","This","class","also","contain","the","added","tokens","in","a","unified","way","on","top","of","all","tokenizers","so","we","do","n't","have","to","handle","the","specific","vocabulary","augmentation","methods","of","the","various","underlying","dictionary","structures","(","BPE",",","sentencepiece","...",")","."]
docstr: Add a list of new tokens to the tokenizer class. If the new tokens are not in the vocabulary, they are added to
  it with indices starting from length of the current vocabulary. Special tokens are sometimes already in the
  vocab which is why they have to be handled specifically.
  
  Args:
      new_tokens (`List[str]`or `List[tokenizers.AddedToken]`):
          Token(s) to add in vocabulary. A token is counted as added if it's not already in the vocabulary
          (tested by checking if the tokenizer assign the index of the `unk_token` to them). If a token is part
          of the vocabulary then we simply mark this token as an `AddedToken` which allows to control the
          stripping and normalization of this token. This is NOT possible in `tokenizers`.
      special_tokens (`bool`, *optional*, defaults to `False`):
          Whether or not the tokens should be added as special tokens.
  
  Returns:
      `int`: The number of tokens actually added to the vocabulary.
  
  Examples:
  
  ```python
  # Let's see how to increase the vocabulary of Bert model and tokenizer
  tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
  model = BertModel.from_pretrained("bert-base-uncased")
  
  num_added_toks = tokenizer.add_tokens(["new_tok1", "my_new-tok2"])
  print("We have added", num_added_toks, "tokens")
  # Note: resize_token_embeddings expects to receive the full size of the new vocabulary, i.e. the length of the tokenizer.
  model.resize_token_embeddings(len(tokenizer))
  ```
docstr_tok: ["Add","a","list","of","new","tokens","to","the","tokenizer","class",".","If","the","new","tokens","are","not","in","the","vocabulary",",","they","are","added","to","it","with","indices","starting","from","length","of","the","current","vocabulary",".","Special","tokens","are","sometimes","already","in","the","vocab","which","is","why","they","have","to","be","handled","specifically",".","Args",":","new_tokens","(","`","List","[","str","]","`","or","`","List","[","tokenizers.AddedToken","]","`",")",":","Token","(","s",")","to","add","in","vocabulary",".","A","token","is","counted","as","added","if","it","'s","not","already","in","the","vocabulary","(","tested","by","checking","if","the","tokenizer","assign","the","index","of","the","`","unk_token","`","to","them",")",".","If","a","token","is","part","of","the","vocabulary","then","we","simply","mark","this","token","as","an","`","AddedToken","`","which","allows","to","control","the","stripping","and","normalization","of","this","token",".","This","is","NOT","possible","in","`","tokenizers","`",".","special_tokens","(","`","bool","`",",","*","optional","*",",","defaults","to","`","False","`",")",":","Whether","or","not","the","tokens","should","be","added","as","special","tokens",".","Returns",":","`","int","`",":","The","number","of","tokens","actually","added","to","the","vocabulary",".","Examples",":","``","`","python","#","Let","'s","see","how","to","increase","the","vocabulary","of","Bert","model","and","tokenizer","tokenizer","=","BertTokenizer.from_pretrained","(","``","bert-base-uncased","''",")","model","=","BertModel.from_pretrained","(","``","bert-base-uncased","''",")","num_added_toks","=","tokenizer.add_tokens","(","[","``","new_tok1","''",",","``","my_new-tok2","''","]",")","print","(","``","We","have","added","''",",","num_added_toks",",","``","tokens","''",")","#","Note",":","resize_token_embeddings","expects","to","receive","the","full","size","of","the","new","vocabulary",",","i.e",".","the","length","of","the","tokenizer",".","model.resize_token_embeddings","(","len","(","tokenizer",")",")","``","`"]
code: def _add_tokens(
      self,
      new_tokens: Union[List[str], List[AddedToken]],
      special_tokens: bool = False,
  ) -> int:
      """
      Add a list of new tokens to the tokenizer class. If the new tokens are not in the vocabulary, they are added to
      it with indices starting from length of the current vocabulary. Special tokens are sometimes already in the
      vocab which is why they have to be handled specifically.
  
      Args:
          new_tokens (`List[str]`or `List[tokenizers.AddedToken]`):
              Token(s) to add in vocabulary. A token is counted as added if it's not already in the vocabulary
              (tested by checking if the tokenizer assign the index of the `unk_token` to them). If a token is part
              of the vocabulary then we simply mark this token as an `AddedToken` which allows to control the
              stripping and normalization of this token. This is NOT possible in `tokenizers`.
          special_tokens (`bool`, *optional*, defaults to `False`):
              Whether or not the tokens should be added as special tokens.
  
      Returns:
          `int`: The number of tokens actually added to the vocabulary.
  
      Examples:
  
      ```python
      # Let's see how to increase the vocabulary of Bert model and tokenizer
      tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
      model = BertModel.from_pretrained("bert-base-uncased")
  
      num_added_toks = tokenizer.add_tokens(["new_tok1", "my_new-tok2"])
      print("We have added", num_added_toks, "tokens")
      # Note: resize_token_embeddings expects to receive the full size of the new vocabulary, i.e. the length of the tokenizer.
      model.resize_token_embeddings(len(tokenizer))
      ```"""
      added_tokens = 0
      if new_tokens is None:
          return added_tokens
      # TODO this is fairly slow to improve!
      current_vocab = self.get_vocab().copy()
      new_idx = len(
          current_vocab
      )  # only call this once, len gives the last index + 1
      for token in new_tokens:
          if not isinstance(token, (str, AddedToken)):
              raise TypeError(
                  f"Token {token} is not a string but a {type(token)}."
              )
          if str(token) == "":
              continue
          if isinstance(token, str):
              if token in self._added_tokens_encoder:
                  continue
              else:
                  # very important for fast and slow equivalence!
                  is_special = (
                      token in self.all_special_tokens or special_tokens
                  )
                  token = AddedToken(
                      token,
                      rstrip=False,
                      lstrip=False,
                      normalized=not is_special,
                      special=is_special,
                  )
          elif special_tokens:
              # doing token.special=True changes the normalization! will fix in rust
              # this is important and the only reason why the AddedTokens in each class are normalized by default
              token.__setstate__(
                  {"special": True, "normalized": token.normalized}
              )
          if token in self._added_tokens_decoder:
              continue
          if (
              not token.special
              and token.normalized
              and getattr(self, "do_lower_case", False)
          ):
              # Normalize if requested
              token.content = token.content.lower()
          if token.content not in current_vocab:
              token_index = new_idx + added_tokens
              current_vocab[token.content] = token_index
              added_tokens += 1
          else:
              token_index = current_vocab[token.content]
  
          if (
              token.special
              and str(token) not in self.all_special_tokens
          ):
              self._additional_special_tokens.append(token)
          # the setter automatically updates the reverse map
          self._added_tokens_decoder[token_index] = token
          self._added_tokens_encoder[token.content] = token_index
          if self.verbose:
              logger.info(f"Adding {token} to the vocabulary")
  
      self._update_trie()
      return added_tokens
  
imports: import bisect
  import itertools
  import re
  import unicodedata
  from collections import OrderedDict
  from typing import Any, Dict, List, Optional, Tuple, Union, overload
  from .tokenization_utils_base import ENCODE_KWARGS_DOCSTRING, ENCODE_PLUS_ADDITIONAL_KWARGS_DOCSTRING, INIT_TOKENIZER_DOCSTRING, AddedToken, BatchEncoding, EncodedInput, EncodedInputPair, PreTokenizedInput, PreTokenizedInputPair, PreTrainedTokenizerBase, TextInput, TextInputPair, TruncationStrategy
  from .utils import PaddingStrategy, TensorType, add_end_docstrings, logging
label_desc: Use image node_id 9 for calling the PreTrainedTokenizer obj's underlying member method code with example usage: obj._add_tokens(new_tokens, special_tokens) and returns: added_tokens, added_tokens