class_docstr: Construct a T5 tokenizer. Based on [SentencePiece](https://github.com/google/sentencepiece).
  
  This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to
  this superclass for more information regarding those methods.
  
  Args:
      vocab_file (`str`):
          [SentencePiece](https://github.com/google/sentencepiece) file (generally has a *.spm* extension) that
          contains the vocabulary necessary to instantiate a tokenizer.
      eos_token (`str`, *optional*, defaults to `"</s>"`):
          The end of sequence token.
  
          <Tip>
  
          When building a sequence using special tokens, this is not the token that is used for the end of sequence.
          The token used is the `sep_token`.
  
          </Tip>
  
      unk_token (`str`, *optional*, defaults to `"<unk>"`):
          The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
          token instead.
      pad_token (`str`, *optional*, defaults to `"<pad>"`):
          The token used for padding, for example when batching sequences of different lengths.
      extra_ids (`int`, *optional*, defaults to 100):
          Add a number of extra ids added to the end of the vocabulary for use as sentinels. These tokens are
          accessible as "<extra_id_{%d}>" where "{%d}" is a number between 0 and extra_ids-1. Extra tokens are
          indexed from the end of the vocabulary up to beginning ("<extra_id_0>" is the last token in the vocabulary
          like in T5 preprocessing see
          [here](https://github.com/google-research/text-to-text-transfer-transformer/blob/9fd7b14a769417be33bc6c850f9598764913c833/t5/data/preprocessors.py#L2117)).
      additional_special_tokens (`List[str]`, *optional*):
          Additional special tokens used by the tokenizer.
      sp_model_kwargs (`dict`, *optional*):
          Will be passed to the `SentencePieceProcessor.__init__()` method. The [Python wrapper for
          SentencePiece](https://github.com/google/sentencepiece/tree/master/python) can be used, among other things,
          to set:
  
          - `enable_sampling`: Enable subword regularization.
          - `nbest_size`: Sampling parameters for unigram. Invalid for BPE-Dropout.
  
            - `nbest_size = {0,1}`: No sampling is performed.
            - `nbest_size > 1`: samples from the nbest_size results.
            - `nbest_size < 0`: assuming that nbest_size is infinite and samples from the all hypothesis (lattice)
              using forward-filtering-and-backward-sampling algorithm.
  
          - `alpha`: Smoothing parameter for unigram sampling, and dropout probability of merge operations for
            BPE-dropout.
  
  Attributes:
      sp_model (`SentencePieceProcessor`):
          The *SentencePiece* processor that is used for every conversion (string, tokens and IDs).
code: def __init__(
      self,
      vocab_file,
      eos_token="</s>",
      unk_token="<unk>",
      pad_token="<pad>",
      extra_ids=100,
      additional_special_tokens=None,
      sp_model_kwargs: Optional[Dict[str, Any]] = None,
      **kwargs,
  ) -> None:
      # Add extra_ids to the special token list
      if extra_ids > 0 and additional_special_tokens is None:
          additional_special_tokens = [
              f"<extra_id_{i}>" for i in range(extra_ids)
          ]
      elif extra_ids > 0 and additional_special_tokens is not None:
          # Check that we have the right number of extra_id special tokens
          extra_tokens = len(
              set(
                  filter(
                      lambda x: bool("extra_id" in str(x)),
                      additional_special_tokens,
                  )
              )
          )
          if extra_tokens != extra_ids:
              raise ValueError(
                  f"Both extra_ids ({extra_ids}) and additional_special_tokens ({additional_special_tokens}) are"
                  " provided to T5Tokenizer. In this case the additional_special_tokens must include the extra_ids"
                  " tokens"
              )
  
      self.sp_model_kwargs = (
          {} if sp_model_kwargs is None else sp_model_kwargs
      )
  
      super().__init__(
          eos_token=eos_token,
          unk_token=unk_token,
          pad_token=pad_token,
          extra_ids=extra_ids,
          additional_special_tokens=additional_special_tokens,
          sp_model_kwargs=self.sp_model_kwargs,
          **kwargs,
      )
  
      self.vocab_file = vocab_file
      self._extra_ids = extra_ids
  
      self.sp_model = spm.SentencePieceProcessor(**self.sp_model_kwargs)
      self.sp_model.Load(vocab_file)
  
imports: import os
  import re
  import warnings
  from shutil import copyfile
  from typing import Any, Dict, List, Optional, Tuple
  import sentencepiece
  from ...tokenization_utils import PreTrainedTokenizer
  from ...utils import logging
label_desc: Use image node_id 1 to create a new T5Tokenizer object from inherited base classes: PreTrainedTokenizer with example: obj = T5Tokenizer(vocab_file, eos_token, unk_token, pad_token, extra_ids, additional_special_tokens, sp_model_kwargs)