class_docstr: Construct a T5 tokenizer. Based on [SentencePiece](https://github.com/google/sentencepiece).
  
  This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to
  this superclass for more information regarding those methods.
  
  Args:
      vocab_file (`str`):
          [SentencePiece](https://github.com/google/sentencepiece) file (generally has a *.spm* extension) that
          contains the vocabulary necessary to instantiate a tokenizer.
      eos_token (`str`, *optional*, defaults to `"</s>"`):
          The end of sequence token.
  
          <Tip>
  
          When building a sequence using special tokens, this is not the token that is used for the end of sequence.
          The token used is the `sep_token`.
  
          </Tip>
  
      unk_token (`str`, *optional*, defaults to `"<unk>"`):
          The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
          token instead.
      pad_token (`str`, *optional*, defaults to `"<pad>"`):
          The token used for padding, for example when batching sequences of different lengths.
      extra_ids (`int`, *optional*, defaults to 100):
          Add a number of extra ids added to the end of the vocabulary for use as sentinels. These tokens are
          accessible as "<extra_id_{%d}>" where "{%d}" is a number between 0 and extra_ids-1. Extra tokens are
          indexed from the end of the vocabulary up to beginning ("<extra_id_0>" is the last token in the vocabulary
          like in T5 preprocessing see
          [here](https://github.com/google-research/text-to-text-transfer-transformer/blob/9fd7b14a769417be33bc6c850f9598764913c833/t5/data/preprocessors.py#L2117)).
      additional_special_tokens (`List[str]`, *optional*):
          Additional special tokens used by the tokenizer.
      sp_model_kwargs (`dict`, *optional*):
          Will be passed to the `SentencePieceProcessor.__init__()` method. The [Python wrapper for
          SentencePiece](https://github.com/google/sentencepiece/tree/master/python) can be used, among other things,
          to set:
  
          - `enable_sampling`: Enable subword regularization.
          - `nbest_size`: Sampling parameters for unigram. Invalid for BPE-Dropout.
  
            - `nbest_size = {0,1}`: No sampling is performed.
            - `nbest_size > 1`: samples from the nbest_size results.
            - `nbest_size < 0`: assuming that nbest_size is infinite and samples from the all hypothesis (lattice)
              using forward-filtering-and-backward-sampling algorithm.
  
          - `alpha`: Smoothing parameter for unigram sampling, and dropout probability of merge operations for
            BPE-dropout.
  
  Attributes:
      sp_model (`SentencePieceProcessor`):
          The *SentencePiece* processor that is used for every conversion (string, tokens and IDs).
docstr: Take as input a string and return a list of strings (tokens) for words/sub-words
docstr_tok: ["Take","as","input","a","string","and","return","a","list","of","strings","(","tokens",")","for","words\/sub-words"]
code: def _tokenize(self, text: str) -> List[str]:
      """Take as input a string and return a list of strings (tokens) for words/sub-words"""
      return self.sp_model.encode(text, out_type=str)
  
imports: import os
  import re
  import warnings
  from shutil import copyfile
  from typing import Any, Dict, List, Optional, Tuple
  import sentencepiece
  from ...tokenization_utils import PreTrainedTokenizer
  from ...utils import logging
label_desc: Use image node_id 11 for calling the T5Tokenizer obj's underlying member method code with example usage: obj._tokenize(text) and returns: self