class_docstr: Construct a Siglip tokenizer. Based on [SentencePiece](https://github.com/google/sentencepiece).
  
  This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to
  this superclass for more information regarding those methods.
  
  Args:
      vocab_file (`str`):
          [SentencePiece](https://github.com/google/sentencepiece) file (generally has a *.spm* extension) that
          contains the vocabulary necessary to instantiate a tokenizer.
      eos_token (`str`, *optional*, defaults to `"</s>"`):
          The end of sequence token.
      unk_token (`str`, *optional*, defaults to `"<unk>"`):
          The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
          token instead.
      pad_token (`str`, *optional*, defaults to `"</s>"`):
          The token used for padding, for example when batching sequences of different lengths.
      additional_special_tokens (`List[str]`, *optional*):
          Additional special tokens used by the tokenizer.
      sp_model_kwargs (`dict`, *optional*):
          Will be passed to the `SentencePieceProcessor.__init__()` method. The [Python wrapper for
          SentencePiece](https://github.com/google/sentencepiece/tree/master/python) can be used, among other things,
          to set:
  
          - `enable_sampling`: Enable subword regularization.
          - `nbest_size`: Sampling parameters for unigram. Invalid for BPE-Dropout.
  
            - `nbest_size = {0,1}`: No sampling is performed.
            - `nbest_size > 1`: samples from the nbest_size results.
            - `nbest_size < 0`: assuming that nbest_size is infinite and samples from the all hypothesis (lattice)
              using forward-filtering-and-backward-sampling algorithm.
  
          - `alpha`: Smoothing parameter for unigram sampling, and dropout probability of merge operations for
            BPE-dropout.
      model_max_length (`int`, *optional*, defaults to 64):
          The maximum length (in number of tokens) for model inputs.
      do_lower_case (`bool`, *optional*, defaults to `True`):
          Whether or not to lowercase the input when tokenizing.
docstr: Returns a tokenized string.
  
  We de-activated the `add_dummy_prefix` option, thus the sentencepiece internals will always strip any
  SPIECE_UNDERLINE.
  
  For example: `self.sp_model.encode(f"{SPIECE_UNDERLINE}Hey", out_type = str)` will give `['H', 'e', 'y']` instead of `['▁He', 'y']`.
  
  Thus we always encode `f"{unk_token}text"` and strip the `unk_token`. Here is an example with `unk_token = "<unk>"` and `unk_token_length = 4`.
  `self.tokenizer.sp_model.encode("<unk> Hey", out_type = str)[4:]`.
docstr_tok: ["Returns","a","tokenized","string",".","We","de-activated","the","`","add_dummy_prefix","`","option",",","thus","the","sentencepiece","internals","will","always","strip","any","SPIECE_UNDERLINE",".","For","example",":","`","self.sp_model.encode","(","f","''","{","SPIECE_UNDERLINE","}","Hey","''",",","out_type","=","str",")","`","will","give","`","[","'","H","'",",","'","e","'",",","'","y","'","]","`","instead","of","`","[","'\u2581He","'",",","'","y","'","]","`",".","Thus","we","always","encode","`","f","''","{","unk_token","}","text","''","`","and","strip","the","`","unk_token","`",".","Here","is","an","example","with","`","unk_token","=","``","<","unk",">","''","`","and","`","unk_token_length","=","4","`",".","`","self.tokenizer.sp_model.encode","(","``","<","unk",">","Hey","''",",","out_type","=","str",")","[","4",":","]","`","."]
code: def _tokenize(self, text, **kwargs):
      """
      Returns a tokenized string.
  
      We de-activated the `add_dummy_prefix` option, thus the sentencepiece internals will always strip any
      SPIECE_UNDERLINE.
  
      For example: `self.sp_model.encode(f"{SPIECE_UNDERLINE}Hey", out_type = str)` will give `['H', 'e', 'y']` instead of `['▁He', 'y']`.
  
      Thus we always encode `f"{unk_token}text"` and strip the `unk_token`. Here is an example with `unk_token = "<unk>"` and `unk_token_length = 4`.
      `self.tokenizer.sp_model.encode("<unk> Hey", out_type = str)[4:]`.
      """
      text = self.canonicalize_text(
          text, keep_punctuation_exact_string=None
      )
      tokens = self.sp_model.encode(text, out_type=str)
  
      # 1. Encode string + prefix ex: "<unk> Hey"
      tokens = self.sp_model.encode(self.unk_token + text, out_type=str)
      # 2. Remove self.unk_token from ['<','unk','>', '▁Hey']
      return (
          tokens[self.unk_token_length :]
          if len(tokens) >= self.unk_token_length
          else tokens
      )
  
imports: import os
  import re
  import string
  import warnings
  from shutil import copyfile
  from typing import TYPE_CHECKING, Any, Dict, List, Optional, Tuple
  import sentencepiece
  from ...convert_slow_tokenizer import import_protobuf
  from ...tokenization_utils import PreTrainedTokenizer
  from ...tokenization_utils_base import AddedToken
  from ...utils import logging, requires_backends
label_desc: Use image node_id 15 for calling the SiglipTokenizer obj's underlying member method code with example usage: obj._tokenize(text) and returns: unknown