class_docstr: Base class for all fast tokenizers (wrapping HuggingFace tokenizers library).
  
  Inherits from [`~tokenization_utils_base.PreTrainedTokenizerBase`].
  
  Handles all the shared methods for tokenization and special tokens, as well as methods for
  downloading/caching/loading pretrained tokenizers, as well as adding tokens to the vocabulary.
  
  This class also contains the added tokens in a unified way on top of all tokenizers so we don't have to handle the
  specific vocabulary augmentation methods of the various underlying dictionary structures (BPE, sentencepiece...).
class_docstr_tok: ["Base","class","for","all","fast","tokenizers","(","wrapping","HuggingFace","tokenizers","library",")",".","Inherits","from","[","`","~tokenization_utils_base.PreTrainedTokenizerBase","`","]",".","Handles","all","the","shared","methods","for","tokenization","and","special","tokens",",","as","well","as","methods","for","downloading\/caching\/loading","pretrained","tokenizers",",","as","well","as","adding","tokens","to","the","vocabulary",".","This","class","also","contains","the","added","tokens","in","a","unified","way","on","top","of","all","tokenizers","so","we","do","n't","have","to","handle","the","specific","vocabulary","augmentation","methods","of","the","various","underlying","dictionary","structures","(","BPE",",","sentencepiece","...",")","."]
docstr: Define the truncation and the padding strategies for fast tokenizers (provided by HuggingFace tokenizers
  library) and restore the tokenizer settings afterwards.
  
  The provided tokenizer has no padding / truncation strategy before the managed section. If your tokenizer set a
  padding / truncation strategy before, then it will be reset to no padding / truncation when exiting the managed
  section.
  
  Args:
      padding_strategy ([`~utils.PaddingStrategy`]):
          The kind of padding that will be applied to the input
      truncation_strategy ([`~tokenization_utils_base.TruncationStrategy`]):
          The kind of truncation that will be applied to the input
      max_length (`int`):
          The maximum size of a sequence.
      stride (`int`):
          The stride to use when handling overflow.
      pad_to_multiple_of (`int`, *optional*):
          If set will pad the sequence to a multiple of the provided value. This is especially useful to enable
          the use of Tensor Cores on NVIDIA hardware with compute capability >= 7.5 (Volta).
docstr_tok: ["Define","the","truncation","and","the","padding","strategies","for","fast","tokenizers","(","provided","by","HuggingFace","tokenizers","library",")","and","restore","the","tokenizer","settings","afterwards",".","The","provided","tokenizer","has","no","padding","\/","truncation","strategy","before","the","managed","section",".","If","your","tokenizer","set","a","padding","\/","truncation","strategy","before",",","then","it","will","be","reset","to","no","padding","\/","truncation","when","exiting","the","managed","section",".","Args",":","padding_strategy","(","[","`","~utils.PaddingStrategy","`","]",")",":","The","kind","of","padding","that","will","be","applied","to","the","input","truncation_strategy","(","[","`","~tokenization_utils_base.TruncationStrategy","`","]",")",":","The","kind","of","truncation","that","will","be","applied","to","the","input","max_length","(","`","int","`",")",":","The","maximum","size","of","a","sequence",".","stride","(","`","int","`",")",":","The","stride","to","use","when","handling","overflow",".","pad_to_multiple_of","(","`","int","`",",","*","optional","*",")",":","If","set","will","pad","the","sequence","to","a","multiple","of","the","provided","value",".","This","is","especially","useful","to","enable","the","use","of","Tensor","Cores","on","NVIDIA","hardware","with","compute","capability",">","=","7.5","(","Volta",")","."]
code: def set_truncation_and_padding(
      self,
      padding_strategy: PaddingStrategy,
      truncation_strategy: TruncationStrategy,
      max_length: int,
      stride: int,
      pad_to_multiple_of: Optional[int],
  ):
      """
      Define the truncation and the padding strategies for fast tokenizers (provided by HuggingFace tokenizers
      library) and restore the tokenizer settings afterwards.
  
      The provided tokenizer has no padding / truncation strategy before the managed section. If your tokenizer set a
      padding / truncation strategy before, then it will be reset to no padding / truncation when exiting the managed
      section.
  
      Args:
          padding_strategy ([`~utils.PaddingStrategy`]):
              The kind of padding that will be applied to the input
          truncation_strategy ([`~tokenization_utils_base.TruncationStrategy`]):
              The kind of truncation that will be applied to the input
          max_length (`int`):
              The maximum size of a sequence.
          stride (`int`):
              The stride to use when handling overflow.
          pad_to_multiple_of (`int`, *optional*):
              If set will pad the sequence to a multiple of the provided value. This is especially useful to enable
              the use of Tensor Cores on NVIDIA hardware with compute capability >= 7.5 (Volta).
      """
      _truncation = self._tokenizer.truncation
      _padding = self._tokenizer.padding
      # Set truncation and padding on the backend tokenizer
      if truncation_strategy == TruncationStrategy.DO_NOT_TRUNCATE:
          if _truncation is not None:
              self._tokenizer.no_truncation()
      else:
          target = {
              "max_length": max_length,
              "stride": stride,
              "strategy": truncation_strategy.value,
              "direction": self.truncation_side,
          }
  
          # _truncation might contain more keys that the target `transformers`
          # supports. Use only the target keys to trigger `enable_truncation`.
          # This should enable this code to works on various `tokenizers`
          # targets.
          if _truncation is None:
              current = None
          else:
              current = {k: _truncation.get(k, None) for k in target}
  
          if current != target:
              self._tokenizer.enable_truncation(**target)
  
      if padding_strategy == PaddingStrategy.DO_NOT_PAD:
          if _padding is not None:
              self._tokenizer.no_padding()
      else:
          length = (
              max_length
              if padding_strategy == PaddingStrategy.MAX_LENGTH
              else None
          )
          target = {
              "length": length,
              "direction": self.padding_side,
              "pad_id": self.pad_token_id,
              "pad_token": self.pad_token,
              "pad_type_id": self.pad_token_type_id,
              "pad_to_multiple_of": pad_to_multiple_of,
          }
          if _padding != target:
              self._tokenizer.enable_padding(**target)
  
imports: import copy
  import json
  import os
  from collections import defaultdict
  from typing import Any, Dict, List, Optional, Tuple, Union
  import tokenizers.pre_tokenizers
  from tokenizers import Encoding
  from tokenizers import Tokenizer
  from tokenizers.decoders import Decoder
  from tokenizers.trainers import BpeTrainer, UnigramTrainer, WordLevelTrainer, WordPieceTrainer
  from .convert_slow_tokenizer import convert_slow_tokenizer
  from .tokenization_utils import PreTrainedTokenizer
  from .tokenization_utils_base import INIT_TOKENIZER_DOCSTRING, AddedToken, BatchEncoding, PreTokenizedInput, PreTokenizedInputPair, PreTrainedTokenizerBase, SpecialTokensMixin, TextInput, TextInputPair, TruncationStrategy
  from .utils import PaddingStrategy, add_end_docstrings, logging
label_desc: Use image node_id 18 for calling the PreTrainedTokenizerFast obj's underlying member method code with example usage: obj.set_truncation_and_padding(padding_strategy, truncation_strategy, max_length, stride, pad_to_multiple_of) without return types