class_docstr: Construct an GPTSw3 tokenizer. Based on [SentencePiece](https://github.com/google/sentencepiece).
  
  This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to
  this superclass for more information regarding those methods.
  
  Example usage:
  ```python
  >>> from transformers import GPTSw3Tokenizer
  
  >>> tokenizer = GPTSw3Tokenizer.from_pretrained("AI-Sweden-Models/gpt-sw3-126m")
  >>> tokenizer("Svenska är kul!")["input_ids"]
  [1814, 377, 3617, 63504]
  ```
  
  Args:
      vocab_file (`str`):
          [SentencePiece](https://github.com/google/sentencepiece) file (generally has a *.spm* extension) that
          contains the vocabulary necessary to instantiate a tokenizer.
      do_lower_case (`bool`, *optional*, defaults to `False`):
          Whether or not to lowercase the input when tokenizing.
      remove_space (`bool`, *optional*, defaults to `False`):
          Whether or not to strip the text when tokenizing (removing excess spaces before and after the string).
      keep_accents (`bool`, *optional*, defaults to `False`):
          Whether or not to keep accents when tokenizing.
      pad_token (`str`, *optional*):
          The token used for padding, for example when batching sequences of different lengths. If not provided, will
          default to '<pad>' or '<unk>' depending on model size.
      unk_token (`str`, *optional*):
          The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
          token instead. If not provided, will default to '<unk>'.
      eos_token (`str`, *optional*):
          The end of sequence token seen during pretraining. If not provided, will default to '<|endoftext|>'
      bos_token (`str`, *optional*):
          The beginning of sequence token that can be used for downstream task, was not seen during pretraining. If
          not provided, will default to '<s>' or '<|endoftext|>', depending on model size.
      sp_model_kwargs (`dict`, *optional*):
          Will be passed to the `SentencePieceProcessor.__init__()` method. The [Python wrapper for
          SentencePiece](https://github.com/google/sentencepiece/tree/master/python) can be used, among other things,
          to set:
  
          - `enable_sampling`: Enable subword regularization.
          - `nbest_size`: Sampling parameters for unigram. Invalid for BPE-Dropout.
  
            - `nbest_size = {0,1}`: No sampling is performed.
            - `nbest_size > 1`: samples from the nbest_size results.
            - `nbest_size < 0`: assuming that nbest_size is infinite and samples from the all hypothesis (lattice)
              using forward-filtering-and-backward-sampling algorithm.
  
          - `alpha`: Smoothing parameter for unigram sampling, and dropout probability of merge operations for
            BPE-dropout.
  
  Attributes:
      sp_model (`SentencePieceProcessor`):
          The *SentencePiece* processor that is used for every conversion (string, tokens and IDs).
      whitespaces (`set`):
          The whitespaces that are replaced in the whitespace normalization in preprocessing.
      non_printing_characters_re (`Pattern`):
          The compiled regular expression to remove non-printing characters in preprocessing.
code: def __init__(
      self,
      vocab_file,
      do_lower_case=False,
      remove_space=False,
      keep_accents=False,
      pad_token=None,
      unk_token=None,
      eos_token=None,
      bos_token=None,
      sp_model_kwargs: Optional[Dict[str, Any]] = None,
      **kwargs,
  ) -> None:
      self.sp_model_kwargs = (
          {} if sp_model_kwargs is None else sp_model_kwargs
      )
  
      name_or_path = kwargs.get("name_or_path")
      if name_or_path is None:
          logger.warning(
              "name_or_path not provided, will work for all GPTSw3 models except gpt-sw3-7b,"
              " you are testing the model, this can safely be ignored"
          )
          name_or_path = "None"
  
      # Default definitions for our 2 tokenizer versions, with None-checks to enable proper testing
      eos_token = "<|endoftext|>" if eos_token is None else eos_token
      unk_token = "<unk>" if unk_token is None else unk_token
      if "gpt-sw3-7b" in name_or_path:
          pad_token = unk_token if pad_token is None else pad_token
          bos_token = eos_token if bos_token is None else bos_token
      else:
          pad_token = "<pad>" if pad_token is None else pad_token
          bos_token = "<s>" if bos_token is None else bos_token
  
      self.do_lower_case = do_lower_case
      self.remove_space = remove_space
      self.keep_accents = keep_accents
      self.vocab_file = vocab_file
  
      self.sp_model = spm.SentencePieceProcessor(**self.sp_model_kwargs)
      self.sp_model.Load(vocab_file)
  
      # Used for whitespace normalization in input texts
      # fmt : off
      self.whitespaces = {
          " ",
          " ",
          " ",
          " ",
          " ",
          "　",
          " ",
          " ",
          " ",
          " ",
          "￼",
          "",
      }
      # fmt : on
  
      # Regular expression to remove non-printing characters (e.g. some unicode control chars) in preprocessing
      self.non_printing_characters_re = re.compile(
          f"[{''.join(map(chr, list(range(0, 9)) + list(range(11, 32)) + list(range(127, 160)) + [160, 173, 8203]))}]"
      )
  
      super().__init__(
          do_lower_case=do_lower_case,
          remove_space=remove_space,
          keep_accents=keep_accents,
          bos_token=bos_token,
          eos_token=eos_token,
          unk_token=unk_token,
          pad_token=pad_token,
          sp_model_kwargs=self.sp_model_kwargs,
          **kwargs,
      )
  
imports: import os
  import re
  import unicodedata
  from shutil import copyfile
  from typing import Any, Dict, List, Optional, Tuple, Union
  import sentencepiece
  from ...tokenization_utils import PreTrainedTokenizer
  from ...utils import is_torch_available, logging
label_desc: Use image node_id 1 to create a new GPTSw3Tokenizer object from inherited base classes: PreTrainedTokenizer with example: obj = GPTSw3Tokenizer(vocab_file, do_lower_case, remove_space, keep_accents, pad_token, unk_token, eos_token, bos_token, sp_model_kwargs)