class_docstr: Construct an GPTSw3 tokenizer. Based on [SentencePiece](https://github.com/google/sentencepiece).
  
  This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to
  this superclass for more information regarding those methods.
  
  Example usage:
  ```python
  >>> from transformers import GPTSw3Tokenizer
  
  >>> tokenizer = GPTSw3Tokenizer.from_pretrained("AI-Sweden-Models/gpt-sw3-126m")
  >>> tokenizer("Svenska är kul!")["input_ids"]
  [1814, 377, 3617, 63504]
  ```
  
  Args:
      vocab_file (`str`):
          [SentencePiece](https://github.com/google/sentencepiece) file (generally has a *.spm* extension) that
          contains the vocabulary necessary to instantiate a tokenizer.
      do_lower_case (`bool`, *optional*, defaults to `False`):
          Whether or not to lowercase the input when tokenizing.
      remove_space (`bool`, *optional*, defaults to `False`):
          Whether or not to strip the text when tokenizing (removing excess spaces before and after the string).
      keep_accents (`bool`, *optional*, defaults to `False`):
          Whether or not to keep accents when tokenizing.
      pad_token (`str`, *optional*):
          The token used for padding, for example when batching sequences of different lengths. If not provided, will
          default to '<pad>' or '<unk>' depending on model size.
      unk_token (`str`, *optional*):
          The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
          token instead. If not provided, will default to '<unk>'.
      eos_token (`str`, *optional*):
          The end of sequence token seen during pretraining. If not provided, will default to '<|endoftext|>'
      bos_token (`str`, *optional*):
          The beginning of sequence token that can be used for downstream task, was not seen during pretraining. If
          not provided, will default to '<s>' or '<|endoftext|>', depending on model size.
      sp_model_kwargs (`dict`, *optional*):
          Will be passed to the `SentencePieceProcessor.__init__()` method. The [Python wrapper for
          SentencePiece](https://github.com/google/sentencepiece/tree/master/python) can be used, among other things,
          to set:
  
          - `enable_sampling`: Enable subword regularization.
          - `nbest_size`: Sampling parameters for unigram. Invalid for BPE-Dropout.
  
            - `nbest_size = {0,1}`: No sampling is performed.
            - `nbest_size > 1`: samples from the nbest_size results.
            - `nbest_size < 0`: assuming that nbest_size is infinite and samples from the all hypothesis (lattice)
              using forward-filtering-and-backward-sampling algorithm.
  
          - `alpha`: Smoothing parameter for unigram sampling, and dropout probability of merge operations for
            BPE-dropout.
  
  Attributes:
      sp_model (`SentencePieceProcessor`):
          The *SentencePiece* processor that is used for every conversion (string, tokens and IDs).
      whitespaces (`set`):
          The whitespaces that are replaced in the whitespace normalization in preprocessing.
      non_printing_characters_re (`Pattern`):
          The compiled regular expression to remove non-printing characters in preprocessing.
docstr: Encodes a text or batch of texts to token ids using preprocessing and the raw SP tokenizer. This has reduced
  functionality but is often much faster.
  
  Does NOT handle special tokens correctly, these can manually be added as ids afterwards.
  
  Does NOT support padding, these can manually be added as ids afterwards.
  
  Use default HuggingFace tokenization methods for full functionality.
  
  Args:
      text (`str` or `List[str]`): One or several text(s) to convert to token ids.
      return_tensors (`str` or `bool`): Returns PyTorch tensors if set to True or "pt"
  
  Returns:
      `List[int]`, `List[List[int]]`, or `torch.Tensor`: The encoded text(s) as token ids.
docstr_tok: ["Encodes","a","text","or","batch","of","texts","to","token","ids","using","preprocessing","and","the","raw","SP","tokenizer",".","This","has","reduced","functionality","but","is","often","much","faster",".","Does","NOT","handle","special","tokens","correctly",",","these","can","manually","be","added","as","ids","afterwards",".","Does","NOT","support","padding",",","these","can","manually","be","added","as","ids","afterwards",".","Use","default","HuggingFace","tokenization","methods","for","full","functionality",".","Args",":","text","(","`","str","`","or","`","List","[","str","]","`",")",":","One","or","several","text","(","s",")","to","convert","to","token","ids",".","return_tensors","(","`","str","`","or","`","bool","`",")",":","Returns","PyTorch","tensors","if","set","to","True","or","``","pt","''","Returns",":","`","List","[","int","]","`",",","`","List","[","List","[","int","]","]","`",",","or","`","torch.Tensor","`",":","The","encoded","text","(","s",")","as","token","ids","."]
code: def encode_fast(
      self,
      text: Union[str, List[str]],
      return_tensors: Union[str, bool] = False,
  ) -> Union[List[int], List[List[int]], "torch.Tensor"]:
      """
      Encodes a text or batch of texts to token ids using preprocessing and the raw SP tokenizer. This has reduced
      functionality but is often much faster.
  
      Does NOT handle special tokens correctly, these can manually be added as ids afterwards.
  
      Does NOT support padding, these can manually be added as ids afterwards.
  
      Use default HuggingFace tokenization methods for full functionality.
  
      Args:
          text (`str` or `List[str]`): One or several text(s) to convert to token ids.
          return_tensors (`str` or `bool`): Returns PyTorch tensors if set to True or "pt"
  
      Returns:
          `List[int]`, `List[List[int]]`, or `torch.Tensor`: The encoded text(s) as token ids.
      """
  
      if isinstance(text, str):
          text = self.preprocess_text(text)
          token_ids = self.sp_model.encode(text)
      else:
          text = [self.preprocess_text(t) for t in text]
          token_ids = self.sp_model.encode(text)
  
      if return_tensors is True or return_tensors == "pt":
          token_ids = torch.tensor(token_ids)
  
      return token_ids
  
imports: import os
  import re
  import unicodedata
  from shutil import copyfile
  from typing import Any, Dict, List, Optional, Tuple, Union
  import sentencepiece
  from ...tokenization_utils import PreTrainedTokenizer
  from ...utils import is_torch_available, logging
label_desc: Use image node_id 13 for calling the GPTSw3Tokenizer obj's underlying member method code with example usage: obj.encode_fast(text, return_tensors) and returns: token_ids