class_docstr: Construct a "fast" CodeGen tokenizer (backed by HuggingFace's *tokenizers* library). Based on byte-level
  Byte-Pair-Encoding.
  
  This tokenizer has been trained to treat spaces like parts of the tokens (a bit like sentencepiece) so a word will
  be encoded differently whether it is at the beginning of the sentence (without space) or not:
  
  ```
  >>> from transformers import CodeGenTokenizerFast
  >>> tokenizer = CodeGenTokenizerFast.from_pretrained("Salesforce/codegen-350M-mono")
  >>> tokenizer("Hello world")['input_ids']
  [15496, 995]
  >>> tokenizer(" Hello world")['input_ids']
  [18435, 995]
  ```
  
  You can get around that behavior by passing `add_prefix_space=True` when instantiating this tokenizer, but since
  the model was not pretrained this way, it might yield a decrease in performance.
  
  <Tip>
  
  When used with `is_split_into_words=True`, this tokenizer needs to be instantiated with `add_prefix_space=True`.
  
  </Tip>
  
  This tokenizer inherits from [`PreTrainedTokenizerFast`] which contains most of the main methods. Users should
  refer to this superclass for more information regarding those methods.
  
  Args:
      vocab_file (`str`):
          Path to the vocabulary file.
      merges_file (`str`):
          Path to the merges file.
      errors (`str`, *optional*, defaults to `"replace"`):
          Paradigm to follow when decoding bytes to UTF-8. See
          [bytes.decode](https://docs.python.org/3/library/stdtypes.html#bytes.decode) for more information.
      unk_token (`str`, *optional*, defaults to `<|endoftext|>`):
          The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
          token instead.
      bos_token (`str`, *optional*, defaults to `<|endoftext|>`):
          The beginning of sequence token.
      eos_token (`str`, *optional*, defaults to `<|endoftext|>`):
          The end of sequence token.
      add_prefix_space (`bool`, *optional*, defaults to `False`):
          Whether or not to add an initial space to the input. This allows to treat the leading word just as any
          other word. (CodeGen tokenizer detect beginning of words by the preceding space).
      trim_offsets (`bool`, *optional*, defaults to `True`):
          Whether or not the post-processing step should trim offsets to avoid including whitespaces.
class_docstr_tok: ["Construct","a","``","fast","''","CodeGen","tokenizer","(","backed","by","HuggingFace","'s","*","tokenizers","*","library",")",".","Based","on","byte-level","Byte-Pair-Encoding",".","This","tokenizer","has","been","trained","to","treat","spaces","like","parts","of","the","tokens","(","a","bit","like","sentencepiece",")","so","a","word","will","be","encoded","differently","whether","it","is","at","the","beginning","of","the","sentence","(","without","space",")","or","not",":","``","`",">",">",">","from","transformers","import","CodeGenTokenizerFast",">",">",">","tokenizer","=","CodeGenTokenizerFast.from_pretrained","(","``","Salesforce\/codegen-350M-mono","''",")",">",">",">","tokenizer","(","``","Hello","world","''",")","[","'input_ids","'","]","[","15496",",","995","]",">",">",">","tokenizer","(","``","Hello","world","''",")","[","'input_ids","'","]","[","18435",",","995","]","``","`","You","can","get","around","that","behavior","by","passing","`","add_prefix_space=True","`","when","instantiating","this","tokenizer",",","but","since","the","model","was","not","pretrained","this","way",",","it","might","yield","a","decrease","in","performance",".","<","Tip",">","When","used","with","`","is_split_into_words=True","`",",","this","tokenizer","needs","to","be","instantiated","with","`","add_prefix_space=True","`",".","<","\/Tip",">","This","tokenizer","inherits","from","[","`","PreTrainedTokenizerFast","`","]","which","contains","most","of","the","main","methods",".","Users","should","refer","to","this","superclass","for","more","information","regarding","those","methods",".","Args",":","vocab_file","(","`","str","`",")",":","Path","to","the","vocabulary","file",".","merges_file","(","`","str","`",")",":","Path","to","the","merges","file",".","errors","(","`","str","`",",","*","optional","*",",","defaults","to","`","``","replace","''","`",")",":","Paradigm","to","follow","when","decoding","bytes","to","UTF-8",".","See","[","bytes.decode","]","(","https",":","\/\/docs.python.org\/3\/library\/stdtypes.html","#","bytes.decode",")","for","more","information",".","unk_token","(","`","str","`",",","*","optional","*",",","defaults","to","`","<","|endoftext|",">","`",")",":","The","unknown","token",".","A","token","that","is","not","in","the","vocabulary","can","not","be","converted","to","an","ID","and","is","set","to","be","this","token","instead",".","bos_token","(","`","str","`",",","*","optional","*",",","defaults","to","`","<","|endoftext|",">","`",")",":","The","beginning","of","sequence","token",".","eos_token","(","`","str","`",",","*","optional","*",",","defaults","to","`","<","|endoftext|",">","`",")",":","The","end","of","sequence","token",".","add_prefix_space","(","`","bool","`",",","*","optional","*",",","defaults","to","`","False","`",")",":","Whether","or","not","to","add","an","initial","space","to","the","input",".","This","allows","to","treat","the","leading","word","just","as","any","other","word",".","(","CodeGen","tokenizer","detect","beginning","of","words","by","the","preceding","space",")",".","trim_offsets","(","`","bool","`",",","*","optional","*",",","defaults","to","`","True","`",")",":","Whether","or","not","the","post-processing","step","should","trim","offsets","to","avoid","including","whitespaces","."]
docstr:         Converts a sequence of ids in a string, using the tokenizer and vocabulary with options to remove special
          tokens and clean up tokenization spaces.
  
          Similar to doing `self.convert_tokens_to_string(self.convert_ids_to_tokens(token_ids))`.
  
          Args:
              token_ids (`Union[int, List[int], np.ndarray, torch.Tensor, tf.Tensor]`):
                  List of tokenized input ids. Can be obtained using the `__call__` method.
              skip_special_tokens (`bool`, *optional*, defaults to `False`):
                  Whether or not to remove special tokens in the decoding.
              clean_up_tokenization_spaces (`bool`, *optional*, defaults to `True`):
                  Whether or not to clean up the tokenization spaces.
              truncate_before_pattern (`List[str]`, *optional*, defaults to `None`):
                  A list of regular expression strings that will be used to truncate the returned string. This can be
                  used to remove extra pieces of code (e.g. truncate if observing a comment symbol "#" at the beginning
                  of a new line). An example pattern could be `["^#", re.escape("<|endoftext|>"), "^'''", "
  
  
  "]`.
              kwargs (additional keyword arguments, *optional*):
                  Will be passed to the underlying model specific decode method.
  
          Returns:
              `str`: The decoded sentence.
          
docstr_tok: ["Converts","a","sequence","of","ids","in","a","string",",","using","the","tokenizer","and","vocabulary","with","options","to","remove","special","tokens","and","clean","up","tokenization","spaces",".","Similar","to","doing","`","self.convert_tokens_to_string","(","self.convert_ids_to_tokens","(","token_ids",")",")","`",".","Args",":","token_ids","(","`","Union","[","int",",","List","[","int","]",",","np.ndarray",",","torch.Tensor",",","tf.Tensor","]","`",")",":","List","of","tokenized","input","ids",".","Can","be","obtained","using","the","`","__call__","`","method",".","skip_special_tokens","(","`","bool","`",",","*","optional","*",",","defaults","to","`","False","`",")",":","Whether","or","not","to","remove","special","tokens","in","the","decoding",".","clean_up_tokenization_spaces","(","`","bool","`",",","*","optional","*",",","defaults","to","`","True","`",")",":","Whether","or","not","to","clean","up","the","tokenization","spaces",".","truncate_before_pattern","(","`","List","[","str","]","`",",","*","optional","*",",","defaults","to","`","None","`",")",":","A","list","of","regular","expression","strings","that","will","be","used","to","truncate","the","returned","string",".","This","can","be","used","to","remove","extra","pieces","of","code","(","e.g",".","truncate","if","observing","a","comment","symbol","``","#","''","at","the","beginning","of","a","new","line",")",".","An","example","pattern","could","be","`","[","``","^","#","''",",","re.escape","(","``","<","|endoftext|",">","''",")",",","``","^","''","'","''",",","``","''","]","`",".","kwargs","(","additional","keyword","arguments",",","*","optional","*",")",":","Will","be","passed","to","the","underlying","model","specific","decode","method",".","Returns",":","`","str","`",":","The","decoded","sentence","."]
code: def decode(
      self,
      token_ids: Union[
          int, List[int], "np.ndarray", "torch.Tensor", "tf.Tensor"
      ],
      skip_special_tokens: bool = False,
      clean_up_tokenization_spaces: bool = True,
      truncate_before_pattern: Optional[List[str]] = None,
      **kwargs,
  ) -> str:
      """
      Converts a sequence of ids in a string, using the tokenizer and vocabulary with options to remove special
      tokens and clean up tokenization spaces.
  
      Similar to doing `self.convert_tokens_to_string(self.convert_ids_to_tokens(token_ids))`.
  
      Args:
          token_ids (`Union[int, List[int], np.ndarray, torch.Tensor, tf.Tensor]`):
              List of tokenized input ids. Can be obtained using the `__call__` method.
          skip_special_tokens (`bool`, *optional*, defaults to `False`):
              Whether or not to remove special tokens in the decoding.
          clean_up_tokenization_spaces (`bool`, *optional*, defaults to `True`):
              Whether or not to clean up the tokenization spaces.
          truncate_before_pattern (`List[str]`, *optional*, defaults to `None`):
              A list of regular expression strings that will be used to truncate the returned string. This can be
              used to remove extra pieces of code (e.g. truncate if observing a comment symbol "#" at the beginning
              of a new line). An example pattern could be `["^#", re.escape("<|endoftext|>"), "^'''", "\n\n\n"]`.
          kwargs (additional keyword arguments, *optional*):
              Will be passed to the underlying model specific decode method.
  
      Returns:
          `str`: The decoded sentence.
      """
  
      decoded_text = super().decode(
          token_ids=token_ids,
          skip_special_tokens=skip_special_tokens,
          clean_up_tokenization_spaces=clean_up_tokenization_spaces,
          **kwargs,
      )
  
      if (
          truncate_before_pattern is not None
          and len(truncate_before_pattern) > 0
      ):
          decoded_text = self.truncate(
              decoded_text, truncate_before_pattern
          )
  
      return decoded_text
  
file_path: H2O/h2o_flexgen/benchmark/third_party/transformers/src/transformers/models/codegen/tokenization_codegen_fast.py
imports: import json
  import re
  from typing import TYPE_CHECKING, List, Optional, Tuple, Union
  import numpy
  from ...utils import is_tf_available, is_torch_available, logging
  from tokenizers import pre_tokenizers
  from ...tokenization_utils_base import BatchEncoding
  from ...tokenization_utils_fast import PreTrainedTokenizerFast
  from .tokenization_codegen import CodeGenTokenizer
label_desc: Use image node_id 5 for calling the CodeGenTokenizerFast obj's underlying member method code with example usage: obj.decode(token_ids, skip_special_tokens, clean_up_tokenization_spaces, truncate_before_pattern) and returns: decoded_text