class_docstr: Construct a "fast" CodeGen tokenizer (backed by HuggingFace's *tokenizers* library). Based on byte-level
  Byte-Pair-Encoding.
  
  This tokenizer has been trained to treat spaces like parts of the tokens (a bit like sentencepiece) so a word will
  be encoded differently whether it is at the beginning of the sentence (without space) or not:
  
  ```python
  >>> from transformers import CodeGenTokenizerFast
  
  >>> tokenizer = CodeGenTokenizerFast.from_pretrained("Salesforce/codegen-350M-mono")
  >>> tokenizer("Hello world")["input_ids"]
  [15496, 995]
  
  >>> tokenizer(" Hello world")["input_ids"]
  [18435, 995]
  ```
  
  You can get around that behavior by passing `add_prefix_space=True` when instantiating this tokenizer, but since
  the model was not pretrained this way, it might yield a decrease in performance.
  
  <Tip>
  
  When used with `is_split_into_words=True`, this tokenizer needs to be instantiated with `add_prefix_space=True`.
  
  </Tip>
  
  This tokenizer inherits from [`PreTrainedTokenizerFast`] which contains most of the main methods. Users should
  refer to this superclass for more information regarding those methods.
  
  Args:
      vocab_file (`str`, *optional*):
          Path to the vocabulary file.
      merges_file (`str`, *optional*):
          Path to the merges file.
      tokenizer_file (`str`, *optional*):
          Path to [tokenizers](https://github.com/huggingface/tokenizers) file (generally has a .json extension) that
          contains everything needed to load the tokenizer.
      unk_token (`str`, *optional*, defaults to `"<|endoftext|>"`):
          The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
          token instead.
      bos_token (`str`, *optional*, defaults to `"<|endoftext|>"`):
          The beginning of sequence token.
      eos_token (`str`, *optional*, defaults to `"<|endoftext|>"`):
          The end of sequence token.
      add_prefix_space (`bool`, *optional*, defaults to `False`):
          Whether or not to add an initial space to the input. This allows to treat the leading word just as any
          other word. (CodeGen tokenizer detect beginning of words by the preceding space).
class_docstr_tok: ["Construct","a","``","fast","''","CodeGen","tokenizer","(","backed","by","HuggingFace","'s","*","tokenizers","*","library",")",".","Based","on","byte-level","Byte-Pair-Encoding",".","This","tokenizer","has","been","trained","to","treat","spaces","like","parts","of","the","tokens","(","a","bit","like","sentencepiece",")","so","a","word","will","be","encoded","differently","whether","it","is","at","the","beginning","of","the","sentence","(","without","space",")","or","not",":","``","`","python",">",">",">","from","transformers","import","CodeGenTokenizerFast",">",">",">","tokenizer","=","CodeGenTokenizerFast.from_pretrained","(","``","Salesforce\/codegen-350M-mono","''",")",">",">",">","tokenizer","(","``","Hello","world","''",")","[","``","input_ids","''","]","[","15496",",","995","]",">",">",">","tokenizer","(","``","Hello","world","''",")","[","``","input_ids","''","]","[","18435",",","995","]","``","`","You","can","get","around","that","behavior","by","passing","`","add_prefix_space=True","`","when","instantiating","this","tokenizer",",","but","since","the","model","was","not","pretrained","this","way",",","it","might","yield","a","decrease","in","performance",".","<","Tip",">","When","used","with","`","is_split_into_words=True","`",",","this","tokenizer","needs","to","be","instantiated","with","`","add_prefix_space=True","`",".","<","\/Tip",">","This","tokenizer","inherits","from","[","`","PreTrainedTokenizerFast","`","]","which","contains","most","of","the","main","methods",".","Users","should","refer","to","this","superclass","for","more","information","regarding","those","methods",".","Args",":","vocab_file","(","`","str","`",",","*","optional","*",")",":","Path","to","the","vocabulary","file",".","merges_file","(","`","str","`",",","*","optional","*",")",":","Path","to","the","merges","file",".","tokenizer_file","(","`","str","`",",","*","optional","*",")",":","Path","to","[","tokenizers","]","(","https",":","\/\/github.com\/huggingface\/tokenizers",")","file","(","generally","has","a",".json","extension",")","that","contains","everything","needed","to","load","the","tokenizer",".","unk_token","(","`","str","`",",","*","optional","*",",","defaults","to","`","``","<","|endoftext|",">","''","`",")",":","The","unknown","token",".","A","token","that","is","not","in","the","vocabulary","can","not","be","converted","to","an","ID","and","is","set","to","be","this","token","instead",".","bos_token","(","`","str","`",",","*","optional","*",",","defaults","to","`","``","<","|endoftext|",">","''","`",")",":","The","beginning","of","sequence","token",".","eos_token","(","`","str","`",",","*","optional","*",",","defaults","to","`","``","<","|endoftext|",">","''","`",")",":","The","end","of","sequence","token",".","add_prefix_space","(","`","bool","`",",","*","optional","*",",","defaults","to","`","False","`",")",":","Whether","or","not","to","add","an","initial","space","to","the","input",".","This","allows","to","treat","the","leading","word","just","as","any","other","word",".","(","CodeGen","tokenizer","detect","beginning","of","words","by","the","preceding","space",")","."]
code: def _batch_encode_plus(self, *args, **kwargs) -> BatchEncoding:
      is_split_into_words = kwargs.get("is_split_into_words", False)
      assert self.add_prefix_space or not is_split_into_words, (
          f"You need to instantiate {self.__class__.__name__} with add_prefix_space=True "
          "to use it with pretokenized inputs."
      )
  
      return super()._batch_encode_plus(*args, **kwargs)
  
file_path: transformers/src/transformers/models/codegen/tokenization_codegen_fast.py
imports: import json
  import re
  from typing import TYPE_CHECKING, List, Optional, Tuple, Union
  import numpy
  from ...utils import is_tf_available, is_torch_available, logging
  from tokenizers import pre_tokenizers
  from ...tokenization_utils_base import BatchEncoding
  from ...tokenization_utils_fast import PreTrainedTokenizerFast
  from .tokenization_codegen import CodeGenTokenizer
label_desc: Use image node_id 2 for calling the CodeGenTokenizerFast obj's underlying member method code with example usage: obj._batch_encode_plus() and returns: super