class_docstr: Construct a Llama tokenizer. Based on byte-level Byte-Pair-Encoding. The default padding token is unset as there is
  no padding token in the original model.
  
  Args:
      vocab_file (`str`):
          Path to the vocabulary file.
      unk_token (`str` or `tokenizers.AddedToken`, *optional*, defaults to `"<unk>"`):
          The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
          token instead.
      bos_token (`str` or `tokenizers.AddedToken`, *optional*, defaults to `"<s>"`):
          The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token.
      eos_token (`str` or `tokenizers.AddedToken`, *optional*, defaults to `"</s>"`):
          The end of sequence token.
      pad_token (`str` or `tokenizers.AddedToken`, *optional*):
          A special token used to make arrays of tokens the same size for batching purpose. Will then be ignored by
          attention mechanisms or loss computation.
      sp_model_kwargs (`Dict[str, Any]`, `Optional`, *optional*):
          Will be passed to the `SentencePieceProcessor.__init__()` method. The [Python wrapper for
          SentencePiece](https://github.com/google/sentencepiece/tree/master/python) can be used, among other things,
          to set:
  
          - `enable_sampling`: Enable subword regularization.
          - `nbest_size`: Sampling parameters for unigram. Invalid for BPE-Dropout.
  
            - `nbest_size = {0,1}`: No sampling is performed.
            - `nbest_size > 1`: samples from the nbest_size results.
            - `nbest_size < 0`: assuming that nbest_size is infinite and samples from the all hypothesis (lattice)
              using forward-filtering-and-backward-sampling algorithm.
  
          - `alpha`: Smoothing parameter for unigram sampling, and dropout probability of merge operations for
            BPE-dropout.
  
      add_bos_token (`bool`, *optional*, defaults to `True`):
          Whether or not to add an `bos_token` at the start of sequences.
      add_eos_token (`bool`, *optional*, defaults to `False`):
          Whether or not to add an `eos_token` at the end of sequences.
      clean_up_tokenization_spaces (`bool`, *optional*, defaults to `False`):
          Whether or not to cleanup spaces after decoding, cleanup consists in removing potential artifacts like
          extra spaces.
      use_default_system_prompt (`bool`, *optional*, defaults to `False`):
          Whether or not the default system prompt for Llama should be used.
      spaces_between_special_tokens (`bool`, *optional*, defaults to `False`):
          Whether or not to add spaces between special tokens.
      legacy (`bool`, *optional*):
          Whether or not the `legacy` behavior of the tokenizer should be used. Legacy is before the merge of #24622
          and #25224 which includes fixes to properly handle tokens that appear after special tokens. A simple
          example:
  
          - `legacy=True`:
          ```python
          >>> from transformers import T5Tokenizer
  
          >>> tokenizer = T5Tokenizer.from_pretrained("t5-base", legacy=True)
          >>> tokenizer.encode("Hello <extra_id_0>.")
          [8774, 32099, 3, 5, 1]
          ```
          - `legacy=False`:
          ```python
          >>> from transformers import T5Tokenizer
  
          >>> tokenizer = T5Tokenizer.from_pretrained("t5-base", legacy=False)
          >>> tokenizer.encode("Hello <extra_id_0>.")  # the extra space `[3]` is no longer here
          [8774, 32099, 5, 1]
          ```
          Checkout the [pull request](https://github.com/huggingface/transformers/pull/24565) for more details.
class_docstr_tok: ["Construct","a","Llama","tokenizer",".","Based","on","byte-level","Byte-Pair-Encoding",".","The","default","padding","token","is","unset","as","there","is","no","padding","token","in","the","original","model",".","Args",":","vocab_file","(","`","str","`",")",":","Path","to","the","vocabulary","file",".","unk_token","(","`","str","`","or","`","tokenizers.AddedToken","`",",","*","optional","*",",","defaults","to","`","``","<","unk",">","''","`",")",":","The","unknown","token",".","A","token","that","is","not","in","the","vocabulary","can","not","be","converted","to","an","ID","and","is","set","to","be","this","token","instead",".","bos_token","(","`","str","`","or","`","tokenizers.AddedToken","`",",","*","optional","*",",","defaults","to","`","``","<","s",">","''","`",")",":","The","beginning","of","sequence","token","that","was","used","during","pretraining",".","Can","be","used","a","sequence","classifier","token",".","eos_token","(","`","str","`","or","`","tokenizers.AddedToken","`",",","*","optional","*",",","defaults","to","`","``","<","\/s",">","''","`",")",":","The","end","of","sequence","token",".","pad_token","(","`","str","`","or","`","tokenizers.AddedToken","`",",","*","optional","*",")",":","A","special","token","used","to","make","arrays","of","tokens","the","same","size","for","batching","purpose",".","Will","then","be","ignored","by","attention","mechanisms","or","loss","computation",".","sp_model_kwargs","(","`","Dict","[","str",",","Any","]","`",",","`","Optional","`",",","*","optional","*",")",":","Will","be","passed","to","the","`","SentencePieceProcessor.__init__","(",")","`","method",".","The","[","Python","wrapper","for","SentencePiece","]","(","https",":","\/\/github.com\/google\/sentencepiece\/tree\/master\/python",")","can","be","used",",","among","other","things",",","to","set",":","-","`","enable_sampling","`",":","Enable","subword","regularization",".","-","`","nbest_size","`",":","Sampling","parameters","for","unigram",".","Invalid","for","BPE-Dropout",".","-","`","nbest_size","=","{","0,1","}","`",":","No","sampling","is","performed",".","-","`","nbest_size",">","1","`",":","samples","from","the","nbest_size","results",".","-","`","nbest_size","<","0","`",":","assuming","that","nbest_size","is","infinite","and","samples","from","the","all","hypothesis","(","lattice",")","using","forward-filtering-and-backward-sampling","algorithm",".","-","`","alpha","`",":","Smoothing","parameter","for","unigram","sampling",",","and","dropout","probability","of","merge","operations","for","BPE-dropout",".","add_bos_token","(","`","bool","`",",","*","optional","*",",","defaults","to","`","True","`",")",":","Whether","or","not","to","add","an","`","bos_token","`","at","the","start","of","sequences",".","add_eos_token","(","`","bool","`",",","*","optional","*",",","defaults","to","`","False","`",")",":","Whether","or","not","to","add","an","`","eos_token","`","at","the","end","of","sequences",".","clean_up_tokenization_spaces","(","`","bool","`",",","*","optional","*",",","defaults","to","`","False","`",")",":","Whether","or","not","to","cleanup","spaces","after","decoding",",","cleanup","consists","in","removing","potential","artifacts","like","extra","spaces",".","use_default_system_prompt","(","`","bool","`",",","*","optional","*",",","defaults","to","`","False","`",")",":","Whether","or","not","the","default","system","prompt","for","Llama","should","be","used",".","spaces_between_special_tokens","(","`","bool","`",",","*","optional","*",",","defaults","to","`","False","`",")",":","Whether","or","not","to","add","spaces","between","special","tokens",".","legacy","(","`","bool","`",",","*","optional","*",")",":","Whether","or","not","the","`","legacy","`","behavior","of","the","tokenizer","should","be","used",".","Legacy","is","before","the","merge","of","#","24622","and","#","25224","which","includes","fixes","to","properly","handle","tokens","that","appear","after","special","tokens",".","A","simple","example",":","-","`","legacy=True","`",":","``","`","python",">",">",">","from","transformers","import","T5Tokenizer",">",">",">","tokenizer","=","T5Tokenizer.from_pretrained","(","``","t5-base","''",",","legacy=True",")",">",">",">","tokenizer.encode","(","``","Hello","<","extra_id_0",">",".","''",")","[","8774",",","32099",",","3",",","5",",","1","]","``","`","-","`","legacy=False","`",":","``","`","python",">",">",">","from","transformers","import","T5Tokenizer",">",">",">","tokenizer","=","T5Tokenizer.from_pretrained","(","``","t5-base","''",",","legacy=False",")",">",">",">","tokenizer.encode","(","``","Hello","<","extra_id_0",">",".","''",")","#","the","extra","space","`","[","3","]","`","is","no","longer","here","[","8774",",","32099",",","5",",","1","]","``","`","Checkout","the","[","pull","request","]","(","https",":","\/\/github.com\/huggingface\/transformers\/pull\/24565",")","for","more","details","."]
docstr: Converts a token (str) in an id using the vocab.
docstr_tok: ["Converts","a","token","(","str",")","in","an","id","using","the","vocab","."]
code: def _convert_token_to_id(self, token):
      """Converts a token (str) in an id using the vocab."""
      return self.sp_model.piece_to_id(token)
  
imports: import os
  from shutil import copyfile
  from typing import TYPE_CHECKING, Any, Dict, List, Optional, Tuple
  import sentencepiece
  from ...convert_slow_tokenizer import import_protobuf
  from ...tokenization_utils import AddedToken, PreTrainedTokenizer
  from ...utils import logging
label_desc: Use image node_id 10 for calling the LlamaTokenizer obj's underlying member method code with example usage: obj._convert_token_to_id(token) and returns: self