class_docstr: Constructs a Jukebox tokenizer. Jukebox can be conditioned on 3 different inputs :
      - Artists, unique ids are associated to each artist from the provided dictionary.
      - Genres, unique ids are associated to each genre from the provided dictionary.
      - Lyrics, character based tokenization. Must be initialized with the list of characters that are inside the
      vocabulary.
  
  This tokenizer does not require training. It should be able to process a different number of inputs:
  as the conditioning of the model can be done on the three different queries. If None is provided, defaults values will be used.:
  
  Depending on the number of genres on which the model should be conditioned (`n_genres`).
  ```python
  >>> from transformers import JukeboxTokenizer
  
  >>> tokenizer = JukeboxTokenizer.from_pretrained("openai/jukebox-1b-lyrics")
  >>> tokenizer("Alan Jackson", "Country Rock", "old town road")["input_ids"]
  [tensor([[   0,    0,    0, 6785,  546,   41,   38,   30,   76,   46,   41,   49,
             40,   76,   44,   41,   27,   30]]), tensor([[  0,   0,   0, 145,   0]]), tensor([[  0,   0,   0, 145,   0]])]
  ```
  
  You can get around that behavior by passing `add_prefix_space=True` when instantiating this tokenizer or when you
  call it on some text, but since the model was not pretrained this way, it might yield a decrease in performance.
  
  <Tip>
  
  If nothing is provided, the genres and the artist will either be selected randomly or set to None
  
  </Tip>
  
  This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to:
  this superclass for more information regarding those methods.
  
  However the code does not allow that and only supports composing from various genres.
  
  Args:
      artists_file (`str`):
          Path to the vocabulary file which contains a mapping between artists and ids. The default file supports
          both "v2" and "v3"
      genres_file (`str`):
          Path to the vocabulary file which contain a mapping between genres and ids.
      lyrics_file (`str`):
          Path to the vocabulary file which contains the accepted characters for the lyrics tokenization.
      version (`List[str]`, `optional`, default to `["v3", "v2", "v2"]`) :
          List of the tokenizer versions. The `5b-lyrics`'s top level prior model was trained using `v3` instead of
          `v2`.
      n_genres (`int`, `optional`, defaults to 1):
          Maximum number of genres to use for composition.
      max_n_lyric_tokens (`int`, `optional`, defaults to 512):
          Maximum number of lyric tokens to keep.
      unk_token (`str`, *optional*, defaults to `"<|endoftext|>"`):
          The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
          token instead.
class_docstr_tok: ["Constructs","a","Jukebox","tokenizer",".","Jukebox","can","be","conditioned","on","3","different","inputs",":","-","Artists",",","unique","ids","are","associated","to","each","artist","from","the","provided","dictionary",".","-","Genres",",","unique","ids","are","associated","to","each","genre","from","the","provided","dictionary",".","-","Lyrics",",","character","based","tokenization",".","Must","be","initialized","with","the","list","of","characters","that","are","inside","the","vocabulary",".","This","tokenizer","does","not","require","training",".","It","should","be","able","to","process","a","different","number","of","inputs",":","as","the","conditioning","of","the","model","can","be","done","on","the","three","different","queries",".","If","None","is","provided",",","defaults","values","will","be","used",".",":","Depending","on","the","number","of","genres","on","which","the","model","should","be","conditioned","(","`","n_genres","`",")",".","``","`","python",">",">",">","from","transformers","import","JukeboxTokenizer",">",">",">","tokenizer","=","JukeboxTokenizer.from_pretrained","(","``","openai\/jukebox-1b-lyrics","''",")",">",">",">","tokenizer","(","``","Alan","Jackson","''",",","``","Country","Rock","''",",","``","old","town","road","''",")","[","``","input_ids","''","]","[","tensor","(","[","[","0",",","0",",","0",",","6785",",","546",",","41",",","38",",","30",",","76",",","46",",","41",",","49",",","40",",","76",",","44",",","41",",","27",",","30","]","]",")",",","tensor","(","[","[","0",",","0",",","0",",","145",",","0","]","]",")",",","tensor","(","[","[","0",",","0",",","0",",","145",",","0","]","]",")","]","``","`","You","can","get","around","that","behavior","by","passing","`","add_prefix_space=True","`","when","instantiating","this","tokenizer","or","when","you","call","it","on","some","text",",","but","since","the","model","was","not","pretrained","this","way",",","it","might","yield","a","decrease","in","performance",".","<","Tip",">","If","nothing","is","provided",",","the","genres","and","the","artist","will","either","be","selected","randomly","or","set","to","None","<","\/Tip",">","This","tokenizer","inherits","from","[","`","PreTrainedTokenizer","`","]","which","contains","most","of","the","main","methods",".","Users","should","refer","to",":","this","superclass","for","more","information","regarding","those","methods",".","However","the","code","does","not","allow","that","and","only","supports","composing","from","various","genres",".","Args",":","artists_file","(","`","str","`",")",":","Path","to","the","vocabulary","file","which","contains","a","mapping","between","artists","and","ids",".","The","default","file","supports","both","``","v2","''","and","``","v3","''","genres_file","(","`","str","`",")",":","Path","to","the","vocabulary","file","which","contain","a","mapping","between","genres","and","ids",".","lyrics_file","(","`","str","`",")",":","Path","to","the","vocabulary","file","which","contains","the","accepted","characters","for","the","lyrics","tokenization",".","version","(","`","List","[","str","]","`",",","`","optional","`",",","default","to","`","[","``","v3","''",",","``","v2","''",",","``","v2","''","]","`",")",":","List","of","the","tokenizer","versions",".","The","`","5b-lyrics","`","'s","top","level","prior","model","was","trained","using","`","v3","`","instead","of","`","v2","`",".","n_genres","(","`","int","`",",","`","optional","`",",","defaults","to","1",")",":","Maximum","number","of","genres","to","use","for","composition",".","max_n_lyric_tokens","(","`","int","`",",","`","optional","`",",","defaults","to","512",")",":","Maximum","number","of","lyric","tokens","to","keep",".","unk_token","(","`","str","`",",","*","optional","*",",","defaults","to","`","``","<","|endoftext|",">","''","`",")",":","The","unknown","token",".","A","token","that","is","not","in","the","vocabulary","can","not","be","converted","to","an","ID","and","is","set","to","be","this","token","instead","."]
docstr: Performs any necessary transformations before tokenization.
  
  Args:
      artist (`str`):
          The artist name to prepare. This will mostly lower the string
      genres (`str`):
          The genre name to prepare. This will mostly lower the string.
      lyrics (`str`):
          The lyrics to prepare.
      is_split_into_words (`bool`, *optional*, defaults to `False`):
          Whether or not the input is already pre-tokenized (e.g., split into words). If set to `True`, the
          tokenizer assumes the input is already split into words (for instance, by splitting it on whitespace)
          which it will tokenize. This is useful for NER or token classification.
docstr_tok: ["Performs","any","necessary","transformations","before","tokenization",".","Args",":","artist","(","`","str","`",")",":","The","artist","name","to","prepare",".","This","will","mostly","lower","the","string","genres","(","`","str","`",")",":","The","genre","name","to","prepare",".","This","will","mostly","lower","the","string",".","lyrics","(","`","str","`",")",":","The","lyrics","to","prepare",".","is_split_into_words","(","`","bool","`",",","*","optional","*",",","defaults","to","`","False","`",")",":","Whether","or","not","the","input","is","already","pre-tokenized","(","e.g.",",","split","into","words",")",".","If","set","to","`","True","`",",","the","tokenizer","assumes","the","input","is","already","split","into","words","(","for","instance",",","by","splitting","it","on","whitespace",")","which","it","will","tokenize",".","This","is","useful","for","NER","or","token","classification","."]
code: def prepare_for_tokenization(
      self,
      artists: str,
      genres: str,
      lyrics: str,
      is_split_into_words: bool = False,
  ) -> Tuple[str, str, str, Dict[str, Any]]:
      """
      Performs any necessary transformations before tokenization.
  
      Args:
          artist (`str`):
              The artist name to prepare. This will mostly lower the string
          genres (`str`):
              The genre name to prepare. This will mostly lower the string.
          lyrics (`str`):
              The lyrics to prepare.
          is_split_into_words (`bool`, *optional*, defaults to `False`):
              Whether or not the input is already pre-tokenized (e.g., split into words). If set to `True`, the
              tokenizer assumes the input is already split into words (for instance, by splitting it on whitespace)
              which it will tokenize. This is useful for NER or token classification.
      """
      for idx in range(len(self.version)):
          if self.version[idx] == "v3":
              artists[idx] = artists[idx].lower()
              genres[idx] = [genres[idx].lower()]
          else:
              artists[idx] = self._normalize(artists[idx]) + ".v2"
              genres[idx] = [
                  self._normalize(genre) + ".v2"
                  for genre in genres[idx].split("_")
              ]  # split is for the full dictionary with combined genres
  
      if self.version[0] == "v2":
          self.out_of_vocab = regex.compile(
              r"[^A-Za-z0-9.,:;!?\-'\"()\[\] \t\n]+"
          )
          vocab = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789.,:;!?-+'\"()[] \t\n"
          self.vocab = {
              vocab[index]: index + 1 for index in range(len(vocab))
          }
          self.vocab["<unk>"] = 0
          self.n_vocab = len(vocab) + 1
          self.lyrics_encoder = self.vocab
          self.lyrics_decoder = {v: k for k, v in self.vocab.items()}
          self.lyrics_decoder[0] = ""
      else:
          self.out_of_vocab = regex.compile(
              r"[^A-Za-z0-9.,:;!?\-+'\"()\[\] \t\n]+"
          )
  
      lyrics = self._run_strip_accents(lyrics)
      lyrics = lyrics.replace("\\", "\n")
      lyrics = self.out_of_vocab.sub("", lyrics), [], []
      return artists, genres, lyrics
  
imports: import json
  import os
  import re
  import unicodedata
  from json.encoder import INFINITY
  from typing import Any, Dict, List, Optional, Tuple, Union
  import numpy
  import regex
  from ...tokenization_utils import AddedToken, PreTrainedTokenizer
  from ...tokenization_utils_base import BatchEncoding
  from ...utils import TensorType, is_flax_available, is_tf_available, is_torch_available, logging
  from ...utils.generic import _is_jax, _is_numpy
label_desc: Use image node_id 7 for calling the JukeboxTokenizer obj's underlying member method code with example usage: obj.prepare_for_tokenization(artists, genres, lyrics, is_split_into_words) and returns: artists,  genres,  lyrics