class_docstr: Construct a LUKE tokenizer.
  
  This tokenizer inherits from [`RobertaTokenizer`] which contains most of the main methods. Users should refer to
  this superclass for more information regarding those methods. Compared to [`RobertaTokenizer`], [`LukeTokenizer`]
  also creates entity sequences, namely `entity_ids`, `entity_attention_mask`, `entity_token_type_ids`, and
  `entity_position_ids` to be used by the LUKE model.
  
  Args:
      vocab_file (`str`):
          Path to the vocabulary file.
      merges_file (`str`):
          Path to the merges file.
      entity_vocab_file (`str`):
          Path to the entity vocabulary file.
      task (`str`, *optional*):
          Task for which you want to prepare sequences. One of `"entity_classification"`,
          `"entity_pair_classification"`, or `"entity_span_classification"`. If you specify this argument, the entity
          sequence is automatically created based on the given entity span(s).
      max_entity_length (`int`, *optional*, defaults to 32):
          The maximum length of `entity_ids`.
      max_mention_length (`int`, *optional*, defaults to 30):
          The maximum number of tokens inside an entity span.
      entity_token_1 (`str`, *optional*, defaults to `<ent>`):
          The special token used to represent an entity span in a word token sequence. This token is only used when
          `task` is set to `"entity_classification"` or `"entity_pair_classification"`.
      entity_token_2 (`str`, *optional*, defaults to `<ent2>`):
          The special token used to represent an entity span in a word token sequence. This token is only used when
          `task` is set to `"entity_pair_classification"`.
class_docstr_tok: ["Construct","a","LUKE","tokenizer",".","This","tokenizer","inherits","from","[","`","RobertaTokenizer","`","]","which","contains","most","of","the","main","methods",".","Users","should","refer","to","this","superclass","for","more","information","regarding","those","methods",".","Compared","to","[","`","RobertaTokenizer","`","]",",","[","`","LukeTokenizer","`","]","also","creates","entity","sequences",",","namely","`","entity_ids","`",",","`","entity_attention_mask","`",",","`","entity_token_type_ids","`",",","and","`","entity_position_ids","`","to","be","used","by","the","LUKE","model",".","Args",":","vocab_file","(","`","str","`",")",":","Path","to","the","vocabulary","file",".","merges_file","(","`","str","`",")",":","Path","to","the","merges","file",".","entity_vocab_file","(","`","str","`",")",":","Path","to","the","entity","vocabulary","file",".","task","(","`","str","`",",","*","optional","*",")",":","Task","for","which","you","want","to","prepare","sequences",".","One","of","`","``","entity_classification","''","`",",","`","``","entity_pair_classification","''","`",",","or","`","``","entity_span_classification","''","`",".","If","you","specify","this","argument",",","the","entity","sequence","is","automatically","created","based","on","the","given","entity","span","(","s",")",".","max_entity_length","(","`","int","`",",","*","optional","*",",","defaults","to","32",")",":","The","maximum","length","of","`","entity_ids","`",".","max_mention_length","(","`","int","`",",","*","optional","*",",","defaults","to","30",")",":","The","maximum","number","of","tokens","inside","an","entity","span",".","entity_token_1","(","`","str","`",",","*","optional","*",",","defaults","to","`","<","ent",">","`",")",":","The","special","token","used","to","represent","an","entity","span","in","a","word","token","sequence",".","This","token","is","only","used","when","`","task","`","is","set","to","`","``","entity_classification","''","`","or","`","``","entity_pair_classification","''","`",".","entity_token_2","(","`","str","`",",","*","optional","*",",","defaults","to","`","<","ent2",">","`",")",":","The","special","token","used","to","represent","an","entity","span","in","a","word","token","sequence",".","This","token","is","only","used","when","`","task","`","is","set","to","`","``","entity_pair_classification","''","`","."]
docstr: Pad a single encoded input or a batch of encoded inputs up to predefined length or to the max sequence length
  in the batch. Padding side (left/right) padding token ids are defined at the tokenizer level (with
  `self.padding_side`, `self.pad_token_id` and `self.pad_token_type_id`) .. note:: If the `encoded_inputs` passed
  are dictionary of numpy arrays, PyTorch tensors or TensorFlow tensors, the result will use the same type unless
  you provide a different tensor type with `return_tensors`. In the case of PyTorch tensors, you will lose the
  specific device of your tensors however.
  
  Args:
      encoded_inputs ([`BatchEncoding`], list of [`BatchEncoding`], `Dict[str, List[int]]`, `Dict[str, List[List[int]]` or `List[Dict[str, List[int]]]`):
          Tokenized inputs. Can represent one input ([`BatchEncoding`] or `Dict[str, List[int]]`) or a batch of
          tokenized inputs (list of [`BatchEncoding`], *Dict[str, List[List[int]]]* or *List[Dict[str,
          List[int]]]*) so you can use this method during preprocessing as well as in a PyTorch Dataloader
          collate function. Instead of `List[int]` you can have tensors (numpy arrays, PyTorch tensors or
          TensorFlow tensors), see the note above for the return type.
      padding (`bool`, `str` or [`~utils.PaddingStrategy`], *optional*, defaults to `True`):
           Select a strategy to pad the returned sequences (according to the model's padding side and padding
           index) among:
  
          - `True` or `'longest'`: Pad to the longest sequence in the batch (or no padding if only a single
            sequence if provided).
          - `'max_length'`: Pad to a maximum length specified with the argument `max_length` or to the maximum
            acceptable input length for the model if that argument is not provided.
          - `False` or `'do_not_pad'` (default): No padding (i.e., can output a batch with sequences of different
            lengths).
      max_length (`int`, *optional*):
          Maximum length of the returned list and optionally padding length (see above).
      max_entity_length (`int`, *optional*):
          The maximum length of the entity sequence.
      pad_to_multiple_of (`int`, *optional*):
          If set will pad the sequence to a multiple of the provided value. This is especially useful to enable
          the use of Tensor Cores on NVIDIA hardware with compute capability >= 7.5 (Volta).
      return_attention_mask (`bool`, *optional*):
          Whether to return the attention mask. If left to the default, will return the attention mask according
          to the specific tokenizer's default, defined by the `return_outputs` attribute. [What are attention
          masks?](../glossary#attention-mask)
      return_tensors (`str` or [`~utils.TensorType`], *optional*):
          If set, will return tensors instead of list of python integers. Acceptable values are:
  
          - `'tf'`: Return TensorFlow `tf.constant` objects.
          - `'pt'`: Return PyTorch `torch.Tensor` objects.
          - `'np'`: Return Numpy `np.ndarray` objects.
      verbose (`bool`, *optional*, defaults to `True`):
          Whether or not to print more information and warnings.
docstr_tok: ["Pad","a","single","encoded","input","or","a","batch","of","encoded","inputs","up","to","predefined","length","or","to","the","max","sequence","length","in","the","batch",".","Padding","side","(","left\/right",")","padding","token","ids","are","defined","at","the","tokenizer","level","(","with","`","self.padding_side","`",",","`","self.pad_token_id","`","and","`","self.pad_token_type_id","`",")","..","note",":",":","If","the","`","encoded_inputs","`","passed","are","dictionary","of","numpy","arrays",",","PyTorch","tensors","or","TensorFlow","tensors",",","the","result","will","use","the","same","type","unless","you","provide","a","different","tensor","type","with","`","return_tensors","`",".","In","the","case","of","PyTorch","tensors",",","you","will","lose","the","specific","device","of","your","tensors","however",".","Args",":","encoded_inputs","(","[","`","BatchEncoding","`","]",",","list","of","[","`","BatchEncoding","`","]",",","`","Dict","[","str",",","List","[","int","]","]","`",",","`","Dict","[","str",",","List","[","List","[","int","]","]","`","or","`","List","[","Dict","[","str",",","List","[","int","]","]","]","`",")",":","Tokenized","inputs",".","Can","represent","one","input","(","[","`","BatchEncoding","`","]","or","`","Dict","[","str",",","List","[","int","]","]","`",")","or","a","batch","of","tokenized","inputs","(","list","of","[","`","BatchEncoding","`","]",",","*","Dict","[","str",",","List","[","List","[","int","]","]","]","*","or","*","List","[","Dict","[","str",",","List","[","int","]","]","]","*",")","so","you","can","use","this","method","during","preprocessing","as","well","as","in","a","PyTorch","Dataloader","collate","function",".","Instead","of","`","List","[","int","]","`","you","can","have","tensors","(","numpy","arrays",",","PyTorch","tensors","or","TensorFlow","tensors",")",",","see","the","note","above","for","the","return","type",".","padding","(","`","bool","`",",","`","str","`","or","[","`","~utils.PaddingStrategy","`","]",",","*","optional","*",",","defaults","to","`","True","`",")",":","Select","a","strategy","to","pad","the","returned","sequences","(","according","to","the","model","'s","padding","side","and","padding","index",")","among",":","-","`","True","`","or","`","'longest","'","`",":","Pad","to","the","longest","sequence","in","the","batch","(","or","no","padding","if","only","a","single","sequence","if","provided",")",".","-","`","'max_length","'","`",":","Pad","to","a","maximum","length","specified","with","the","argument","`","max_length","`","or","to","the","maximum","acceptable","input","length","for","the","model","if","that","argument","is","not","provided",".","-","`","False","`","or","`","'do_not_pad","'","`","(","default",")",":","No","padding","(","i.e.",",","can","output","a","batch","with","sequences","of","different","lengths",")",".","max_length","(","`","int","`",",","*","optional","*",")",":","Maximum","length","of","the","returned","list","and","optionally","padding","length","(","see","above",")",".","max_entity_length","(","`","int","`",",","*","optional","*",")",":","The","maximum","length","of","the","entity","sequence",".","pad_to_multiple_of","(","`","int","`",",","*","optional","*",")",":","If","set","will","pad","the","sequence","to","a","multiple","of","the","provided","value",".","This","is","especially","useful","to","enable","the","use","of","Tensor","Cores","on","NVIDIA","hardware","with","compute","capability",">","=","7.5","(","Volta",")",".","return_attention_mask","(","`","bool","`",",","*","optional","*",")",":","Whether","to","return","the","attention","mask",".","If","left","to","the","default",",","will","return","the","attention","mask","according","to","the","specific","tokenizer","'s","default",",","defined","by","the","`","return_outputs","`","attribute",".","[","What","are","attention","masks","?","]","(","..","\/glossary","#","attention-mask",")","return_tensors","(","`","str","`","or","[","`","~utils.TensorType","`","]",",","*","optional","*",")",":","If","set",",","will","return","tensors","instead","of","list","of","python","integers",".","Acceptable","values","are",":","-","`","'tf","'","`",":","Return","TensorFlow","`","tf.constant","`","objects",".","-","`","'pt","'","`",":","Return","PyTorch","`","torch.Tensor","`","objects",".","-","`","'np","'","`",":","Return","Numpy","`","np.ndarray","`","objects",".","verbose","(","`","bool","`",",","*","optional","*",",","defaults","to","`","True","`",")",":","Whether","or","not","to","print","more","information","and","warnings","."]
code: def pad(
      self,
      encoded_inputs: Union[
          BatchEncoding,
          List[BatchEncoding],
          Dict[str, EncodedInput],
          Dict[str, List[EncodedInput]],
          List[Dict[str, EncodedInput]],
      ],
      padding: Union[bool, str, PaddingStrategy] = True,
      max_length: Optional[int] = None,
      max_entity_length: Optional[int] = None,
      pad_to_multiple_of: Optional[int] = None,
      return_attention_mask: Optional[bool] = None,
      return_tensors: Optional[Union[str, TensorType]] = None,
      verbose: bool = True,
  ) -> BatchEncoding:
      """
      Pad a single encoded input or a batch of encoded inputs up to predefined length or to the max sequence length
      in the batch. Padding side (left/right) padding token ids are defined at the tokenizer level (with
      `self.padding_side`, `self.pad_token_id` and `self.pad_token_type_id`) .. note:: If the `encoded_inputs` passed
      are dictionary of numpy arrays, PyTorch tensors or TensorFlow tensors, the result will use the same type unless
      you provide a different tensor type with `return_tensors`. In the case of PyTorch tensors, you will lose the
      specific device of your tensors however.
  
      Args:
          encoded_inputs ([`BatchEncoding`], list of [`BatchEncoding`], `Dict[str, List[int]]`, `Dict[str, List[List[int]]` or `List[Dict[str, List[int]]]`):
              Tokenized inputs. Can represent one input ([`BatchEncoding`] or `Dict[str, List[int]]`) or a batch of
              tokenized inputs (list of [`BatchEncoding`], *Dict[str, List[List[int]]]* or *List[Dict[str,
              List[int]]]*) so you can use this method during preprocessing as well as in a PyTorch Dataloader
              collate function. Instead of `List[int]` you can have tensors (numpy arrays, PyTorch tensors or
              TensorFlow tensors), see the note above for the return type.
          padding (`bool`, `str` or [`~utils.PaddingStrategy`], *optional*, defaults to `True`):
               Select a strategy to pad the returned sequences (according to the model's padding side and padding
               index) among:
  
              - `True` or `'longest'`: Pad to the longest sequence in the batch (or no padding if only a single
                sequence if provided).
              - `'max_length'`: Pad to a maximum length specified with the argument `max_length` or to the maximum
                acceptable input length for the model if that argument is not provided.
              - `False` or `'do_not_pad'` (default): No padding (i.e., can output a batch with sequences of different
                lengths).
          max_length (`int`, *optional*):
              Maximum length of the returned list and optionally padding length (see above).
          max_entity_length (`int`, *optional*):
              The maximum length of the entity sequence.
          pad_to_multiple_of (`int`, *optional*):
              If set will pad the sequence to a multiple of the provided value. This is especially useful to enable
              the use of Tensor Cores on NVIDIA hardware with compute capability >= 7.5 (Volta).
          return_attention_mask (`bool`, *optional*):
              Whether to return the attention mask. If left to the default, will return the attention mask according
              to the specific tokenizer's default, defined by the `return_outputs` attribute. [What are attention
              masks?](../glossary#attention-mask)
          return_tensors (`str` or [`~utils.TensorType`], *optional*):
              If set, will return tensors instead of list of python integers. Acceptable values are:
  
              - `'tf'`: Return TensorFlow `tf.constant` objects.
              - `'pt'`: Return PyTorch `torch.Tensor` objects.
              - `'np'`: Return Numpy `np.ndarray` objects.
          verbose (`bool`, *optional*, defaults to `True`):
              Whether or not to print more information and warnings.
      """
      # If we have a list of dicts, let's convert it in a dict of lists
      # We do this to allow using this method as a collate_fn function in PyTorch Dataloader
      if isinstance(encoded_inputs, (list, tuple)) and isinstance(
          encoded_inputs[0], Mapping
      ):
          encoded_inputs = {
              key: [example[key] for example in encoded_inputs]
              for key in encoded_inputs[0].keys()
          }
  
      # The model's main input name, usually `input_ids`, has be passed for padding
      if self.model_input_names[0] not in encoded_inputs:
          raise ValueError(
              "You should supply an encoding or a list of encodings to this method "
              f"that includes {self.model_input_names[0]}, but you provided {list(encoded_inputs.keys())}"
          )
  
      required_input = encoded_inputs[self.model_input_names[0]]
  
      if not required_input:
          if return_attention_mask:
              encoded_inputs["attention_mask"] = []
          return encoded_inputs
  
      # If we have PyTorch/TF/NumPy tensors/arrays as inputs, we cast them as python objects
      # and rebuild them afterwards if no return_tensors is specified
      # Note that we lose the specific device the tensor may be on for PyTorch
  
      first_element = required_input[0]
      if isinstance(first_element, (list, tuple)):
          # first_element might be an empty list/tuple in some edge cases so we grab the first non empty element.
          index = 0
          while len(required_input[index]) == 0:
              index += 1
          if index < len(required_input):
              first_element = required_input[index][0]
      # At this state, if `first_element` is still a list/tuple, it's an empty one so there is nothing to do.
      if not isinstance(first_element, (int, list, tuple)):
          if is_tf_tensor(first_element):
              return_tensors = (
                  "tf" if return_tensors is None else return_tensors
              )
          elif is_torch_tensor(first_element):
              return_tensors = (
                  "pt" if return_tensors is None else return_tensors
              )
          elif isinstance(first_element, np.ndarray):
              return_tensors = (
                  "np" if return_tensors is None else return_tensors
              )
          else:
              raise ValueError(
                  f"type of {first_element} unknown: {type(first_element)}. "
                  "Should be one of a python, numpy, pytorch or tensorflow object."
              )
  
          for key, value in encoded_inputs.items():
              encoded_inputs[key] = to_py_obj(value)
  
      # Convert padding_strategy in PaddingStrategy
      (
          padding_strategy,
          _,
          max_length,
          _,
      ) = self._get_padding_truncation_strategies(
          padding=padding, max_length=max_length, verbose=verbose
      )
  
      if max_entity_length is None:
          max_entity_length = self.max_entity_length
  
      required_input = encoded_inputs[self.model_input_names[0]]
      if required_input and not isinstance(
          required_input[0], (list, tuple)
      ):
          encoded_inputs = self._pad(
              encoded_inputs,
              max_length=max_length,
              max_entity_length=max_entity_length,
              padding_strategy=padding_strategy,
              pad_to_multiple_of=pad_to_multiple_of,
              return_attention_mask=return_attention_mask,
          )
          return BatchEncoding(
              encoded_inputs, tensor_type=return_tensors
          )
  
      batch_size = len(required_input)
      if any(len(v) != batch_size for v in encoded_inputs.values()):
          raise ValueError(
              "Some items in the output dictionary have a different batch size than others."
          )
  
      if padding_strategy == PaddingStrategy.LONGEST:
          max_length = max(len(inputs) for inputs in required_input)
          max_entity_length = (
              max(
                  len(inputs) for inputs in encoded_inputs["entity_ids"]
              )
              if "entity_ids" in encoded_inputs
              else 0
          )
          padding_strategy = PaddingStrategy.MAX_LENGTH
  
      batch_outputs = {}
      for i in range(batch_size):
          inputs = dict((k, v[i]) for k, v in encoded_inputs.items())
          outputs = self._pad(
              inputs,
              max_length=max_length,
              max_entity_length=max_entity_length,
              padding_strategy=padding_strategy,
              pad_to_multiple_of=pad_to_multiple_of,
              return_attention_mask=return_attention_mask,
          )
  
          for key, value in outputs.items():
              if key not in batch_outputs:
                  batch_outputs[key] = []
              batch_outputs[key].append(value)
  
      return BatchEncoding(batch_outputs, tensor_type=return_tensors)
  
imports: import itertools
  import json
  import os
  from collections.abc import Mapping
  from typing import Dict, List, Optional, Tuple, Union
  import numpy
  from ...None import RobertaTokenizer
  from ...tokenization_utils_base import ENCODE_KWARGS_DOCSTRING, AddedToken, BatchEncoding, EncodedInput, PaddingStrategy, TensorType, TextInput, TextInputPair, TruncationStrategy, to_py_obj
  from ...utils import add_end_docstrings, is_tf_tensor, is_torch_tensor, logging
label_desc: Use image node_id 9 for calling the LukeTokenizer obj's underlying member method code with example usage: obj.pad(encoded_inputs, padding, max_length, max_entity_length, pad_to_multiple_of, return_attention_mask, return_tensors, verbose) and returns: BatchEncoding, encoded_inputs, BatchEncoding