class_docstr: Construct a "fast" LayoutLMv3 tokenizer (backed by HuggingFace's *tokenizers* library). Based on BPE.
  
  This tokenizer inherits from [`PreTrainedTokenizerFast`] which contains most of the main methods. Users should
  refer to this superclass for more information regarding those methods.
  
  Args:
      vocab_file (`str`):
          Path to the vocabulary file.
      merges_file (`str`):
          Path to the merges file.
      errors (`str`, *optional*, defaults to `"replace"`):
          Paradigm to follow when decoding bytes to UTF-8. See
          [bytes.decode](https://docs.python.org/3/library/stdtypes.html#bytes.decode) for more information.
      bos_token (`str`, *optional*, defaults to `"<s>"`):
          The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token.
  
          <Tip>
  
          When building a sequence using special tokens, this is not the token that is used for the beginning of
          sequence. The token used is the `cls_token`.
  
          </Tip>
  
      eos_token (`str`, *optional*, defaults to `"</s>"`):
          The end of sequence token.
  
          <Tip>
  
          When building a sequence using special tokens, this is not the token that is used for the end of sequence.
          The token used is the `sep_token`.
  
          </Tip>
  
      sep_token (`str`, *optional*, defaults to `"</s>"`):
          The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences for
          sequence classification or for a text and a question for question answering. It is also used as the last
          token of a sequence built with special tokens.
      cls_token (`str`, *optional*, defaults to `"<s>"`):
          The classifier token which is used when doing sequence classification (classification of the whole sequence
          instead of per-token classification). It is the first token of the sequence when built with special tokens.
      unk_token (`str`, *optional*, defaults to `"<unk>"`):
          The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
          token instead.
      pad_token (`str`, *optional*, defaults to `"<pad>"`):
          The token used for padding, for example when batching sequences of different lengths.
      mask_token (`str`, *optional*, defaults to `"<mask>"`):
          The token used for masking values. This is the token used when training this model with masked language
          modeling. This is the token which the model will try to predict.
      add_prefix_space (`bool`, *optional*, defaults to `False`):
          Whether or not to add an initial space to the input. This allows to treat the leading word just as any
          other word. (RoBERTa tokenizer detect beginning of words by the preceding space).
      trim_offsets (`bool`, *optional*, defaults to `True`):
          Whether the post processing step should trim offsets to avoid including whitespaces.
      cls_token_box (`List[int]`, *optional*, defaults to `[0, 0, 0, 0]`):
          The bounding box to use for the special [CLS] token.
      sep_token_box (`List[int]`, *optional*, defaults to `[0, 0, 0, 0]`):
          The bounding box to use for the special [SEP] token.
      pad_token_box (`List[int]`, *optional*, defaults to `[0, 0, 0, 0]`):
          The bounding box to use for the special [PAD] token.
      pad_token_label (`int`, *optional*, defaults to -100):
          The label to use for padding tokens. Defaults to -100, which is the `ignore_index` of PyTorch's
          CrossEntropyLoss.
      only_label_first_subword (`bool`, *optional*, defaults to `True`):
          Whether or not to only label the first subword, in case word labels are provided.
class_docstr_tok: ["Construct","a","``","fast","''","LayoutLMv3","tokenizer","(","backed","by","HuggingFace","'s","*","tokenizers","*","library",")",".","Based","on","BPE",".","This","tokenizer","inherits","from","[","`","PreTrainedTokenizerFast","`","]","which","contains","most","of","the","main","methods",".","Users","should","refer","to","this","superclass","for","more","information","regarding","those","methods",".","Args",":","vocab_file","(","`","str","`",")",":","Path","to","the","vocabulary","file",".","merges_file","(","`","str","`",")",":","Path","to","the","merges","file",".","errors","(","`","str","`",",","*","optional","*",",","defaults","to","`","``","replace","''","`",")",":","Paradigm","to","follow","when","decoding","bytes","to","UTF-8",".","See","[","bytes.decode","]","(","https",":","\/\/docs.python.org\/3\/library\/stdtypes.html","#","bytes.decode",")","for","more","information",".","bos_token","(","`","str","`",",","*","optional","*",",","defaults","to","`","``","<","s",">","''","`",")",":","The","beginning","of","sequence","token","that","was","used","during","pretraining",".","Can","be","used","a","sequence","classifier","token",".","<","Tip",">","When","building","a","sequence","using","special","tokens",",","this","is","not","the","token","that","is","used","for","the","beginning","of","sequence",".","The","token","used","is","the","`","cls_token","`",".","<","\/Tip",">","eos_token","(","`","str","`",",","*","optional","*",",","defaults","to","`","``","<","\/s",">","''","`",")",":","The","end","of","sequence","token",".","<","Tip",">","When","building","a","sequence","using","special","tokens",",","this","is","not","the","token","that","is","used","for","the","end","of","sequence",".","The","token","used","is","the","`","sep_token","`",".","<","\/Tip",">","sep_token","(","`","str","`",",","*","optional","*",",","defaults","to","`","``","<","\/s",">","''","`",")",":","The","separator","token",",","which","is","used","when","building","a","sequence","from","multiple","sequences",",","e.g",".","two","sequences","for","sequence","classification","or","for","a","text","and","a","question","for","question","answering",".","It","is","also","used","as","the","last","token","of","a","sequence","built","with","special","tokens",".","cls_token","(","`","str","`",",","*","optional","*",",","defaults","to","`","``","<","s",">","''","`",")",":","The","classifier","token","which","is","used","when","doing","sequence","classification","(","classification","of","the","whole","sequence","instead","of","per-token","classification",")",".","It","is","the","first","token","of","the","sequence","when","built","with","special","tokens",".","unk_token","(","`","str","`",",","*","optional","*",",","defaults","to","`","``","<","unk",">","''","`",")",":","The","unknown","token",".","A","token","that","is","not","in","the","vocabulary","can","not","be","converted","to","an","ID","and","is","set","to","be","this","token","instead",".","pad_token","(","`","str","`",",","*","optional","*",",","defaults","to","`","``","<","pad",">","''","`",")",":","The","token","used","for","padding",",","for","example","when","batching","sequences","of","different","lengths",".","mask_token","(","`","str","`",",","*","optional","*",",","defaults","to","`","``","<","mask",">","''","`",")",":","The","token","used","for","masking","values",".","This","is","the","token","used","when","training","this","model","with","masked","language","modeling",".","This","is","the","token","which","the","model","will","try","to","predict",".","add_prefix_space","(","`","bool","`",",","*","optional","*",",","defaults","to","`","False","`",")",":","Whether","or","not","to","add","an","initial","space","to","the","input",".","This","allows","to","treat","the","leading","word","just","as","any","other","word",".","(","RoBERTa","tokenizer","detect","beginning","of","words","by","the","preceding","space",")",".","trim_offsets","(","`","bool","`",",","*","optional","*",",","defaults","to","`","True","`",")",":","Whether","the","post","processing","step","should","trim","offsets","to","avoid","including","whitespaces",".","cls_token_box","(","`","List","[","int","]","`",",","*","optional","*",",","defaults","to","`","[","0",",","0",",","0",",","0","]","`",")",":","The","bounding","box","to","use","for","the","special","[","CLS","]","token",".","sep_token_box","(","`","List","[","int","]","`",",","*","optional","*",",","defaults","to","`","[","0",",","0",",","0",",","0","]","`",")",":","The","bounding","box","to","use","for","the","special","[","SEP","]","token",".","pad_token_box","(","`","List","[","int","]","`",",","*","optional","*",",","defaults","to","`","[","0",",","0",",","0",",","0","]","`",")",":","The","bounding","box","to","use","for","the","special","[","PAD","]","token",".","pad_token_label","(","`","int","`",",","*","optional","*",",","defaults","to","-100",")",":","The","label","to","use","for","padding","tokens",".","Defaults","to","-100",",","which","is","the","`","ignore_index","`","of","PyTorch's","CrossEntropyLoss",".","only_label_first_subword","(","`","bool","`",",","*","optional","*",",","defaults","to","`","True","`",")",":","Whether","or","not","to","only","label","the","first","subword",",","in","case","word","labels","are","provided","."]
docstr: Main method to tokenize and prepare for the model one or several sequence(s) or one or several pair(s) of
  sequences with word-level normalized bounding boxes and optional labels.
  
  Args:
      text (`str`, `List[str]`, `List[List[str]]`):
          The sequence or batch of sequences to be encoded. Each sequence can be a string, a list of strings
          (words of a single example or questions of a batch of examples) or a list of list of strings (batch of
          words).
      text_pair (`List[str]`, `List[List[str]]`):
          The sequence or batch of sequences to be encoded. Each sequence should be a list of strings
          (pretokenized string).
      boxes (`List[List[int]]`, `List[List[List[int]]]`):
          Word-level bounding boxes. Each bounding box should be normalized to be on a 0-1000 scale.
      word_labels (`List[int]`, `List[List[int]]`, *optional*):
          Word-level integer labels (for token classification tasks such as FUNSD, CORD).
docstr_tok: ["Main","method","to","tokenize","and","prepare","for","the","model","one","or","several","sequence","(","s",")","or","one","or","several","pair","(","s",")","of","sequences","with","word-level","normalized","bounding","boxes","and","optional","labels",".","Args",":","text","(","`","str","`",",","`","List","[","str","]","`",",","`","List","[","List","[","str","]","]","`",")",":","The","sequence","or","batch","of","sequences","to","be","encoded",".","Each","sequence","can","be","a","string",",","a","list","of","strings","(","words","of","a","single","example","or","questions","of","a","batch","of","examples",")","or","a","list","of","list","of","strings","(","batch","of","words",")",".","text_pair","(","`","List","[","str","]","`",",","`","List","[","List","[","str","]","]","`",")",":","The","sequence","or","batch","of","sequences","to","be","encoded",".","Each","sequence","should","be","a","list","of","strings","(","pretokenized","string",")",".","boxes","(","`","List","[","List","[","int","]","]","`",",","`","List","[","List","[","List","[","int","]","]","]","`",")",":","Word-level","bounding","boxes",".","Each","bounding","box","should","be","normalized","to","be","on","a","0-1000","scale",".","word_labels","(","`","List","[","int","]","`",",","`","List","[","List","[","int","]","]","`",",","*","optional","*",")",":","Word-level","integer","labels","(","for","token","classification","tasks","such","as","FUNSD",",","CORD",")","."]
code: def __call__(
      self,
      text: Union[
          TextInput,
          PreTokenizedInput,
          List[TextInput],
          List[PreTokenizedInput],
      ],
      text_pair: Optional[
          Union[PreTokenizedInput, List[PreTokenizedInput]]
      ] = None,
      boxes: Union[List[List[int]], List[List[List[int]]]] = None,
      word_labels: Optional[Union[List[int], List[List[int]]]] = None,
      add_special_tokens: bool = True,
      padding: Union[bool, str, PaddingStrategy] = False,
      truncation: Union[bool, str, TruncationStrategy] = None,
      max_length: Optional[int] = None,
      stride: int = 0,
      pad_to_multiple_of: Optional[int] = None,
      return_tensors: Optional[Union[str, TensorType]] = None,
      return_token_type_ids: Optional[bool] = None,
      return_attention_mask: Optional[bool] = None,
      return_overflowing_tokens: bool = False,
      return_special_tokens_mask: bool = False,
      return_offsets_mapping: bool = False,
      return_length: bool = False,
      verbose: bool = True,
      **kwargs,
  ) -> BatchEncoding:
      """
      Main method to tokenize and prepare for the model one or several sequence(s) or one or several pair(s) of
      sequences with word-level normalized bounding boxes and optional labels.
  
      Args:
          text (`str`, `List[str]`, `List[List[str]]`):
              The sequence or batch of sequences to be encoded. Each sequence can be a string, a list of strings
              (words of a single example or questions of a batch of examples) or a list of list of strings (batch of
              words).
          text_pair (`List[str]`, `List[List[str]]`):
              The sequence or batch of sequences to be encoded. Each sequence should be a list of strings
              (pretokenized string).
          boxes (`List[List[int]]`, `List[List[List[int]]]`):
              Word-level bounding boxes. Each bounding box should be normalized to be on a 0-1000 scale.
          word_labels (`List[int]`, `List[List[int]]`, *optional*):
              Word-level integer labels (for token classification tasks such as FUNSD, CORD).
      """
  
      # Input type checking for clearer error
      def _is_valid_text_input(t):
          if isinstance(t, str):
              # Strings are fine
              return True
          elif isinstance(t, (list, tuple)):
              # List are fine as long as they are...
              if len(t) == 0:
                  # ... empty
                  return True
              elif isinstance(t[0], str):
                  # ... list of strings
                  return True
              elif isinstance(t[0], (list, tuple)):
                  # ... list with an empty list or with a list of strings
                  return len(t[0]) == 0 or isinstance(t[0][0], str)
              else:
                  return False
          else:
              return False
  
      if text_pair is not None:
          # in case text + text_pair are provided, text = questions, text_pair = words
          if not _is_valid_text_input(text):
              raise ValueError(
                  "text input must of type `str` (single example) or `List[str]` (batch of examples). "
              )
          if not isinstance(text_pair, (list, tuple)):
              raise ValueError(
                  "Words must be of type `List[str]` (single pretokenized example), "
                  "or `List[List[str]]` (batch of pretokenized examples)."
              )
      else:
          # in case only text is provided => must be words
          if not isinstance(text, (list, tuple)):
              raise ValueError(
                  "Words must be of type `List[str]` (single pretokenized example), "
                  "or `List[List[str]]` (batch of pretokenized examples)."
              )
  
      if text_pair is not None:
          is_batched = isinstance(text, (list, tuple))
      else:
          is_batched = (
              isinstance(text, (list, tuple))
              and text
              and isinstance(text[0], (list, tuple))
          )
  
      words = text if text_pair is None else text_pair
      if boxes is None:
          raise ValueError(
              "You must provide corresponding bounding boxes"
          )
      if is_batched:
          if len(words) != len(boxes):
              raise ValueError(
                  "You must provide words and boxes for an equal amount of examples"
              )
          for words_example, boxes_example in zip(words, boxes):
              if len(words_example) != len(boxes_example):
                  raise ValueError(
                      "You must provide as many words as there are bounding boxes"
                  )
      else:
          if len(words) != len(boxes):
              raise ValueError(
                  "You must provide as many words as there are bounding boxes"
              )
  
      if is_batched:
          if text_pair is not None and len(text) != len(text_pair):
              raise ValueError(
                  f"batch length of `text`: {len(text)} does not match batch length of `text_pair`:"
                  f" {len(text_pair)}."
              )
          batch_text_or_text_pairs = (
              list(zip(text, text_pair))
              if text_pair is not None
              else text
          )
          is_pair = bool(text_pair is not None)
          return self.batch_encode_plus(
              batch_text_or_text_pairs=batch_text_or_text_pairs,
              is_pair=is_pair,
              boxes=boxes,
              word_labels=word_labels,
              add_special_tokens=add_special_tokens,
              padding=padding,
              truncation=truncation,
              max_length=max_length,
              stride=stride,
              pad_to_multiple_of=pad_to_multiple_of,
              return_tensors=return_tensors,
              return_token_type_ids=return_token_type_ids,
              return_attention_mask=return_attention_mask,
              return_overflowing_tokens=return_overflowing_tokens,
              return_special_tokens_mask=return_special_tokens_mask,
              return_offsets_mapping=return_offsets_mapping,
              return_length=return_length,
              verbose=verbose,
              **kwargs,
          )
      else:
          return self.encode_plus(
              text=text,
              text_pair=text_pair,
              boxes=boxes,
              word_labels=word_labels,
              add_special_tokens=add_special_tokens,
              padding=padding,
              truncation=truncation,
              max_length=max_length,
              stride=stride,
              pad_to_multiple_of=pad_to_multiple_of,
              return_tensors=return_tensors,
              return_token_type_ids=return_token_type_ids,
              return_attention_mask=return_attention_mask,
              return_overflowing_tokens=return_overflowing_tokens,
              return_special_tokens_mask=return_special_tokens_mask,
              return_offsets_mapping=return_offsets_mapping,
              return_length=return_length,
              verbose=verbose,
              **kwargs,
          )
  
file_path: H2O/h2o_flexgen/benchmark/third_party/transformers/src/transformers/models/layoutlmv3/tokenization_layoutlmv3_fast.py
imports: import json
  from typing import Dict, List, Optional, Tuple, Union
  from tokenizers import pre_tokenizers, processors
  from ...tokenization_utils_base import BatchEncoding, EncodedInput, PaddingStrategy, PreTokenizedInput, TensorType, TextInput, TextInputPair, TruncationStrategy
  from ...tokenization_utils_fast import PreTrainedTokenizerFast
  from ...utils import add_end_docstrings, logging
  from .tokenization_layoutlmv3 import LAYOUTLMV3_ENCODE_KWARGS_DOCSTRING, LAYOUTLMV3_ENCODE_PLUS_ADDITIONAL_KWARGS_DOCSTRING, LayoutLMv3Tokenizer
label_desc: Use image node_id 2 for calling the LayoutLMv3TokenizerFast obj's underlying member method code with example usage: obj.__call__(text, text_pair, boxes, word_labels, add_special_tokens, padding, truncation, max_length, stride, pad_to_multiple_of, return_tensors, return_token_type_ids, return_attention_mask, return_overflowing_tokens, return_special_tokens_mask, return_offsets_mapping, return_length, verbose) and returns: self, self, True, False, True, True, unknown, False