class_docstr: Construct a MarkupLM tokenizer. Based on byte-level Byte-Pair-Encoding (BPE). [`MarkupLMTokenizer`] can be used to
  turn HTML strings into to token-level `input_ids`, `attention_mask`, `token_type_ids`, `xpath_tags_seq` and
  `xpath_tags_seq`. This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods.
  Users should refer to this superclass for more information regarding those methods.
  
  Args:
      vocab_file (`str`):
          Path to the vocabulary file.
      merges_file (`str`):
          Path to the merges file.
      errors (`str`, *optional*, defaults to `"replace"`):
          Paradigm to follow when decoding bytes to UTF-8. See
          [bytes.decode](https://docs.python.org/3/library/stdtypes.html#bytes.decode) for more information.
      bos_token (`str`, *optional*, defaults to `"<s>"`):
          The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token.
  
          <Tip>
  
          When building a sequence using special tokens, this is not the token that is used for the beginning of
          sequence. The token used is the `cls_token`.
  
          </Tip>
  
      eos_token (`str`, *optional*, defaults to `"</s>"`):
          The end of sequence token.
  
          <Tip>
  
          When building a sequence using special tokens, this is not the token that is used for the end of sequence.
          The token used is the `sep_token`.
  
          </Tip>
  
      sep_token (`str`, *optional*, defaults to `"</s>"`):
          The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences for
          sequence classification or for a text and a question for question answering. It is also used as the last
          token of a sequence built with special tokens.
      cls_token (`str`, *optional*, defaults to `"<s>"`):
          The classifier token which is used when doing sequence classification (classification of the whole sequence
          instead of per-token classification). It is the first token of the sequence when built with special tokens.
      unk_token (`str`, *optional*, defaults to `"<unk>"`):
          The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
          token instead.
      pad_token (`str`, *optional*, defaults to `"<pad>"`):
          The token used for padding, for example when batching sequences of different lengths.
      mask_token (`str`, *optional*, defaults to `"<mask>"`):
          The token used for masking values. This is the token used when training this model with masked language
          modeling. This is the token which the model will try to predict.
      add_prefix_space (`bool`, *optional*, defaults to `False`):
          Whether or not to add an initial space to the input. This allows to treat the leading word just as any
          other word. (RoBERTa tokenizer detect beginning of words by the preceding space).
docstr: Prepares a sequence or a pair of sequences so that it can be used by the model. It adds special tokens,
  truncates sequences if overflowing while taking into account the special tokens and manages a moving window
  (with user defined stride) for overflowing tokens. Please Note, for *text_pair* different than `None` and
  *truncation_strategy = longest_first* or `True`, it is not possible to return overflowing tokens. Such a
  combination of arguments will raise an error.
  
  Node-level `xpaths` are turned into token-level `xpath_tags_seq` and `xpath_subs_seq`. If provided, node-level
  `node_labels` are turned into token-level `labels`. The node label is used for the first token of the node,
  while remaining tokens are labeled with -100, such that they will be ignored by the loss function.
  
  Args:
      text (`str`, `List[str]`, `List[List[str]]`):
          The first sequence to be encoded. This can be a string, a list of strings or a list of list of strings.
      text_pair (`List[str]` or `List[int]`, *optional*):
          Optional second sequence to be encoded. This can be a list of strings (nodes of a single example) or a
          list of list of strings (nodes of a batch of examples).
code: def prepare_for_model(
      self,
      text: Union[TextInput, PreTokenizedInput],
      text_pair: Optional[PreTokenizedInput] = None,
      xpaths: Optional[List[List[int]]] = None,
      node_labels: Optional[List[int]] = None,
      add_special_tokens: bool = True,
      padding: Union[bool, str, PaddingStrategy] = False,
      truncation: Union[bool, str, TruncationStrategy] = None,
      max_length: Optional[int] = None,
      stride: int = 0,
      pad_to_multiple_of: Optional[int] = None,
      return_tensors: Optional[Union[str, TensorType]] = None,
      return_token_type_ids: Optional[bool] = None,
      return_attention_mask: Optional[bool] = None,
      return_overflowing_tokens: bool = False,
      return_special_tokens_mask: bool = False,
      return_offsets_mapping: bool = False,
      return_length: bool = False,
      verbose: bool = True,
      prepend_batch_axis: bool = False,
      **kwargs,
  ) -> BatchEncoding:
      """
      Prepares a sequence or a pair of sequences so that it can be used by the model. It adds special tokens,
      truncates sequences if overflowing while taking into account the special tokens and manages a moving window
      (with user defined stride) for overflowing tokens. Please Note, for *text_pair* different than `None` and
      *truncation_strategy = longest_first* or `True`, it is not possible to return overflowing tokens. Such a
      combination of arguments will raise an error.
  
      Node-level `xpaths` are turned into token-level `xpath_tags_seq` and `xpath_subs_seq`. If provided, node-level
      `node_labels` are turned into token-level `labels`. The node label is used for the first token of the node,
      while remaining tokens are labeled with -100, such that they will be ignored by the loss function.
  
      Args:
          text (`str`, `List[str]`, `List[List[str]]`):
              The first sequence to be encoded. This can be a string, a list of strings or a list of list of strings.
          text_pair (`List[str]` or `List[int]`, *optional*):
              Optional second sequence to be encoded. This can be a list of strings (nodes of a single example) or a
              list of list of strings (nodes of a batch of examples).
      """
  
      # Backward compatibility for 'truncation_strategy', 'pad_to_max_length'
      (
          padding_strategy,
          truncation_strategy,
          max_length,
          kwargs,
      ) = self._get_padding_truncation_strategies(
          padding=padding,
          truncation=truncation,
          max_length=max_length,
          pad_to_multiple_of=pad_to_multiple_of,
          verbose=verbose,
          **kwargs,
      )
  
      tokens = []
      pair_tokens = []
      xpath_tags_seq = []
      xpath_subs_seq = []
      pair_xpath_tags_seq = []
      pair_xpath_subs_seq = []
      labels = []
  
      if text_pair is None:
          if node_labels is None:
              # CASE 1: web page classification (training + inference) + CASE 2: token classification (inference)
              for word, xpath in zip(text, xpaths):
                  if len(word) < 1:  # skip empty nodes
                      continue
                  word_tokens = self.tokenize(word)
                  tokens.extend(word_tokens)
                  xpath_tags_list, xpath_subs_list = self.get_xpath_seq(
                      xpath
                  )
                  xpath_tags_seq.extend(
                      [xpath_tags_list] * len(word_tokens)
                  )
                  xpath_subs_seq.extend(
                      [xpath_subs_list] * len(word_tokens)
                  )
          else:
              # CASE 2: token classification (training)
              for word, xpath, label in zip(text, xpaths, node_labels):
                  if len(word) < 1:  # skip empty nodes
                      continue
                  word_tokens = self.tokenize(word)
                  tokens.extend(word_tokens)
                  xpath_tags_list, xpath_subs_list = self.get_xpath_seq(
                      xpath
                  )
                  xpath_tags_seq.extend(
                      [xpath_tags_list] * len(word_tokens)
                  )
                  xpath_subs_seq.extend(
                      [xpath_subs_list] * len(word_tokens)
                  )
                  if self.only_label_first_subword:
                      # Use the real label id for the first token of the word, and padding ids for the remaining tokens
                      labels.extend(
                          [label]
                          + [self.pad_token_label]
                          * (len(word_tokens) - 1)
                      )
                  else:
                      labels.extend([label] * len(word_tokens))
      else:
          # CASE 3: web page question answering (inference)
          # text = question
          # text_pair = nodes
          tokens = self.tokenize(text)
          xpath_tags_seq = [
              self.pad_xpath_tags_seq for _ in range(len(tokens))
          ]
          xpath_subs_seq = [
              self.pad_xpath_subs_seq for _ in range(len(tokens))
          ]
  
          for word, xpath in zip(text_pair, xpaths):
              if len(word) < 1:  # skip empty nodes
                  continue
              word_tokens = self.tokenize(word)
              pair_tokens.extend(word_tokens)
              xpath_tags_list, xpath_subs_list = self.get_xpath_seq(
                  xpath
              )
              pair_xpath_tags_seq.extend(
                  [xpath_tags_list] * len(word_tokens)
              )
              pair_xpath_subs_seq.extend(
                  [xpath_subs_list] * len(word_tokens)
              )
  
      # Create ids + pair_ids
      ids = self.convert_tokens_to_ids(tokens)
      pair_ids = (
          self.convert_tokens_to_ids(pair_tokens)
          if pair_tokens
          else None
      )
  
      if (
          return_overflowing_tokens
          and truncation_strategy == TruncationStrategy.LONGEST_FIRST
          and pair_ids is not None
      ):
          raise ValueError(
              "Not possible to return overflowing tokens for pair of sequences with the "
              "`longest_first`. Please select another truncation strategy than `longest_first`, "
              "for instance `only_second` or `only_first`."
          )
  
      # Compute the total size of the returned encodings
      pair = bool(pair_ids is not None)
      len_ids = len(ids)
      len_pair_ids = len(pair_ids) if pair else 0
      total_len = (
          len_ids
          + len_pair_ids
          + (
              self.num_special_tokens_to_add(pair=pair)
              if add_special_tokens
              else 0
          )
      )
  
      # Truncation: Handle max sequence length
      overflowing_tokens = []
      overflowing_xpath_tags_seq = []
      overflowing_xpath_subs_seq = []
      overflowing_labels = []
      if (
          truncation_strategy != TruncationStrategy.DO_NOT_TRUNCATE
          and max_length
          and total_len > max_length
      ):
          (
              ids,
              xpath_tags_seq,
              xpath_subs_seq,
              pair_ids,
              pair_xpath_tags_seq,
              pair_xpath_subs_seq,
              labels,
              overflowing_tokens,
              overflowing_xpath_tags_seq,
              overflowing_xpath_subs_seq,
              overflowing_labels,
          ) = self.truncate_sequences(
              ids,
              xpath_tags_seq=xpath_tags_seq,
              xpath_subs_seq=xpath_subs_seq,
              pair_ids=pair_ids,
              pair_xpath_tags_seq=pair_xpath_tags_seq,
              pair_xpath_subs_seq=pair_xpath_subs_seq,
              labels=labels,
              num_tokens_to_remove=total_len - max_length,
              truncation_strategy=truncation_strategy,
              stride=stride,
          )
  
      if return_token_type_ids and not add_special_tokens:
          raise ValueError(
              "Asking to return token_type_ids while setting add_special_tokens to False "
              "results in an undefined behavior. Please set add_special_tokens to True or "
              "set return_token_type_ids to None."
          )
  
      # Load from model defaults
      if return_token_type_ids is None:
          return_token_type_ids = (
              "token_type_ids" in self.model_input_names
          )
      if return_attention_mask is None:
          return_attention_mask = (
              "attention_mask" in self.model_input_names
          )
  
      encoded_inputs = {}
  
      if return_overflowing_tokens:
          encoded_inputs["overflowing_tokens"] = overflowing_tokens
          encoded_inputs[
              "overflowing_xpath_tags_seq"
          ] = overflowing_xpath_tags_seq
          encoded_inputs[
              "overflowing_xpath_subs_seq"
          ] = overflowing_xpath_subs_seq
          encoded_inputs["overflowing_labels"] = overflowing_labels
          encoded_inputs["num_truncated_tokens"] = (
              total_len - max_length
          )
  
      # Add special tokens
      if add_special_tokens:
          sequence = self.build_inputs_with_special_tokens(
              ids, pair_ids
          )
          token_type_ids = self.create_token_type_ids_from_sequences(
              ids, pair_ids
          )
          xpath_tags_ids = self.build_xpath_tags_with_special_tokens(
              xpath_tags_seq, pair_xpath_tags_seq
          )
          xpath_subs_ids = self.build_xpath_subs_with_special_tokens(
              xpath_subs_seq, pair_xpath_subs_seq
          )
          if labels:
              labels = (
                  [self.pad_token_label]
                  + labels
                  + [self.pad_token_label]
              )
      else:
          sequence = ids + pair_ids if pair else ids
          token_type_ids = [0] * len(ids) + (
              [0] * len(pair_ids) if pair else []
          )
          xpath_tags_ids = (
              xpath_tags_seq + pair_xpath_tags_seq
              if pair
              else xpath_tags_seq
          )
          xpath_subs_ids = (
              xpath_subs_seq + pair_xpath_subs_seq
              if pair
              else xpath_subs_seq
          )
  
      # Build output dictionary
      encoded_inputs["input_ids"] = sequence
      encoded_inputs["xpath_tags_seq"] = xpath_tags_ids
      encoded_inputs["xpath_subs_seq"] = xpath_subs_ids
      if return_token_type_ids:
          encoded_inputs["token_type_ids"] = token_type_ids
      if return_special_tokens_mask:
          if add_special_tokens:
              encoded_inputs[
                  "special_tokens_mask"
              ] = self.get_special_tokens_mask(ids, pair_ids)
          else:
              encoded_inputs["special_tokens_mask"] = [0] * len(
                  sequence
              )
  
      if labels:
          encoded_inputs["labels"] = labels
  
      # Check lengths
      self._eventual_warn_about_too_long_sequence(
          encoded_inputs["input_ids"], max_length, verbose
      )
  
      # Padding
      if (
          padding_strategy != PaddingStrategy.DO_NOT_PAD
          or return_attention_mask
      ):
          encoded_inputs = self.pad(
              encoded_inputs,
              max_length=max_length,
              padding=padding_strategy.value,
              pad_to_multiple_of=pad_to_multiple_of,
              return_attention_mask=return_attention_mask,
          )
  
      if return_length:
          encoded_inputs["length"] = len(encoded_inputs["input_ids"])
  
      batch_outputs = BatchEncoding(
          encoded_inputs,
          tensor_type=return_tensors,
          prepend_batch_axis=prepend_batch_axis,
      )
  
      return batch_outputs
  
imports: import json
  import os
  from functools import lru_cache
  from typing import Dict, List, Optional, Tuple, Union
  import regex
  from ...file_utils import PaddingStrategy, TensorType, add_end_docstrings
  from ...tokenization_utils import AddedToken, PreTrainedTokenizer
  from ...tokenization_utils_base import ENCODE_KWARGS_DOCSTRING, BatchEncoding, EncodedInput, PreTokenizedInput, TextInput, TextInputPair, TruncationStrategy
  from ...utils import logging
label_desc: Use image node_id 24 for calling the MarkupLMTokenizer obj's underlying member method code with example usage: obj.prepare_for_model(text, text_pair, xpaths, node_labels, add_special_tokens, padding, truncation, max_length, stride, pad_to_multiple_of, return_tensors, return_token_type_ids, return_attention_mask, return_overflowing_tokens, return_special_tokens_mask, return_offsets_mapping, return_length, verbose, prepend_batch_axis) and returns: batch_outputs