class_docstr: Construct a BPE tokenizer for HerBERT.
  
  Peculiarities:
  
  - uses BERT's pre-tokenizer: BaseTokenizer splits tokens on spaces, and also on punctuation. Each occurrence of a
    punctuation character will be treated separately.
  
  - Such pretokenized input is BPE subtokenized
  
  This tokenizer inherits from [`XLMTokenizer`] which contains most of the methods. Users should refer to the
  superclass for more information regarding methods.
docstr: Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and
  adding special tokens. An XLM sequence has the following format:
  
  - single sequence: `<s> X </s>`
  - pair of sequences: `<s> A </s> B </s>`
  
  Args:
      token_ids_0 (`List[int]`):
          List of IDs to which the special tokens will be added.
      token_ids_1 (`List[int]`, *optional*):
          Optional second list of IDs for sequence pairs.
  
  Returns:
      `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens.
docstr_tok: ["Build","model","inputs","from","a","sequence","or","a","pair","of","sequence","for","sequence","classification","tasks","by","concatenating","and","adding","special","tokens",".","An","XLM","sequence","has","the","following","format",":","-","single","sequence",":","`","<","s",">","X","<","\/s",">","`","-","pair","of","sequences",":","`","<","s",">","A","<","\/s",">","B","<","\/s",">","`","Args",":","token_ids_0","(","`","List","[","int","]","`",")",":","List","of","IDs","to","which","the","special","tokens","will","be","added",".","token_ids_1","(","`","List","[","int","]","`",",","*","optional","*",")",":","Optional","second","list","of","IDs","for","sequence","pairs",".","Returns",":","`","List","[","int","]","`",":","List","of","[","input","IDs","]","(","..","\/glossary","#","input-ids",")","with","the","appropriate","special","tokens","."]
code: def build_inputs_with_special_tokens(
      self,
      token_ids_0: List[int],
      token_ids_1: Optional[List[int]] = None,
  ) -> List[int]:
      """
      Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and
      adding special tokens. An XLM sequence has the following format:
  
      - single sequence: `<s> X </s>`
      - pair of sequences: `<s> A </s> B </s>`
  
      Args:
          token_ids_0 (`List[int]`):
              List of IDs to which the special tokens will be added.
          token_ids_1 (`List[int]`, *optional*):
              Optional second list of IDs for sequence pairs.
  
      Returns:
          `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens.
  
      """
      bos = [self.bos_token_id]
      sep = [self.sep_token_id]
  
      if token_ids_1 is None:
          return bos + token_ids_0 + sep
      return bos + token_ids_0 + sep + token_ids_1 + sep
  
imports: import json
  import os
  import re
  import unicodedata
  from typing import List, Optional, Tuple
  from ...tokenization_utils import PreTrainedTokenizer, _is_control, _is_punctuation, _is_whitespace
  from ...utils import logging
label_desc: Use image node_id 14 for calling the HerbertTokenizer obj's underlying member method code with example usage: obj.build_inputs_with_special_tokens(token_ids_0, token_ids_1) and returns: unknown, unknown