class_docstr: Construct a "fast" RoFormer tokenizer (backed by HuggingFace's *tokenizers* library).
  
  [`RoFormerTokenizerFast`] is almost identical to [`BertTokenizerFast`] and runs end-to-end tokenization:
  punctuation splitting and wordpiece. There are some difference between them when tokenizing Chinese.
  
  This tokenizer inherits from [`PreTrainedTokenizerFast`] which contains most of the main methods. Users should
  refer to this superclass for more information regarding those methods.
  
  Example:
  
  ```python
  >>> from transformers import RoFormerTokenizerFast
  
  >>> tokenizer = RoFormerTokenizerFast.from_pretrained("junnyu/roformer_chinese_base")
  >>> tokenizer.tokenize("今天天气非常好。")
  # ['今', '天', '天', '气', '非常', '好', '。']
  ```
class_docstr_tok: ["Construct","a","``","fast","''","RoFormer","tokenizer","(","backed","by","HuggingFace","'s","*","tokenizers","*","library",")",".","[","`","RoFormerTokenizerFast","`","]","is","almost","identical","to","[","`","BertTokenizerFast","`","]","and","runs","end-to-end","tokenization",":","punctuation","splitting","and","wordpiece",".","There","are","some","difference","between","them","when","tokenizing","Chinese",".","This","tokenizer","inherits","from","[","`","PreTrainedTokenizerFast","`","]","which","contains","most","of","the","main","methods",".","Users","should","refer","to","this","superclass","for","more","information","regarding","those","methods",".","Example",":","``","`","python",">",">",">","from","transformers","import","RoFormerTokenizerFast",">",">",">","tokenizer","=","RoFormerTokenizerFast.from_pretrained","(","``","junnyu\/roformer_chinese_base","''",")",">",">",">","tokenizer.tokenize","(","``","\u4eca\u5929\u5929\u6c14\u975e\u5e38\u597d\u3002","''",")","#","[","'","\u4eca","'",",","'","\u5929","'",",","'","\u5929","'",",","'","\u6c14","'",",","'\u975e\u5e38","'",",","'","\u597d","'",",","'\u3002","'","]","``","`"]
docstr: Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and
  adding special tokens. A RoFormer sequence has the following format:
  
  - single sequence: `[CLS] X [SEP]`
  - pair of sequences: `[CLS] A [SEP] B [SEP]`
  
  Args:
      token_ids_0 (`List[int]`):
          List of IDs to which the special tokens will be added.
      token_ids_1 (`List[int]`, *optional*):
          Optional second list of IDs for sequence pairs.
  
  Returns:
      `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens.
docstr_tok: ["Build","model","inputs","from","a","sequence","or","a","pair","of","sequence","for","sequence","classification","tasks","by","concatenating","and","adding","special","tokens",".","A","RoFormer","sequence","has","the","following","format",":","-","single","sequence",":","`","[","CLS","]","X","[","SEP","]","`","-","pair","of","sequences",":","`","[","CLS","]","A","[","SEP","]","B","[","SEP","]","`","Args",":","token_ids_0","(","`","List","[","int","]","`",")",":","List","of","IDs","to","which","the","special","tokens","will","be","added",".","token_ids_1","(","`","List","[","int","]","`",",","*","optional","*",")",":","Optional","second","list","of","IDs","for","sequence","pairs",".","Returns",":","`","List","[","int","]","`",":","List","of","[","input","IDs","]","(","..","\/glossary","#","input-ids",")","with","the","appropriate","special","tokens","."]
code: def build_inputs_with_special_tokens(
      self, token_ids_0, token_ids_1=None
  ):
      """
      Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and
      adding special tokens. A RoFormer sequence has the following format:
  
      - single sequence: `[CLS] X [SEP]`
      - pair of sequences: `[CLS] A [SEP] B [SEP]`
  
      Args:
          token_ids_0 (`List[int]`):
              List of IDs to which the special tokens will be added.
          token_ids_1 (`List[int]`, *optional*):
              Optional second list of IDs for sequence pairs.
  
      Returns:
          `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens.
      """
      output = [self.cls_token_id] + token_ids_0 + [self.sep_token_id]
  
      if token_ids_1:
          output += token_ids_1 + [self.sep_token_id]
  
      return output
  
file_path: H2O/h2o_flexgen/benchmark/third_party/transformers/src/transformers/models/roformer/tokenization_roformer_fast.py
imports: import json
  from typing import List, Optional, Tuple
  from tokenizers import normalizers
  from tokenizers.pre_tokenizers import BertPreTokenizer, PreTokenizer
  from ...tokenization_utils_fast import PreTrainedTokenizerFast
  from ...utils import logging
  from .tokenization_roformer import RoFormerTokenizer
  from .tokenization_utils import JiebaPreTokenizer
label_desc: Use image node_id 4 for calling the RoFormerTokenizerFast obj's underlying member method code with example usage: obj.build_inputs_with_special_tokens(token_ids_0, token_ids_1) and returns: output