class_docstr: Construct a "fast" RoFormer tokenizer (backed by HuggingFace's *tokenizers* library).
  
  [`RoFormerTokenizerFast`] is almost identical to [`BertTokenizerFast`] and runs end-to-end tokenization:
  punctuation splitting and wordpiece. There are some difference between them when tokenizing Chinese.
  
  This tokenizer inherits from [`PreTrainedTokenizerFast`] which contains most of the main methods. Users should
  refer to this superclass for more information regarding those methods.
  
  Example:
  
  ```python
  >>> from transformers import RoFormerTokenizerFast
  
  >>> tokenizer = RoFormerTokenizerFast.from_pretrained("junnyu/roformer_chinese_base")
  >>> tokenizer.tokenize("今天天气非常好。")
  # ['今', '天', '天', '气', '非常', '好', '。']
  ```
class_docstr_tok: ["Construct","a","``","fast","''","RoFormer","tokenizer","(","backed","by","HuggingFace","'s","*","tokenizers","*","library",")",".","[","`","RoFormerTokenizerFast","`","]","is","almost","identical","to","[","`","BertTokenizerFast","`","]","and","runs","end-to-end","tokenization",":","punctuation","splitting","and","wordpiece",".","There","are","some","difference","between","them","when","tokenizing","Chinese",".","This","tokenizer","inherits","from","[","`","PreTrainedTokenizerFast","`","]","which","contains","most","of","the","main","methods",".","Users","should","refer","to","this","superclass","for","more","information","regarding","those","methods",".","Example",":","``","`","python",">",">",">","from","transformers","import","RoFormerTokenizerFast",">",">",">","tokenizer","=","RoFormerTokenizerFast.from_pretrained","(","``","junnyu\/roformer_chinese_base","''",")",">",">",">","tokenizer.tokenize","(","``","\u4eca\u5929\u5929\u6c14\u975e\u5e38\u597d\u3002","''",")","#","[","'","\u4eca","'",",","'","\u5929","'",",","'","\u5929","'",",","'","\u6c14","'",",","'\u975e\u5e38","'",",","'","\u597d","'",",","'\u3002","'","]","``","`"]
code: def __init__(
      self,
      vocab_file=None,
      tokenizer_file=None,
      do_lower_case=True,
      unk_token="[UNK]",
      sep_token="[SEP]",
      pad_token="[PAD]",
      cls_token="[CLS]",
      mask_token="[MASK]",
      tokenize_chinese_chars=True,
      strip_accents=None,
      **kwargs,
  ):
      super().__init__(
          vocab_file,
          tokenizer_file=tokenizer_file,
          do_lower_case=do_lower_case,
          unk_token=unk_token,
          sep_token=sep_token,
          pad_token=pad_token,
          cls_token=cls_token,
          mask_token=mask_token,
          tokenize_chinese_chars=tokenize_chinese_chars,
          strip_accents=strip_accents,
          **kwargs,
      )
  
      pre_tok_state = json.loads(
          self.backend_tokenizer.normalizer.__getstate__()
      )
      if (
          pre_tok_state.get("lowercase", do_lower_case) != do_lower_case
          or pre_tok_state.get("strip_accents", strip_accents)
          != strip_accents
      ):
          pre_tok_class = getattr(
              normalizers, pre_tok_state.pop("type")
          )
          pre_tok_state["lowercase"] = do_lower_case
          pre_tok_state["strip_accents"] = strip_accents
          self.backend_tokenizer.normalizer = pre_tok_class(
              **pre_tok_state
          )
  
      self.do_lower_case = do_lower_case
  
file_path: H2O/h2o_flexgen/benchmark/third_party/transformers/src/transformers/models/roformer/tokenization_roformer_fast.py
imports: import json
  from typing import List, Optional, Tuple
  from tokenizers import normalizers
  from tokenizers.pre_tokenizers import BertPreTokenizer, PreTokenizer
  from ...tokenization_utils_fast import PreTrainedTokenizerFast
  from ...utils import logging
  from .tokenization_roformer import RoFormerTokenizer
  from .tokenization_utils import JiebaPreTokenizer
label_desc: Use image node_id 1 to create a new RoFormerTokenizerFast object from inherited base classes: PreTrainedTokenizerFast with example: obj = RoFormerTokenizerFast(vocab_file, tokenizer_file, do_lower_case, unk_token, sep_token, pad_token, cls_token, mask_token, tokenize_chinese_chars, strip_accents)