class_docstr: Construct a "fast" REALM tokenizer (backed by HuggingFace's *tokenizers* library). Based on WordPiece.
  
  [`RealmTokenizerFast`] is identical to [`BertTokenizerFast`] and runs end-to-end tokenization: punctuation
  splitting and wordpiece.
  
  This tokenizer inherits from [`PreTrainedTokenizerFast`] which contains most of the main methods. Users should
  refer to this superclass for more information regarding those methods.
  
  Args:
      vocab_file (`str`):
          File containing the vocabulary.
      do_lower_case (`bool`, *optional*, defaults to `True`):
          Whether or not to lowercase the input when tokenizing.
      unk_token (`str`, *optional*, defaults to `"[UNK]"`):
          The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
          token instead.
      sep_token (`str`, *optional*, defaults to `"[SEP]"`):
          The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences for
          sequence classification or for a text and a question for question answering. It is also used as the last
          token of a sequence built with special tokens.
      pad_token (`str`, *optional*, defaults to `"[PAD]"`):
          The token used for padding, for example when batching sequences of different lengths.
      cls_token (`str`, *optional*, defaults to `"[CLS]"`):
          The classifier token which is used when doing sequence classification (classification of the whole sequence
          instead of per-token classification). It is the first token of the sequence when built with special tokens.
      mask_token (`str`, *optional*, defaults to `"[MASK]"`):
          The token used for masking values. This is the token used when training this model with masked language
          modeling. This is the token which the model will try to predict.
      clean_text (`bool`, *optional*, defaults to `True`):
          Whether or not to clean the text before tokenization by removing any control characters and replacing all
          whitespaces by the classic one.
      tokenize_chinese_chars (`bool`, *optional*, defaults to `True`):
          Whether or not to tokenize Chinese characters. This should likely be deactivated for Japanese (see [this
          issue](https://github.com/huggingface/transformers/issues/328)).
      strip_accents (`bool`, *optional*):
          Whether or not to strip all accents. If this option is not specified, then it will be determined by the
          value for `lowercase` (as in the original BERT).
      wordpieces_prefix (`str`, *optional*, defaults to `"##"`):
          The prefix for subwords.
docstr: Encode a batch of text or text pair. This method is similar to regular __call__ method but has the following
  differences:
  
      1. Handle additional num_candidate axis. (batch_size, num_candidates, text)
      2. Always pad the sequences to *max_length*.
      3. Must specify *max_length* in order to stack packs of candidates into a batch.
  
      - single sequence: `[CLS] X [SEP]`
      - pair of sequences: `[CLS] A [SEP] B [SEP]`
  
  Args:
      text (`List[List[str]]`):
          The batch of sequences to be encoded. Each sequence must be in this format: (batch_size,
          num_candidates, text).
      text_pair (`List[List[str]]`, *optional*):
          The batch of sequences to be encoded. Each sequence must be in this format: (batch_size,
          num_candidates, text).
      **kwargs:
          Keyword arguments of the __call__ method.
  
  Returns:
      [`BatchEncoding`]: Encoded text or text pair.
  
  Example:
  
  ```python
  >>> from transformers import RealmTokenizerFast
  
  >>> # batch_size = 2, num_candidates = 2
  >>> text = [["Hello world!", "Nice to meet you!"], ["The cute cat.", "The adorable dog."]]
  
  >>> tokenizer = RealmTokenizerFast.from_pretrained("google/realm-cc-news-pretrained-encoder")
  >>> tokenized_text = tokenizer.batch_encode_candidates(text, max_length=10, return_tensors="pt")
  ```
docstr_tok: ["Encode","a","batch","of","text","or","text","pair",".","This","method","is","similar","to","regular","__call__","method","but","has","the","following","differences",":","1",".","Handle","additional","num_candidate","axis",".","(","batch_size",",","num_candidates",",","text",")","2",".","Always","pad","the","sequences","to","*","max_length","*",".","3",".","Must","specify","*","max_length","*","in","order","to","stack","packs","of","candidates","into","a","batch",".","-","single","sequence",":","`","[","CLS","]","X","[","SEP","]","`","-","pair","of","sequences",":","`","[","CLS","]","A","[","SEP","]","B","[","SEP","]","`","Args",":","text","(","`","List","[","List","[","str","]","]","`",")",":","The","batch","of","sequences","to","be","encoded",".","Each","sequence","must","be","in","this","format",":","(","batch_size",",","num_candidates",",","text",")",".","text_pair","(","`","List","[","List","[","str","]","]","`",",","*","optional","*",")",":","The","batch","of","sequences","to","be","encoded",".","Each","sequence","must","be","in","this","format",":","(","batch_size",",","num_candidates",",","text",")",".","*","*","kwargs",":","Keyword","arguments","of","the","__call__","method",".","Returns",":","[","`","BatchEncoding","`","]",":","Encoded","text","or","text","pair",".","Example",":","``","`","python",">",">",">","from","transformers","import","RealmTokenizerFast",">",">",">","#","batch_size","=","2",",","num_candidates","=","2",">",">",">","text","=","[","[","``","Hello","world","!","``",",","``","Nice","to","meet","you","!","``","]",",","[","``","The","cute","cat",".","``",",","``","The","adorable","dog",".","''","]","]",">",">",">","tokenizer","=","RealmTokenizerFast.from_pretrained","(","``","google\/realm-cc-news-pretrained-encoder","''",")",">",">",">","tokenized_text","=","tokenizer.batch_encode_candidates","(","text",",","max_length=10",",","return_tensors=","''","pt","''",")","``","`"]
code: def batch_encode_candidates(self, text, **kwargs):
      r"""
      Encode a batch of text or text pair. This method is similar to regular __call__ method but has the following
      differences:
  
          1. Handle additional num_candidate axis. (batch_size, num_candidates, text)
          2. Always pad the sequences to *max_length*.
          3. Must specify *max_length* in order to stack packs of candidates into a batch.
  
          - single sequence: `[CLS] X [SEP]`
          - pair of sequences: `[CLS] A [SEP] B [SEP]`
  
      Args:
          text (`List[List[str]]`):
              The batch of sequences to be encoded. Each sequence must be in this format: (batch_size,
              num_candidates, text).
          text_pair (`List[List[str]]`, *optional*):
              The batch of sequences to be encoded. Each sequence must be in this format: (batch_size,
              num_candidates, text).
          **kwargs:
              Keyword arguments of the __call__ method.
  
      Returns:
          [`BatchEncoding`]: Encoded text or text pair.
  
      Example:
  
      ```python
      >>> from transformers import RealmTokenizerFast
  
      >>> # batch_size = 2, num_candidates = 2
      >>> text = [["Hello world!", "Nice to meet you!"], ["The cute cat.", "The adorable dog."]]
  
      >>> tokenizer = RealmTokenizerFast.from_pretrained("google/realm-cc-news-pretrained-encoder")
      >>> tokenized_text = tokenizer.batch_encode_candidates(text, max_length=10, return_tensors="pt")
      ```"""
  
      # Always using a fixed sequence length to encode in order to stack candidates into a batch.
      kwargs["padding"] = PaddingStrategy.MAX_LENGTH
  
      batch_text = text
      batch_text_pair = kwargs.pop("text_pair", None)
      return_tensors = kwargs.pop("return_tensors", None)
  
      output_data = {
          "input_ids": [],
          "attention_mask": [],
          "token_type_ids": [],
      }
  
      for idx, candidate_text in enumerate(batch_text):
          if batch_text_pair is not None:
              candidate_text_pair = batch_text_pair[idx]
          else:
              candidate_text_pair = None
  
          encoded_candidates = super().__call__(
              candidate_text,
              candidate_text_pair,
              return_tensors=None,
              **kwargs,
          )
  
          encoded_input_ids = encoded_candidates.get("input_ids")
          encoded_attention_mask = encoded_candidates.get(
              "attention_mask"
          )
          encoded_token_type_ids = encoded_candidates.get(
              "token_type_ids"
          )
  
          if encoded_input_ids is not None:
              output_data["input_ids"].append(encoded_input_ids)
          if encoded_attention_mask is not None:
              output_data["attention_mask"].append(
                  encoded_attention_mask
              )
          if encoded_token_type_ids is not None:
              output_data["token_type_ids"].append(
                  encoded_token_type_ids
              )
  
      output_data = dict(
          (key, item)
          for key, item in output_data.items()
          if len(item) != 0
      )
  
      return BatchEncoding(output_data, tensor_type=return_tensors)
  
file_path: H2O/h2o_flexgen/benchmark/third_party/transformers/src/transformers/models/realm/tokenization_realm_fast.py
imports: import json
  from typing import List, Optional, Tuple
  from tokenizers import normalizers
  from ...tokenization_utils_base import BatchEncoding
  from ...tokenization_utils_fast import PreTrainedTokenizerFast
  from ...utils import PaddingStrategy, logging
  from .tokenization_realm import RealmTokenizer
label_desc: Use image node_id 2 for calling the RealmTokenizerFast obj's underlying member method code with example usage: obj.batch_encode_candidates(text) and returns: BatchEncoding