class_docstr: This is an in-graph tokenizer for BERT. It should be initialized similarly to other tokenizers, using the
  `from_pretrained()` method. It can also be initialized with the `from_tokenizer()` method, which imports settings
  from an existing standard tokenizer object.
  
  In-graph tokenizers, unlike other Hugging Face tokenizers, are actually Keras layers and are designed to be run
  when the model is called, rather than during preprocessing. As a result, they have somewhat more limited options
  than standard tokenizer classes. They are most useful when you want to create an end-to-end model that goes
  straight from `tf.string` inputs to outputs.
  
  Args:
      vocab_list (`list`):
          List containing the vocabulary.
      do_lower_case (`bool`, *optional*, defaults to `True`):
          Whether or not to lowercase the input when tokenizing.
      cls_token_id (`str`, *optional*, defaults to `"[CLS]"`):
          The classifier token which is used when doing sequence classification (classification of the whole sequence
          instead of per-token classification). It is the first token of the sequence when built with special tokens.
      sep_token_id (`str`, *optional*, defaults to `"[SEP]"`):
          The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences for
          sequence classification or for a text and a question for question answering. It is also used as the last
          token of a sequence built with special tokens.
      pad_token_id (`str`, *optional*, defaults to `"[PAD]"`):
          The token used for padding, for example when batching sequences of different lengths.
      padding (`str`, defaults to `"longest"`):
          The type of padding to use. Can be either `"longest"`, to pad only up to the longest sample in the batch,
          or `"max_length", to pad all inputs to the maximum length supported by the tokenizer.
      truncation (`bool`, *optional*, defaults to `True`):
          Whether to truncate the sequence to the maximum length.
      max_length (`int`, *optional*, defaults to `512`):
          The maximum length of the sequence, used for padding (if `padding` is "max_length") and/or truncation (if
          `truncation` is `True`).
      pad_to_multiple_of (`int`, *optional*, defaults to `None`):
          If set, the sequence will be padded to a multiple of this value.
      return_token_type_ids (`bool`, *optional*, defaults to `True`):
          Whether to return token_type_ids.
      return_attention_mask (`bool`, *optional*, defaults to `True`):
          Whether to return the attention_mask.
      use_fast_bert_tokenizer (`bool`, *optional*, defaults to `True`):
          If set to false will use standard TF Text BertTokenizer, making it servable by TF Serving.
class_docstr_tok: ["This","is","an","in-graph","tokenizer","for","BERT",".","It","should","be","initialized","similarly","to","other","tokenizers",",","using","the","`","from_pretrained","(",")","`","method",".","It","can","also","be","initialized","with","the","`","from_tokenizer","(",")","`","method",",","which","imports","settings","from","an","existing","standard","tokenizer","object",".","In-graph","tokenizers",",","unlike","other","Hugging","Face","tokenizers",",","are","actually","Keras","layers","and","are","designed","to","be","run","when","the","model","is","called",",","rather","than","during","preprocessing",".","As","a","result",",","they","have","somewhat","more","limited","options","than","standard","tokenizer","classes",".","They","are","most","useful","when","you","want","to","create","an","end-to-end","model","that","goes","straight","from","`","tf.string","`","inputs","to","outputs",".","Args",":","vocab_list","(","`","list","`",")",":","List","containing","the","vocabulary",".","do_lower_case","(","`","bool","`",",","*","optional","*",",","defaults","to","`","True","`",")",":","Whether","or","not","to","lowercase","the","input","when","tokenizing",".","cls_token_id","(","`","str","`",",","*","optional","*",",","defaults","to","`","``","[","CLS","]","''","`",")",":","The","classifier","token","which","is","used","when","doing","sequence","classification","(","classification","of","the","whole","sequence","instead","of","per-token","classification",")",".","It","is","the","first","token","of","the","sequence","when","built","with","special","tokens",".","sep_token_id","(","`","str","`",",","*","optional","*",",","defaults","to","`","``","[","SEP","]","''","`",")",":","The","separator","token",",","which","is","used","when","building","a","sequence","from","multiple","sequences",",","e.g",".","two","sequences","for","sequence","classification","or","for","a","text","and","a","question","for","question","answering",".","It","is","also","used","as","the","last","token","of","a","sequence","built","with","special","tokens",".","pad_token_id","(","`","str","`",",","*","optional","*",",","defaults","to","`","``","[","PAD","]","''","`",")",":","The","token","used","for","padding",",","for","example","when","batching","sequences","of","different","lengths",".","padding","(","`","str","`",",","defaults","to","`","``","longest","''","`",")",":","The","type","of","padding","to","use",".","Can","be","either","`","``","longest","''","`",",","to","pad","only","up","to","the","longest","sample","in","the","batch",",","or","`","``","max_length","''",",","to","pad","all","inputs","to","the","maximum","length","supported","by","the","tokenizer",".","truncation","(","`","bool","`",",","*","optional","*",",","defaults","to","`","True","`",")",":","Whether","to","truncate","the","sequence","to","the","maximum","length",".","max_length","(","`","int","`",",","*","optional","*",",","defaults","to","`","512","`",")",":","The","maximum","length","of","the","sequence",",","used","for","padding","(","if","`","padding","`","is","``","max_length","''",")","and\/or","truncation","(","if","`","truncation","`","is","`","True","`",")",".","pad_to_multiple_of","(","`","int","`",",","*","optional","*",",","defaults","to","`","None","`",")",":","If","set",",","the","sequence","will","be","padded","to","a","multiple","of","this","value",".","return_token_type_ids","(","`","bool","`",",","*","optional","*",",","defaults","to","`","True","`",")",":","Whether","to","return","token_type_ids",".","return_attention_mask","(","`","bool","`",",","*","optional","*",",","defaults","to","`","True","`",")",":","Whether","to","return","the","attention_mask",".","use_fast_bert_tokenizer","(","`","bool","`",",","*","optional","*",",","defaults","to","`","True","`",")",":","If","set","to","false","will","use","standard","TF","Text","BertTokenizer",",","making","it","servable","by","TF","Serving","."]
docstr: Initialize a `TFBertTokenizer` from an existing `Tokenizer`.
  
  Args:
      tokenizer (`PreTrainedTokenizerBase`):
          The tokenizer to use to initialize the `TFBertTokenizer`.
  
  Examples:
  
  ```python
  from transformers import AutoTokenizer, TFBertTokenizer
  
  tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
  tf_tokenizer = TFBertTokenizer.from_tokenizer(tokenizer)
  ```
docstr_tok: ["Initialize","a","`","TFBertTokenizer","`","from","an","existing","`","Tokenizer","`",".","Args",":","tokenizer","(","`","PreTrainedTokenizerBase","`",")",":","The","tokenizer","to","use","to","initialize","the","`","TFBertTokenizer","`",".","Examples",":","``","`","python","from","transformers","import","AutoTokenizer",",","TFBertTokenizer","tokenizer","=","AutoTokenizer.from_pretrained","(","``","bert-base-uncased","''",")","tf_tokenizer","=","TFBertTokenizer.from_tokenizer","(","tokenizer",")","``","`"]
code: def from_tokenizer(
      cls, tokenizer: "PreTrainedTokenizerBase", **kwargs
  ):  # noqa: F821
      """
      Initialize a `TFBertTokenizer` from an existing `Tokenizer`.
  
      Args:
          tokenizer (`PreTrainedTokenizerBase`):
              The tokenizer to use to initialize the `TFBertTokenizer`.
  
      Examples:
  
      ```python
      from transformers import AutoTokenizer, TFBertTokenizer
  
      tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
      tf_tokenizer = TFBertTokenizer.from_tokenizer(tokenizer)
      ```
      """
      vocab = tokenizer.get_vocab()
      vocab = sorted(
          [(wordpiece, idx) for wordpiece, idx in vocab.items()],
          key=lambda x: x[1],
      )
      vocab_list = [entry[0] for entry in vocab]
      return cls(
          vocab_list=vocab_list,
          do_lower_case=tokenizer.do_lower_case,
          cls_token_id=tokenizer.cls_token_id,
          sep_token_id=tokenizer.sep_token_id,
          pad_token_id=tokenizer.pad_token_id,
          **kwargs,
      )
  
file_path: H2O/h2o_flexgen/benchmark/third_party/transformers/src/transformers/models/bert/tokenization_bert_tf.py
imports: import os
  from typing import List, Union
  import tensorflow
  from tensorflow_text import BertTokenizer
  from tensorflow_text import FastBertTokenizer, ShrinkLongestTrimmer, case_fold_utf8, combine_segments, pad_model_inputs
  from .tokenization_bert import BertTokenizer
label_desc: Use image node_id 2 for calling the TFBertTokenizer obj's underlying member method code with example usage: obj.from_tokenizer(cls, tokenizer) and returns: cls