class_docstr: Tokenizer used for BERT, a faster version with TFLite support.
  
    This tokenizer applies an end-to-end, text string to wordpiece tokenization.
    It is equivalent to `BertTokenizer` for most common scenarios while running
    faster and supporting TFLite. It does not support certain special settings
    (see the docs below).
  
    See `WordpieceTokenizer` for details on the subword tokenization.
  
    For an example of use, see
    https://www.tensorflow.org/text/guide/bert_preprocessing_guide
  
  Attributes:
    vocab: (optional) The list of tokens in the vocabulary.
    suffix_indicator: (optional) The characters prepended to a wordpiece to
      indicate that it is a suffix to another subword.
    max_bytes_per_word: (optional) Max size of input token.
    token_out_type: (optional) The type of the token to return. This can be
      `tf.int64` or `tf.int32` IDs, or `tf.string` subwords.
    unknown_token: (optional) The string value to substitute for an unknown
      token. It must be included in `vocab`.
    no_pretokenization: (optional) By default, the input is split on
      whitespaces and punctuations before applying the Wordpiece tokenization.
      When true, the input is assumed to be pretokenized already.
    support_detokenization: (optional) Whether to make the tokenizer support
      doing detokenization. Setting it to true expands the size of the model
      flatbuffer. As a reference, when using 120k multilingual BERT WordPiece
      vocab, the flatbuffer's size increases from ~5MB to ~6MB.
    fast_wordpiece_model_buffer: (optional) Bytes object (or a uint8 tf.Tenosr)
      that contains the wordpiece model in flatbuffer format (see
      fast_wordpiece_tokenizer_model.fbs). If not `None`, all other arguments
      related to FastWordPieceTokenizer (except `token_output_type`) are
      ignored.
    lower_case_nfd_strip_accents: (optional) .
      - If true, it first lowercases the text, applies NFD normalization, strips
      accents characters, and then replaces control characters with whitespaces.
      - If false, it only replaces control characters with whitespaces.
    fast_bert_normalizer_model_buffer: (optional) bytes object (or a uint8
      tf.Tenosr) that contains the fast bert normalizer model in flatbuffer
      format (see fast_bert_normalizer_model.fbs). If not `None`,
      `lower_case_nfd_strip_accents` is ignored.
class_docstr_tok: ["Tokenizer","used","for","BERT",",","a","faster","version","with","TFLite","support",".","This","tokenizer","applies","an","end-to-end",",","text","string","to","wordpiece","tokenization",".","It","is","equivalent","to","`","BertTokenizer","`","for","most","common","scenarios","while","running","faster","and","supporting","TFLite",".","It","does","not","support","certain","special","settings","(","see","the","docs","below",")",".","See","`","WordpieceTokenizer","`","for","details","on","the","subword","tokenization",".","For","an","example","of","use",",","see","https",":","\/\/www.tensorflow.org\/text\/guide\/bert_preprocessing_guide","Attributes",":","vocab",":","(","optional",")","The","list","of","tokens","in","the","vocabulary",".","suffix_indicator",":","(","optional",")","The","characters","prepended","to","a","wordpiece","to","indicate","that","it","is","a","suffix","to","another","subword",".","max_bytes_per_word",":","(","optional",")","Max","size","of","input","token",".","token_out_type",":","(","optional",")","The","type","of","the","token","to","return",".","This","can","be","`","tf.int64","`","or","`","tf.int32","`","IDs",",","or","`","tf.string","`","subwords",".","unknown_token",":","(","optional",")","The","string","value","to","substitute","for","an","unknown","token",".","It","must","be","included","in","`","vocab","`",".","no_pretokenization",":","(","optional",")","By","default",",","the","input","is","split","on","whitespaces","and","punctuations","before","applying","the","Wordpiece","tokenization",".","When","true",",","the","input","is","assumed","to","be","pretokenized","already",".","support_detokenization",":","(","optional",")","Whether","to","make","the","tokenizer","support","doing","detokenization",".","Setting","it","to","true","expands","the","size","of","the","model","flatbuffer",".","As","a","reference",",","when","using","120k","multilingual","BERT","WordPiece","vocab",",","the","flatbuffer","'s","size","increases","from","~5MB","to","~6MB",".","fast_wordpiece_model_buffer",":","(","optional",")","Bytes","object","(","or","a","uint8","tf.Tenosr",")","that","contains","the","wordpiece","model","in","flatbuffer","format","(","see","fast_wordpiece_tokenizer_model.fbs",")",".","If","not","`","None","`",",","all","other","arguments","related","to","FastWordPieceTokenizer","(","except","`","token_output_type","`",")","are","ignored",".","lower_case_nfd_strip_accents",":","(","optional",")",".","-","If","true",",","it","first","lowercases","the","text",",","applies","NFD","normalization",",","strips","accents","characters",",","and","then","replaces","control","characters","with","whitespaces",".","-","If","false",",","it","only","replaces","control","characters","with","whitespaces",".","fast_bert_normalizer_model_buffer",":","(","optional",")","bytes","object","(","or","a","uint8","tf.Tenosr",")","that","contains","the","fast","bert","normalizer","model","in","flatbuffer","format","(","see","fast_bert_normalizer_model.fbs",")",".","If","not","`","None","`",",","`","lower_case_nfd_strip_accents","`","is","ignored","."]
docstr: Tokenizes a tensor of string tokens into subword tokens for BERT.
  
  Example:
  >>> vocab = ['they', "##'", '##re', 'the', 'great', '##est', '[UNK]']
  >>> tokenizer = FastBertTokenizer(vocab=vocab)
  >>> text_inputs = tf.constant(['greatest'.encode('utf-8')])
  >>> tokenizer.tokenize_with_offsets(text_inputs)
  (<tf.RaggedTensor [[4, 5]]>,
   <tf.RaggedTensor [[0, 5]]>,
   <tf.RaggedTensor [[5, 8]]>)
  
  Args:
    text_input: input: A `Tensor` or `RaggedTensor` of untokenized UTF-8
      strings.
  
  Returns:
    A tuple of `RaggedTensor`s where the first element is the tokens where
    `tokens[i1...iN, j]`, the second element is the starting offsets, the
    third element is the end offset. (Please look at `tokenize` for details
    on tokens.)
docstr_tok: ["Tokenizes","a","tensor","of","string","tokens","into","subword","tokens","for","BERT",".","Example",":",">",">",">","vocab","=","[","'they","'",",","``","#","#","'","''",",","'","#","#","re","'",",","'the","'",",","'great","'",",","'","#","#","est","'",",","'","[","UNK","]","'","]",">",">",">","tokenizer","=","FastBertTokenizer","(","vocab=vocab",")",">",">",">","text_inputs","=","tf.constant","(","[","'greatest'.encode","(","'utf-8","'",")","]",")",">",">",">","tokenizer.tokenize_with_offsets","(","text_inputs",")","(","<","tf.RaggedTensor","[","[","4",",","5","]","]",">",",","<","tf.RaggedTensor","[","[","0",",","5","]","]",">",",","<","tf.RaggedTensor","[","[","5",",","8","]","]",">",")","Args",":","text_input",":","input",":","A","`","Tensor","`","or","`","RaggedTensor","`","of","untokenized","UTF-8","strings",".","Returns",":","A","tuple","of","`","RaggedTensor","`","s","where","the","first","element","is","the","tokens","where","`","tokens","[","i1","...","iN",",","j","]","`",",","the","second","element","is","the","starting","offsets",",","the","third","element","is","the","end","offset",".","(","Please","look","at","`","tokenize","`","for","details","on","tokens",".",")"]
code: def tokenize_with_offsets(self, text_input):
      r"""Tokenizes a tensor of string tokens into subword tokens for BERT.
  
      Example:
      >>> vocab = ['they', "##'", '##re', 'the', 'great', '##est', '[UNK]']
      >>> tokenizer = FastBertTokenizer(vocab=vocab)
      >>> text_inputs = tf.constant(['greatest'.encode('utf-8')])
      >>> tokenizer.tokenize_with_offsets(text_inputs)
      (<tf.RaggedTensor [[4, 5]]>,
       <tf.RaggedTensor [[0, 5]]>,
       <tf.RaggedTensor [[5, 8]]>)
  
      Args:
        text_input: input: A `Tensor` or `RaggedTensor` of untokenized UTF-8
          strings.
  
      Returns:
        A tuple of `RaggedTensor`s where the first element is the tokens where
        `tokens[i1...iN, j]`, the second element is the starting offsets, the
        third element is the end offset. (Please look at `tokenize` for details
        on tokens.)
  
      """
      (
          normalized_input,
          offsets,
      ) = self._fast_bert_normalizer.normalize_with_offsets(text_input)
      (
          wordpieces,
          post_norm_offsets_starts,
          post_norm_offsets_ends,
      ) = self._fast_wordpiece_tokenizer.tokenize_with_offsets(
          normalized_input
      )
      pre_norm_offsets_starts = array_ops.gather(
          offsets, post_norm_offsets_starts, axis=-1, batch_dims=-1
      )
      pre_norm_offsets_ends = array_ops.gather(
          offsets, post_norm_offsets_ends, axis=-1, batch_dims=-1
      )
      return wordpieces, pre_norm_offsets_starts, pre_norm_offsets_ends
  
file_path: text/tensorflow_text/python/ops/fast_bert_tokenizer.py
imports: from __future__ import absolute_import
  from __future__ import division
  from __future__ import print_function
  from tensorflow.python.eager import monitoring
  from tensorflow.python.framework import dtypes
  from tensorflow.python.ops import array_ops
  from tensorflow_text.python.ops.fast_bert_normalizer import FastBertNormalizer
  from tensorflow_text.python.ops.fast_wordpiece_tokenizer import FastWordpieceTokenizer
  from tensorflow_text.python.ops.tokenization import Detokenizer
  from tensorflow_text.python.ops.tokenization import TokenizerWithOffsets
label_desc: Use image node_id 2 for calling the FastBertTokenizer obj's underlying member method code with example usage: obj.tokenize_with_offsets(text_input) and returns: wordpieces,  pre_norm_offsets_starts,  pre_norm_offsets_ends