class_docstr: Tokenizer used for BERT, a faster version with TFLite support.
  
    This tokenizer applies an end-to-end, text string to wordpiece tokenization.
    It is equivalent to `BertTokenizer` for most common scenarios while running
    faster and supporting TFLite. It does not support certain special settings
    (see the docs below).
  
    See `WordpieceTokenizer` for details on the subword tokenization.
  
    For an example of use, see
    https://www.tensorflow.org/text/guide/bert_preprocessing_guide
  
  Attributes:
    vocab: (optional) The list of tokens in the vocabulary.
    suffix_indicator: (optional) The characters prepended to a wordpiece to
      indicate that it is a suffix to another subword.
    max_bytes_per_word: (optional) Max size of input token.
    token_out_type: (optional) The type of the token to return. This can be
      `tf.int64` or `tf.int32` IDs, or `tf.string` subwords.
    unknown_token: (optional) The string value to substitute for an unknown
      token. It must be included in `vocab`.
    no_pretokenization: (optional) By default, the input is split on
      whitespaces and punctuations before applying the Wordpiece tokenization.
      When true, the input is assumed to be pretokenized already.
    support_detokenization: (optional) Whether to make the tokenizer support
      doing detokenization. Setting it to true expands the size of the model
      flatbuffer. As a reference, when using 120k multilingual BERT WordPiece
      vocab, the flatbuffer's size increases from ~5MB to ~6MB.
    fast_wordpiece_model_buffer: (optional) Bytes object (or a uint8 tf.Tenosr)
      that contains the wordpiece model in flatbuffer format (see
      fast_wordpiece_tokenizer_model.fbs). If not `None`, all other arguments
      related to FastWordPieceTokenizer (except `token_output_type`) are
      ignored.
    lower_case_nfd_strip_accents: (optional) .
      - If true, it first lowercases the text, applies NFD normalization, strips
      accents characters, and then replaces control characters with whitespaces.
      - If false, it only replaces control characters with whitespaces.
    fast_bert_normalizer_model_buffer: (optional) bytes object (or a uint8
      tf.Tenosr) that contains the fast bert normalizer model in flatbuffer
      format (see fast_bert_normalizer_model.fbs). If not `None`,
      `lower_case_nfd_strip_accents` is ignored.
class_docstr_tok: ["Tokenizer","used","for","BERT",",","a","faster","version","with","TFLite","support",".","This","tokenizer","applies","an","end-to-end",",","text","string","to","wordpiece","tokenization",".","It","is","equivalent","to","`","BertTokenizer","`","for","most","common","scenarios","while","running","faster","and","supporting","TFLite",".","It","does","not","support","certain","special","settings","(","see","the","docs","below",")",".","See","`","WordpieceTokenizer","`","for","details","on","the","subword","tokenization",".","For","an","example","of","use",",","see","https",":","\/\/www.tensorflow.org\/text\/guide\/bert_preprocessing_guide","Attributes",":","vocab",":","(","optional",")","The","list","of","tokens","in","the","vocabulary",".","suffix_indicator",":","(","optional",")","The","characters","prepended","to","a","wordpiece","to","indicate","that","it","is","a","suffix","to","another","subword",".","max_bytes_per_word",":","(","optional",")","Max","size","of","input","token",".","token_out_type",":","(","optional",")","The","type","of","the","token","to","return",".","This","can","be","`","tf.int64","`","or","`","tf.int32","`","IDs",",","or","`","tf.string","`","subwords",".","unknown_token",":","(","optional",")","The","string","value","to","substitute","for","an","unknown","token",".","It","must","be","included","in","`","vocab","`",".","no_pretokenization",":","(","optional",")","By","default",",","the","input","is","split","on","whitespaces","and","punctuations","before","applying","the","Wordpiece","tokenization",".","When","true",",","the","input","is","assumed","to","be","pretokenized","already",".","support_detokenization",":","(","optional",")","Whether","to","make","the","tokenizer","support","doing","detokenization",".","Setting","it","to","true","expands","the","size","of","the","model","flatbuffer",".","As","a","reference",",","when","using","120k","multilingual","BERT","WordPiece","vocab",",","the","flatbuffer","'s","size","increases","from","~5MB","to","~6MB",".","fast_wordpiece_model_buffer",":","(","optional",")","Bytes","object","(","or","a","uint8","tf.Tenosr",")","that","contains","the","wordpiece","model","in","flatbuffer","format","(","see","fast_wordpiece_tokenizer_model.fbs",")",".","If","not","`","None","`",",","all","other","arguments","related","to","FastWordPieceTokenizer","(","except","`","token_output_type","`",")","are","ignored",".","lower_case_nfd_strip_accents",":","(","optional",")",".","-","If","true",",","it","first","lowercases","the","text",",","applies","NFD","normalization",",","strips","accents","characters",",","and","then","replaces","control","characters","with","whitespaces",".","-","If","false",",","it","only","replaces","control","characters","with","whitespaces",".","fast_bert_normalizer_model_buffer",":","(","optional",")","bytes","object","(","or","a","uint8","tf.Tenosr",")","that","contains","the","fast","bert","normalizer","model","in","flatbuffer","format","(","see","fast_bert_normalizer_model.fbs",")",".","If","not","`","None","`",",","`","lower_case_nfd_strip_accents","`","is","ignored","."]
docstr: Convert a `Tensor` or `RaggedTensor` of wordpiece IDs to string-words.
  
  See `WordpieceTokenizer.detokenize` for details.
  
  Note: `FastBertTokenizer.tokenize`/`FastBertTokenizer.detokenize` does not
  round trip losslessly. The result of `detokenize` will not, in general, have
  the same content or offsets as the input to `tokenize`. This is because the
  "basic tokenization" step, that splits the strings into words before
  applying the `WordpieceTokenizer`, includes irreversible steps like
  lower-casing and splitting on punctuation. `WordpieceTokenizer` on the other
  hand **is** reversible.
  
  Note: This method assumes wordpiece IDs are dense on the interval
  `[0, vocab_size)`.
  
  Example:
  >>> vocab = ['they', "##'", '##re', 'the', 'great', '##est', '[UNK]']
  >>> tokenizer = FastBertTokenizer(vocab=vocab, support_detokenization=True)
  >>> tokenizer.detokenize([[4, 5]])
  <tf.Tensor: shape=(1,), dtype=string, numpy=array([b'greatest'],
  dtype=object)>
  
  Args:
    token_ids: A `RaggedTensor` or `Tensor` with an int dtype.
  
  Returns:
    A `RaggedTensor` with dtype `string` and the same rank as the input
    `token_ids`.
docstr_tok: ["Convert","a","`","Tensor","`","or","`","RaggedTensor","`","of","wordpiece","IDs","to","string-words",".","See","`","WordpieceTokenizer.detokenize","`","for","details",".","Note",":","`","FastBertTokenizer.tokenize","`","\/","`","FastBertTokenizer.detokenize","`","does","not","round","trip","losslessly",".","The","result","of","`","detokenize","`","will","not",",","in","general",",","have","the","same","content","or","offsets","as","the","input","to","`","tokenize","`",".","This","is","because","the","''","basic","tokenization","''","step",",","that","splits","the","strings","into","words","before","applying","the","`","WordpieceTokenizer","`",",","includes","irreversible","steps","like","lower-casing","and","splitting","on","punctuation",".","`","WordpieceTokenizer","`","on","the","other","hand","*","*","is","*","*","reversible",".","Note",":","This","method","assumes","wordpiece","IDs","are","dense","on","the","interval","`","[","0",",","vocab_size",")","`",".","Example",":",">",">",">","vocab","=","[","'they","'",",","``","#","#","'","''",",","'","#","#","re","'",",","'the","'",",","'great","'",",","'","#","#","est","'",",","'","[","UNK","]","'","]",">",">",">","tokenizer","=","FastBertTokenizer","(","vocab=vocab",",","support_detokenization=True",")",">",">",">","tokenizer.detokenize","(","[","[","4",",","5","]","]",")","<","tf.Tensor",":","shape=","(","1",",",")",",","dtype=string",",","numpy=array","(","[","b'greatest","'","]",",","dtype=object",")",">","Args",":","token_ids",":","A","`","RaggedTensor","`","or","`","Tensor","`","with","an","int","dtype",".","Returns",":","A","`","RaggedTensor","`","with","dtype","`","string","`","and","the","same","rank","as","the","input","`","token_ids","`","."]
code: def detokenize(self, token_ids):
      r"""Convert a `Tensor` or `RaggedTensor` of wordpiece IDs to string-words.
  
      See `WordpieceTokenizer.detokenize` for details.
  
      Note: `FastBertTokenizer.tokenize`/`FastBertTokenizer.detokenize` does not
      round trip losslessly. The result of `detokenize` will not, in general, have
      the same content or offsets as the input to `tokenize`. This is because the
      "basic tokenization" step, that splits the strings into words before
      applying the `WordpieceTokenizer`, includes irreversible steps like
      lower-casing and splitting on punctuation. `WordpieceTokenizer` on the other
      hand **is** reversible.
  
      Note: This method assumes wordpiece IDs are dense on the interval
      `[0, vocab_size)`.
  
      Example:
      >>> vocab = ['they', "##'", '##re', 'the', 'great', '##est', '[UNK]']
      >>> tokenizer = FastBertTokenizer(vocab=vocab, support_detokenization=True)
      >>> tokenizer.detokenize([[4, 5]])
      <tf.Tensor: shape=(1,), dtype=string, numpy=array([b'greatest'],
      dtype=object)>
  
      Args:
        token_ids: A `RaggedTensor` or `Tensor` with an int dtype.
  
      Returns:
        A `RaggedTensor` with dtype `string` and the same rank as the input
        `token_ids`.
      """
      return self._fast_wordpiece_tokenizer.detokenize(token_ids)
  
file_path: text/tensorflow_text/python/ops/fast_bert_tokenizer.py
imports: from __future__ import absolute_import
  from __future__ import division
  from __future__ import print_function
  from tensorflow.python.eager import monitoring
  from tensorflow.python.framework import dtypes
  from tensorflow.python.ops import array_ops
  from tensorflow_text.python.ops.fast_bert_normalizer import FastBertNormalizer
  from tensorflow_text.python.ops.fast_wordpiece_tokenizer import FastWordpieceTokenizer
  from tensorflow_text.python.ops.tokenization import Detokenizer
  from tensorflow_text.python.ops.tokenization import TokenizerWithOffsets
label_desc: Use image node_id 4 for calling the FastBertTokenizer obj's underlying member method code with example usage: obj.detokenize(token_ids) and returns: self