class_docstr: Tokenizes a tensor of UTF-8 string tokens into subword pieces.
  
  It employs the linear (as opposed to quadratic) WordPiece algorithm (see the
  [paper](http://go/arxiv/2012.15524)).
  
  Differences compared to the classic
  [WordpieceTokenizer](https://www.tensorflow.org/text/api_docs/python/text/WordpieceTokenizer)
  are as follows (as of 11/2021):
  
    * `unknown_token` cannot be None or empty. That means if a word is too long
      or cannot be tokenized, FastWordpieceTokenizer always returns
      `unknown_token`. In constrast, the original
      [WordpieceTokenizer](https://www.tensorflow.org/text/api_docs/python/text/WordpieceTokenizer)
      would return the original word if `unknown_token` is empty or None.
  
    * `unknown_token` must be included in the vocabulary.
  
    * When `unknown_token` is returned, in tokenize_with_offsets(), the result
      end_offset is set to be the length of the original input word. In
      contrast, when `unknown_token` is returned by the original
      [WordpieceTokenizer](https://www.tensorflow.org/text/api_docs/python/text/WordpieceTokenizer),
      the end_offset is set to be the length of the `unknown_token` string.
  
    * `split_unknown_characters` is not supported.
  
    * `max_chars_per_token` is not used or needed.
  
    * By default the input is assumed to be general text (i.e., sentences), and
      FastWordpieceTokenizer first splits it on whitespaces and punctuations and
      then applies the Wordpiece tokenization (see the parameter
      `no_pretokenization`). If the input already contains single words only,
      please set `no_pretokenization=True` to be consistent with the classic
      [WordpieceTokenizer](https://www.tensorflow.org/text/api_docs/python/text/WordpieceTokenizer).
class_docstr_tok: ["Tokenizes","a","tensor","of","UTF-8","string","tokens","into","subword","pieces",".","It","employs","the","linear","(","as","opposed","to","quadratic",")","WordPiece","algorithm","(","see","the","[","paper","]","(","http",":","\/\/go\/arxiv\/2012.15524",")",")",".","Differences","compared","to","the","classic","[","WordpieceTokenizer","]","(","https",":","\/\/www.tensorflow.org\/text\/api_docs\/python\/text\/WordpieceTokenizer",")","are","as","follows","(","as","of","11\/2021",")",":","*","`","unknown_token","`","can","not","be","None","or","empty",".","That","means","if","a","word","is","too","long","or","can","not","be","tokenized",",","FastWordpieceTokenizer","always","returns","`","unknown_token","`",".","In","constrast",",","the","original","[","WordpieceTokenizer","]","(","https",":","\/\/www.tensorflow.org\/text\/api_docs\/python\/text\/WordpieceTokenizer",")","would","return","the","original","word","if","`","unknown_token","`","is","empty","or","None",".","*","`","unknown_token","`","must","be","included","in","the","vocabulary",".","*","When","`","unknown_token","`","is","returned",",","in","tokenize_with_offsets","(",")",",","the","result","end_offset","is","set","to","be","the","length","of","the","original","input","word",".","In","contrast",",","when","`","unknown_token","`","is","returned","by","the","original","[","WordpieceTokenizer","]","(","https",":","\/\/www.tensorflow.org\/text\/api_docs\/python\/text\/WordpieceTokenizer",")",",","the","end_offset","is","set","to","be","the","length","of","the","`","unknown_token","`","string",".","*","`","split_unknown_characters","`","is","not","supported",".","*","`","max_chars_per_token","`","is","not","used","or","needed",".","*","By","default","the","input","is","assumed","to","be","general","text","(","i.e.",",","sentences",")",",","and","FastWordpieceTokenizer","first","splits","it","on","whitespaces","and","punctuations","and","then","applies","the","Wordpiece","tokenization","(","see","the","parameter","`","no_pretokenization","`",")",".","If","the","input","already","contains","single","words","only",",","please","set","`","no_pretokenization=True","`","to","be","consistent","with","the","classic","[","WordpieceTokenizer","]","(","https",":","\/\/www.tensorflow.org\/text\/api_docs\/python\/text\/WordpieceTokenizer",")","."]
docstr: Detokenizes a tensor of int64 or int32 subword ids into sentences.
  
  Detokenize and tokenize an input string returns itself when the input string
  is normalized and the tokenized wordpieces don't contain `<unk>`.
  
  ### Example:
  >>> vocab = ["they", "##'", "##re", "the", "great", "##est", "[UNK]",
  ...          "'", "re", "ok"]
  >>> tokenizer = FastWordpieceTokenizer(vocab, support_detokenization=True)
  >>> ids = tf.ragged.constant([[0, 1, 2, 3, 4, 5], [9]])
  >>> tokenizer.detokenize(ids)
  <tf.Tensor: shape=(2,), dtype=string,
  ...         numpy=array([b"they're the greatest", b'ok'], dtype=object)>
  >>> ragged_ids = tf.ragged.constant([[[0, 1, 2, 3, 4, 5], [9]], [[4, 5]]])
  >>> tokenizer.detokenize(ragged_ids)
  <tf.RaggedTensor [[b"they're the greatest", b'ok'], [b'greatest']]>
  
  Args:
    input: An N-dimensional `Tensor` or `RaggedTensor` of int64 or int32.
  
  Returns:
    A `RaggedTensor` of sentences that has N - 1 dimension when N > 1.
    Otherwise, a string tensor.
docstr_tok: ["Detokenizes","a","tensor","of","int64","or","int32","subword","ids","into","sentences",".","Detokenize","and","tokenize","an","input","string","returns","itself","when","the","input","string","is","normalized","and","the","tokenized","wordpieces","do","n't","contain","`","<","unk",">","`",".","#","#","#","Example",":",">",">",">","vocab","=","[","``","they","''",",","``","#","#","'","''",",","``","#","#","re","''",",","``","the","''",",","``","great","''",",","``","#","#","est","''",",","``","[","UNK","]","''",",","...","``","'","''",",","``","re","''",",","``","ok","''","]",">",">",">","tokenizer","=","FastWordpieceTokenizer","(","vocab",",","support_detokenization=True",")",">",">",">","ids","=","tf.ragged.constant","(","[","[","0",",","1",",","2",",","3",",","4",",","5","]",",","[","9","]","]",")",">",">",">","tokenizer.detokenize","(","ids",")","<","tf.Tensor",":","shape=","(","2",",",")",",","dtype=string",",","...","numpy=array","(","[","b","''","they","'re","the","greatest","''",",","b'ok","'","]",",","dtype=object",")",">",">",">",">","ragged_ids","=","tf.ragged.constant","(","[","[","[","0",",","1",",","2",",","3",",","4",",","5","]",",","[","9","]","]",",","[","[","4",",","5","]","]","]",")",">",">",">","tokenizer.detokenize","(","ragged_ids",")","<","tf.RaggedTensor","[","[","b","''","they","'re","the","greatest","''",",","b'ok","'","]",",","[","b'greatest","'","]","]",">","Args",":","input",":","An","N-dimensional","`","Tensor","`","or","`","RaggedTensor","`","of","int64","or","int32",".","Returns",":","A","`","RaggedTensor","`","of","sentences","that","has","N","-","1","dimension","when","N",">","1",".","Otherwise",",","a","string","tensor","."]
code: def detokenize(self, input):  # pylint: disable=redefined-builtin
      """Detokenizes a tensor of int64 or int32 subword ids into sentences.
  
      Detokenize and tokenize an input string returns itself when the input string
      is normalized and the tokenized wordpieces don't contain `<unk>`.
  
      ### Example:
      >>> vocab = ["they", "##'", "##re", "the", "great", "##est", "[UNK]",
      ...          "'", "re", "ok"]
      >>> tokenizer = FastWordpieceTokenizer(vocab, support_detokenization=True)
      >>> ids = tf.ragged.constant([[0, 1, 2, 3, 4, 5], [9]])
      >>> tokenizer.detokenize(ids)
      <tf.Tensor: shape=(2,), dtype=string,
      ...         numpy=array([b"they're the greatest", b'ok'], dtype=object)>
      >>> ragged_ids = tf.ragged.constant([[[0, 1, 2, 3, 4, 5], [9]], [[4, 5]]])
      >>> tokenizer.detokenize(ragged_ids)
      <tf.RaggedTensor [[b"they're the greatest", b'ok'], [b'greatest']]>
  
      Args:
        input: An N-dimensional `Tensor` or `RaggedTensor` of int64 or int32.
  
      Returns:
        A `RaggedTensor` of sentences that has N - 1 dimension when N > 1.
        Otherwise, a string tensor.
      """
      name = None
      with ops.name_scope(
          name, "FastWordpieceDetokenize", [input, self._model]
      ):
          # Check that the types are expected and the ragged rank is appropriate.
          subword_ids = (
              ragged_tensor.convert_to_tensor_or_ragged_tensor(input)
          )
          subword_ids = math_ops.cast(subword_ids, dtypes.int32)
          rank = subword_ids.shape.ndims
          if rank is None:
              raise ValueError("input must have a known rank.")
  
          if rank < 2:
              words = self.detokenize(
                  array_ops_stack.stack([subword_ids])
              )
              return words[0]
  
          if not ragged_tensor.is_ragged(subword_ids):
              subword_ids = ragged_tensor.RaggedTensor.from_tensor(
                  subword_ids, ragged_rank=rank - 1
              )
          nested_row_splits = subword_ids.nested_row_splits
          # Detokenize the wordpiece ids to texts.
          words = gen_fast_wordpiece_tokenizer.tf_text_fast_wordpiece_detokenize(
              input_values=subword_ids.flat_values,
              input_row_splits=nested_row_splits[-1],
              wp_model=self._model,
          )
          words = RaggedTensor.from_nested_row_splits(
              words, nested_row_splits[:-1], validate=False
          )
  
          return words
  
imports: from __future__ import absolute_import
  from __future__ import division
  from __future__ import print_function
  from tensorflow.python.eager import monitoring
  from tensorflow.python.framework import constant_op
  from tensorflow.python.framework import dtypes
  from tensorflow.python.framework import ops
  from tensorflow.python.framework import tensor
  from tensorflow.python.ops import array_ops_stack
  from tensorflow.python.ops import math_ops
  from tensorflow.python.ops.ragged import ragged_tensor
  from tensorflow.python.ops.ragged.ragged_tensor import RaggedTensor
  from tensorflow_text.core.pybinds import pywrap_fast_wordpiece_tokenizer_model_builder
  from tensorflow_text.python.ops.tokenization import Detokenizer
  from tensorflow_text.python.ops.tokenization import TokenizerWithOffsets
  from tensorflow.python.framework import load_library
  from tensorflow.python.platform import resource_loader
label_desc: Use image node_id 4 for calling the FastWordpieceTokenizer obj's underlying member method code with example usage: obj.detokenize(input) and returns: words, words