class_docstr: Base class for detokenizer implementations.
  
  A Detokenizer is a module that combines tokens to form strings.  Generally,
  subclasses of `Detokenizer` will also be subclasses of `Tokenizer`; and the
  `detokenize` method will be the inverse of the `tokenize` method.  I.e.,
  `tokenizer.detokenize(tokenizer.tokenize(s)) == s`.
  
  Each Detokenizer subclass must implement a `detokenize` method, which combines
  tokens together to form strings.  E.g.:
  
  >>> class SimpleDetokenizer(tf_text.Detokenizer):
  ...   def detokenize(self, input):
  ...     return tf.strings.reduce_join(input, axis=-1, separator=" ")
  >>> text = tf.ragged.constant([["hello", "world"], ["a", "b", "c"]])
  >>> print(SimpleDetokenizer().detokenize(text))
  tf.Tensor([b'hello world' b'a b c'], shape=(2,), dtype=string)
class_docstr_tok: ["Base","class","for","detokenizer","implementations",".","A","Detokenizer","is","a","module","that","combines","tokens","to","form","strings",".","Generally",",","subclasses","of","`","Detokenizer","`","will","also","be","subclasses","of","`","Tokenizer","`",";","and","the","`","detokenize","`","method","will","be","the","inverse","of","the","`","tokenize","`","method",".","I.e.",",","`","tokenizer.detokenize","(","tokenizer.tokenize","(","s",")",")","==","s","`",".","Each","Detokenizer","subclass","must","implement","a","`","detokenize","`","method",",","which","combines","tokens","together","to","form","strings",".","E.g",".",":",">",">",">","class","SimpleDetokenizer","(","tf_text.Detokenizer",")",":","...","def","detokenize","(","self",",","input",")",":","...","return","tf.strings.reduce_join","(","input",",","axis=-1",",","separator=","''","``",")",">",">",">","text","=","tf.ragged.constant","(","[","[","``","hello","''",",","``","world","''","]",",","[","``","a","''",",","``","b","''",",","``","c","''","]","]",")",">",">",">","print","(","SimpleDetokenizer","(",")",".detokenize","(","text",")",")","tf.Tensor","(","[","b'hello","world","'","b","'","a","b","c","'","]",",","shape=","(","2",",",")",",","dtype=string",")"]
docstr: Assembles the tokens in the input tensor into a string.
  
  Generally, `detokenize` is the inverse of the `tokenize` method, and can
  be used to reconstrct a string from a set of tokens.  This is especially
  helpful in cases where the tokens are integer ids, such as indexes into a
  vocabulary table -- in that case, the tokenized encoding is not very
  human-readable (since it's just a list of integers), so the `detokenize`
  method can be used to turn it back into something that's more readable.
  
  Args:
    input: An N-dimensional UTF-8 string or integer `Tensor` or
      `RaggedTensor`.
  
  Returns:
    An (N-1)-dimensional UTF-8 string `Tensor` or `RaggedTensor`.
docstr_tok: ["Assembles","the","tokens","in","the","input","tensor","into","a","string",".","Generally",",","`","detokenize","`","is","the","inverse","of","the","`","tokenize","`","method",",","and","can","be","used","to","reconstrct","a","string","from","a","set","of","tokens",".","This","is","especially","helpful","in","cases","where","the","tokens","are","integer","ids",",","such","as","indexes","into","a","vocabulary","table","--","in","that","case",",","the","tokenized","encoding","is","not","very","human-readable","(","since","it","'s","just","a","list","of","integers",")",",","so","the","`","detokenize","`","method","can","be","used","to","turn","it","back","into","something","that","'s","more","readable",".","Args",":","input",":","An","N-dimensional","UTF-8","string","or","integer","`","Tensor","`","or","`","RaggedTensor","`",".","Returns",":","An","(","N-1",")","-dimensional","UTF-8","string","`","Tensor","`","or","`","RaggedTensor","`","."]
code: def detokenize(self, input):  # pylint: disable=redefined-builtin
      """Assembles the tokens in the input tensor into a string.
  
      Generally, `detokenize` is the inverse of the `tokenize` method, and can
      be used to reconstrct a string from a set of tokens.  This is especially
      helpful in cases where the tokens are integer ids, such as indexes into a
      vocabulary table -- in that case, the tokenized encoding is not very
      human-readable (since it's just a list of integers), so the `detokenize`
      method can be used to turn it back into something that's more readable.
  
      Args:
        input: An N-dimensional UTF-8 string or integer `Tensor` or
          `RaggedTensor`.
  
      Returns:
        An (N-1)-dimensional UTF-8 string `Tensor` or `RaggedTensor`.
      """
      raise NotImplementedError("Abstract method")
  
file_path: text/tensorflow_text/python/ops/tokenization.py
imports: from __future__ import absolute_import
  from __future__ import division
  from __future__ import print_function
  import abc
  from tensorflow.python.module import module
  from tensorflow_text.python.ops.splitter import Splitter
  from tensorflow_text.python.ops.splitter import SplitterWithOffsets
label_desc: Use image node_id 1 for calling the Detokenizer obj's underlying member method code with example usage: obj.detokenize(input) without return types