class_docstr: Tokenizes UTF-8 by splitting when there is a change in Unicode script.
  
  By default, this tokenizer leaves out scripts matching the whitespace unicode
  property (use the `keep_whitespace` argument to keep it), so in this case the
  results are similar to the `WhitespaceTokenizer`. Any punctuation
  will get its own token (since it is in a different script), and any script
  change in the input string will be the location of a split.
  
  Example:
  >>> tokenizer = tf_text.UnicodeScriptTokenizer()
  >>> tokens = tokenizer.tokenize(["xy.,z de", "fg?h", "abαβ"])
  >>> print(tokens.to_list())
  [[b'xy', b'.,', b'z', b'de'], [b'fg', b'?', b'h'],
   [b'ab', b'\xce\xb1\xce\xb2']]
  
  >>> tokens = tokenizer.tokenize(u"累計7239人")
  >>> print(tokens)
  tf.Tensor([b'\xe7\xb4\xaf\xe8\xa8\x88' b'7239' b'\xe4\xba\xba'], shape=(3,),
            dtype=string)
  
  Both the punctuation and the whitespace in the first string have been split,
  but the punctuation run is present as a token while the whitespace isn't
  emitted (by default). The third example shows the case of a script change
  without any whitespace. This results in a split at that boundary point.
class_docstr_tok: ["Tokenizes","UTF-8","by","splitting","when","there","is","a","change","in","Unicode","script",".","By","default",",","this","tokenizer","leaves","out","scripts","matching","the","whitespace","unicode","property","(","use","the","`","keep_whitespace","`","argument","to","keep","it",")",",","so","in","this","case","the","results","are","similar","to","the","`","WhitespaceTokenizer","`",".","Any","punctuation","will","get","its","own","token","(","since","it","is","in","a","different","script",")",",","and","any","script","change","in","the","input","string","will","be","the","location","of","a","split",".","Example",":",">",">",">","tokenizer","=","tf_text.UnicodeScriptTokenizer","(",")",">",">",">","tokens","=","tokenizer.tokenize","(","[","``","xy.",",","z","de","''",",","``","fg","?","h","''",",","``","ab\u03b1\u03b2","''","]",")",">",">",">","print","(","tokens.to_list","(",")",")","[","[","b'xy","'",",","b'.",",","'",",","b","'","z","'",",","b'de","'","]",",","[","b'fg","'",",","b","'","?","'",",","b","'","h","'","]",",","[","b'ab","'",",","b'\\xce\\xb1\\xce\\xb2","'","]","]",">",">",">","tokens","=","tokenizer.tokenize","(","u","''","\u7d2f\u8a087239\u4eba","''",")",">",">",">","print","(","tokens",")","tf.Tensor","(","[","b'\\xe7\\xb4\\xaf\\xe8\\xa8\\x88","'","b'7239","'","b'\\xe4\\xba\\xba","'","]",",","shape=","(","3",",",")",",","dtype=string",")","Both","the","punctuation","and","the","whitespace","in","the","first","string","have","been","split",",","but","the","punctuation","run","is","present","as","a","token","while","the","whitespace","isn't","emitted","(","by","default",")",".","The","third","example","shows","the","case","of","a","script","change","without","any","whitespace",".","This","results","in","a","split","at","that","boundary","point","."]
docstr: Tokenizes UTF-8 by splitting when there is a change in Unicode script.
  
  The strings are split when a change in the Unicode script is detected
  between sequential tokens. The script codes used correspond to International
  Components for Unicode (ICU) UScriptCode values. See:
  http://icu-project.org/apiref/icu4c/uscript_8h.html
  
  ICU defined whitespace characters are dropped, unless the keep_whitespace
  option was specified at construction time.
  
  Example:
  >>> tokenizer = tf_text.UnicodeScriptTokenizer()
  >>> tokens = tokenizer.tokenize_with_offsets(["xy.,z de", "abαβ"])
  >>> print(tokens[0].to_list())
  [[b'xy', b'.,', b'z', b'de'], [b'ab', b'\xce\xb1\xce\xb2']]
  >>> print(tokens[1].to_list())
  [[0, 2, 4, 6], [0, 2]]
  >>> print(tokens[2].to_list())
  [[2, 4, 5, 8], [2, 6]]
  
  >>> tokens = tokenizer.tokenize_with_offsets(u"累計7239人")
  >>> print(tokens[0])
  tf.Tensor([b'\xe7\xb4\xaf\xe8\xa8\x88' b'7239' b'\xe4\xba\xba'],
      shape=(3,), dtype=string)
  >>> print(tokens[1])
  tf.Tensor([ 0  6 10], shape=(3,), dtype=int64)
  >>> print(tokens[2])
  tf.Tensor([ 6 10 13], shape=(3,), dtype=int64)
  
  The start_offsets and end_offsets are in byte indices of the original
  string. When calling with multiple string inputs, the offset indices will
  be relative to the individual source strings.
  
  Args:
    input: A `RaggedTensor`or `Tensor` of UTF-8 strings with any shape.
  
  Returns:
    A tuple `(tokens, start_offsets, end_offsets)` where:
  
      * `tokens`: A `RaggedTensor` of tokenized text.
      * `start_offsets`: A `RaggedTensor` of the tokens' starting byte offset.
      * `end_offsets`: A `RaggedTensor` of the tokens' ending byte offset.
docstr_tok: ["Tokenizes","UTF-8","by","splitting","when","there","is","a","change","in","Unicode","script",".","The","strings","are","split","when","a","change","in","the","Unicode","script","is","detected","between","sequential","tokens",".","The","script","codes","used","correspond","to","International","Components","for","Unicode","(","ICU",")","UScriptCode","values",".","See",":","http",":","\/\/icu-project.org\/apiref\/icu4c\/uscript_8h.html","ICU","defined","whitespace","characters","are","dropped",",","unless","the","keep_whitespace","option","was","specified","at","construction","time",".","Example",":",">",">",">","tokenizer","=","tf_text.UnicodeScriptTokenizer","(",")",">",">",">","tokens","=","tokenizer.tokenize_with_offsets","(","[","``","xy.",",","z","de","''",",","``","ab\u03b1\u03b2","''","]",")",">",">",">","print","(","tokens","[","0","]",".to_list","(",")",")","[","[","b'xy","'",",","b'.",",","'",",","b","'","z","'",",","b'de","'","]",",","[","b'ab","'",",","b'\\xce\\xb1\\xce\\xb2","'","]","]",">",">",">","print","(","tokens","[","1","]",".to_list","(",")",")","[","[","0",",","2",",","4",",","6","]",",","[","0",",","2","]","]",">",">",">","print","(","tokens","[","2","]",".to_list","(",")",")","[","[","2",",","4",",","5",",","8","]",",","[","2",",","6","]","]",">",">",">","tokens","=","tokenizer.tokenize_with_offsets","(","u","''","\u7d2f\u8a087239\u4eba","''",")",">",">",">","print","(","tokens","[","0","]",")","tf.Tensor","(","[","b'\\xe7\\xb4\\xaf\\xe8\\xa8\\x88","'","b'7239","'","b'\\xe4\\xba\\xba","'","]",",","shape=","(","3",",",")",",","dtype=string",")",">",">",">","print","(","tokens","[","1","]",")","tf.Tensor","(","[","0","6","10","]",",","shape=","(","3",",",")",",","dtype=int64",")",">",">",">","print","(","tokens","[","2","]",")","tf.Tensor","(","[","6","10","13","]",",","shape=","(","3",",",")",",","dtype=int64",")","The","start_offsets","and","end_offsets","are","in","byte","indices","of","the","original","string",".","When","calling","with","multiple","string","inputs",",","the","offset","indices","will","be","relative","to","the","individual","source","strings",".","Args",":","input",":","A","`","RaggedTensor","`","or","`","Tensor","`","of","UTF-8","strings","with","any","shape",".","Returns",":","A","tuple","`","(","tokens",",","start_offsets",",","end_offsets",")","`","where",":","*","`","tokens","`",":","A","`","RaggedTensor","`","of","tokenized","text",".","*","`","start_offsets","`",":","A","`","RaggedTensor","`","of","the","tokens","'","starting","byte","offset",".","*","`","end_offsets","`",":","A","`","RaggedTensor","`","of","the","tokens","'","ending","byte","offset","."]
code: def tokenize_with_offsets(
      self, input
  ):  # pylint: disable=redefined-builtin
      r"""Tokenizes UTF-8 by splitting when there is a change in Unicode script.
  
      The strings are split when a change in the Unicode script is detected
      between sequential tokens. The script codes used correspond to International
      Components for Unicode (ICU) UScriptCode values. See:
      http://icu-project.org/apiref/icu4c/uscript_8h.html
  
      ICU defined whitespace characters are dropped, unless the keep_whitespace
      option was specified at construction time.
  
      Example:
      >>> tokenizer = tf_text.UnicodeScriptTokenizer()
      >>> tokens = tokenizer.tokenize_with_offsets(["xy.,z de", "abαβ"])
      >>> print(tokens[0].to_list())
      [[b'xy', b'.,', b'z', b'de'], [b'ab', b'\xce\xb1\xce\xb2']]
      >>> print(tokens[1].to_list())
      [[0, 2, 4, 6], [0, 2]]
      >>> print(tokens[2].to_list())
      [[2, 4, 5, 8], [2, 6]]
  
      >>> tokens = tokenizer.tokenize_with_offsets(u"累計7239人")
      >>> print(tokens[0])
      tf.Tensor([b'\xe7\xb4\xaf\xe8\xa8\x88' b'7239' b'\xe4\xba\xba'],
          shape=(3,), dtype=string)
      >>> print(tokens[1])
      tf.Tensor([ 0  6 10], shape=(3,), dtype=int64)
      >>> print(tokens[2])
      tf.Tensor([ 6 10 13], shape=(3,), dtype=int64)
  
      The start_offsets and end_offsets are in byte indices of the original
      string. When calling with multiple string inputs, the offset indices will
      be relative to the individual source strings.
  
      Args:
        input: A `RaggedTensor`or `Tensor` of UTF-8 strings with any shape.
  
      Returns:
        A tuple `(tokens, start_offsets, end_offsets)` where:
  
          * `tokens`: A `RaggedTensor` of tokenized text.
          * `start_offsets`: A `RaggedTensor` of the tokens' starting byte offset.
          * `end_offsets`: A `RaggedTensor` of the tokens' ending byte offset.
      """
      name = None
      with ops.name_scope(name, "UnicodeScriptTokenize", [input]):
          input_tensor = (
              ragged_tensor.convert_to_tensor_or_ragged_tensor(input)
          )
          if input_tensor.shape.ndims is None:
              raise ValueError(
                  "Rank of input_tensor must be statically known."
              )
          if ragged_tensor.is_ragged(input_tensor):
              if input_tensor.flat_values.shape.ndims > 1:
                  # If the flat_values of our ragged tensor is multi-dimensional, we can
                  # process it separately and our output will have the same nested
                  # splits as our input.
                  (tokens, starts, ends) = self.tokenize_with_offsets(
                      input_tensor.flat_values
                  )
                  return (
                      input_tensor.with_flat_values(tokens),
                      input_tensor.with_flat_values(starts),
                      input_tensor.with_flat_values(ends),
                  )
              else:
                  # Recursively process the values of the ragged tensor.
                  (tokens, starts, ends) = self.tokenize_with_offsets(
                      input_tensor.values
                  )
                  return (
                      input_tensor.with_values(tokens),
                      input_tensor.with_values(starts),
                      input_tensor.with_values(ends),
                  )
          else:
              if input_tensor.shape.ndims > 1:
                  # Convert the input tensor to ragged and process it.
                  return self.tokenize_with_offsets(
                      ragged_conversion_ops.from_tensor(input_tensor)
                  )
              elif input_tensor.shape.ndims == 0:
                  (tokens, starts, ends) = self.tokenize_with_offsets(
                      array_ops_stack.stack([input_tensor])
                  )
                  return tokens.values, starts.values, ends.values
              else:
                  # Our rank 1 tensor is the correct shape, so we can process it as
                  # normal
                  return (
                      self._tokenize_with_offsets_encode_decode_wrapper(
                          input_tensor
                      )
                  )
  
imports: from __future__ import absolute_import
  from __future__ import division
  from __future__ import print_function
  from tensorflow.python.eager import monitoring
  from tensorflow.python.framework import dtypes
  from tensorflow.python.framework import ops
  from tensorflow.python.ops import array_ops
  from tensorflow.python.ops import array_ops_stack
  from tensorflow.python.ops import math_ops
  from tensorflow.python.ops import string_ops
  from tensorflow.python.ops.ragged import ragged_conversion_ops
  from tensorflow.python.ops.ragged import ragged_string_ops
  from tensorflow.python.ops.ragged import ragged_tensor
  from tensorflow.python.ops.ragged.ragged_tensor import RaggedTensor
  from tensorflow_text.python.ops.tokenization import TokenizerWithOffsets
  from tensorflow.python.framework import load_library
  from tensorflow.python.platform import resource_loader
label_desc: Use image node_id 3 for calling the UnicodeScriptTokenizer obj's underlying member method code with example usage: obj.tokenize_with_offsets(input) and returns: input_tensor,  input_tensor,  input_tensor, input_tensor,  input_tensor,  input_tensor, self, tokens,  starts,  ends, self