class_docstr: Sentencepiece tokenizer with tf.text interface.
docstr: The main tokenization function.
code: def tokenize(self, inputs):
      """The main tokenization function."""
      input_tensor = ragged_tensor.convert_to_tensor_or_ragged_tensor(
          inputs
      )
      if input_tensor.shape.ndims is None:
          raise ValueError(
              "Rank of input_tensor must be statically known."
          )
      if ragged_tensor.is_ragged(input_tensor):
          # Ensure that input has row_split_dtype is int32
          input_tensor = input_tensor.with_row_splits_dtype(tf.int32)
          # Recursively process the values of the ragged tensor.
          tokens = self.tokenize(input_tensor.flat_values)
          return input_tensor.with_flat_values(tokens)
      else:
          if input_tensor.shape.ndims > 1:
              # Convert the input tensor to ragged and process it.
              return self.tokenize(
                  tf.RaggedTensor.from_tensor(
                      input_tensor, row_splits_dtype=tf.int32
                  )
              )
          elif input_tensor.shape.ndims == 0:
              tokens = self.tokenize(tf.stack([input_tensor]))
              return tokens.values
          else:
              # Our rank 1 tensor is the correct shape, so we can process it as
              # normal.
              (
                  output_values,
                  row_splits,
              ) = gen_fast_sentencepiece_tokenizer.tf_text_fast_sentencepiece_tokenize(
                  self._converted_model,
                  input_tensor,
                  0,
                  0,
                  self._add_bos,
                  self._add_eos,
                  self._reverse,
              )
              tokens = tf.RaggedTensor.from_nested_row_splits(
                  flat_values=output_values,
                  nested_row_splits=[row_splits],
                  validate=False,
              )
              return tokens
  
file_path: text/tensorflow_text/python/ops/fast_sentencepiece_tokenizer.py
imports: import tensorflow.compat.v2
  from tensorflow.python.ops.ragged import ragged_tensor
  from tensorflow_text.core.pybinds import pywrap_model_converter
  from tensorflow.python.framework import load_library
  from tensorflow.python.platform import resource_loader
label_desc: Use image node_id 2 for calling the FastSentencepieceTokenizer obj's underlying member method code with example usage: obj.tokenize(inputs) and returns: input_tensor, self, tokens, tokens