class_docstr: Wrapper around `SentencePieceTokenizer` to keep backwards compatibility.
  
  The current DMVR SentencePiece tokenizer always sets `prepend_bos` when
  initializing the tensorflow processor and returns a sliced tensor when called
  with `prepend_bos=False`. This is problematic when the sentencepiece model is
  not trained with the BOS token (predefined and hard-coded as '<S>'), which is
  the case for T5 tokenizer. This module contains a wrapper for the DMVR
  SentencePiece tokenizer to initialize the tensorflow processor without
  prepending BOS. Instead, it prepends a custom BOS token given as an argument.
code: def string_tensor_to_indices(
      self,
      string_tensor: Union[tf.Tensor, Sequence[str]],
      prepend_bos: bool = False,
      append_eos: bool = False,
      max_num_tokens: Optional[int] = 32,
  ) -> tf.Tensor:
      if self._tf_sp_model is None:
          raise RuntimeError(
              "Model was not initialized. Call `initialize` method."
          )
  
      tokenized = self._tf_sp_model.tokenize(string_tensor)
      tokenized = tokenized if append_eos else tokenized[..., :-1]
  
      # Pad to `max_num_tokens`.
      shape = None if max_num_tokens is None else [None, max_num_tokens]
      tokenized = tokenized.to_tensor(
          default_value=self._pad_token, shape=shape
      )
  
      if prepend_bos:
          tokenized = tf.concat(
              [
                  tf.zeros_like(tokenized[..., 0:1]) + self.bos_id,
                  tokenized[..., :-1],
              ],
              -1,
          )
      return tokenized
  
file_path: scenic/scenic/projects/t5/tokenizer.py
imports: from collections.abc import Sequence
  from typing import Optional
  from typing import Union
  from dmvr import tokenizers
  import tensorflow
  import tensorflow_text
label_desc: Use image node_id 3 for calling the SentencePieceTokenizer obj's underlying member method code with example usage: obj.string_tensor_to_indices(string_tensor, prepend_bos, append_eos, max_num_tokens) and returns: tokenized