class_docstr: Base class for tokenizer implementations that return offsets.
  
  The offsets indicate which substring from the input string was used to
  generate each token.  E.g., if `input` is a single string, then each token
  `token[i]` was generated from the substring `input[starts[i]:ends[i]]`.
  
  Each TokenizerWithOffsets subclass must implement the `tokenize_with_offsets`
  method, which returns a tuple containing both the pieces and the start and
  end offsets where those pieces occurred in the input string.  I.e., if
  `tokens, starts, ends = tokenize_with_offsets(s)`, then each token `token[i]`
  corresponds with `tf.strings.substr(s, starts[i], ends[i] - starts[i])`.
  
  If the tokenizer encodes tokens as strings (and not token ids), then it will
  usually be the case that these corresponding strings are equal; but that is
  not technically required.  For example, a tokenizer might choose to downcase
  strings
  
  Example:
  
  >>> class CharTokenizer(TokenizerWithOffsets):
  ...   def tokenize_with_offsets(self, input):
  ...     chars, starts = tf.strings.unicode_split_with_offsets(input, 'UTF-8')
  ...     lengths = tf.expand_dims(tf.strings.length(input), -1)
  ...     ends = tf.concat([starts[..., 1:], tf.cast(lengths, tf.int64)], -1)
  ...     return chars, starts, ends
  ...   def tokenize(self, input):
  ...     return self.tokenize_with_offsets(input)[0]
  >>> pieces, starts, ends = CharTokenizer().split_with_offsets("a😊c")
  >>> print(pieces.numpy(), starts.numpy(), ends.numpy())
  [b'a' b'\xf0\x9f\x98\x8a' b'c'] [0 1 5] [1 5 6]
class_docstr_tok: ["Base","class","for","tokenizer","implementations","that","return","offsets",".","The","offsets","indicate","which","substring","from","the","input","string","was","used","to","generate","each","token",".","E.g.",",","if","`","input","`","is","a","single","string",",","then","each","token","`","token","[","i","]","`","was","generated","from","the","substring","`","input","[","starts","[","i","]",":","ends","[","i","]","]","`",".","Each","TokenizerWithOffsets","subclass","must","implement","the","`","tokenize_with_offsets","`","method",",","which","returns","a","tuple","containing","both","the","pieces","and","the","start","and","end","offsets","where","those","pieces","occurred","in","the","input","string",".","I.e.",",","if","`","tokens",",","starts",",","ends","=","tokenize_with_offsets","(","s",")","`",",","then","each","token","`","token","[","i","]","`","corresponds","with","`","tf.strings.substr","(","s",",","starts","[","i","]",",","ends","[","i","]","-","starts","[","i","]",")","`",".","If","the","tokenizer","encodes","tokens","as","strings","(","and","not","token","ids",")",",","then","it","will","usually","be","the","case","that","these","corresponding","strings","are","equal",";","but","that","is","not","technically","required",".","For","example",",","a","tokenizer","might","choose","to","downcase","strings","Example",":",">",">",">","class","CharTokenizer","(","TokenizerWithOffsets",")",":","...","def","tokenize_with_offsets","(","self",",","input",")",":","...","chars",",","starts","=","tf.strings.unicode_split_with_offsets","(","input",",","'UTF-8","'",")","...","lengths","=","tf.expand_dims","(","tf.strings.length","(","input",")",",","-1",")","...","ends","=","tf.concat","(","[","starts","[","...",",","1",":","]",",","tf.cast","(","lengths",",","tf.int64",")","]",",","-1",")","...","return","chars",",","starts",",","ends","...","def","tokenize","(","self",",","input",")",":","...","return","self.tokenize_with_offsets","(","input",")","[","0","]",">",">",">","pieces",",","starts",",","ends","=","CharTokenizer","(",")",".split_with_offsets","(","``","a\ud83d\ude0ac","''",")",">",">",">","print","(","pieces.numpy","(",")",",","starts.numpy","(",")",",","ends.numpy","(",")",")","[","b","'","a","'","b'\\xf0\\x9f\\x98\\x8a","'","b","'","c","'","]","[","0","1","5","]","[","1","5","6","]"]
docstr: Alias for `TokenizerWithOffsets.tokenize_with_offsets`.
docstr_tok: ["Alias","for","`","TokenizerWithOffsets.tokenize_with_offsets","`","."]
code: def split_with_offsets(
      self, input
  ):  # pylint: disable=redefined-builtin
      """Alias for `TokenizerWithOffsets.tokenize_with_offsets`."""
      return self.tokenize_with_offsets(input)
  
file_path: text/tensorflow_text/python/ops/tokenization.py
imports: from __future__ import absolute_import
  from __future__ import division
  from __future__ import print_function
  import abc
  from tensorflow.python.module import module
  from tensorflow_text.python.ops.splitter import Splitter
  from tensorflow_text.python.ops.splitter import SplitterWithOffsets
label_desc: Use image node_id 2 for calling the TokenizerWithOffsets obj's underlying member method code with example usage: obj.split_with_offsets(input) and returns: self