class_docstr: Base class for tokenizer implementations.
  
  A Tokenizer is a `text.Splitter` that splits strings into *tokens*.  Tokens
  generally correspond to short substrings of the source string.  Tokens can be
  encoded using either strings or integer ids (where integer ids could be
  created by hashing strings or by looking them up in a fixed vocabulary table
  that maps strings to ids).
  
  Each Tokenizer subclass must implement a `tokenize` method, which splits each
  string in a Tensor into tokens. E.g.:
  
  >>> class SimpleTokenizer(tf_text.Tokenizer):
  ...   def tokenize(self, input):
  ...     return tf.strings.split(input)
  >>> print(SimpleTokenizer().tokenize(["hello world", "this is a test"]))
  <tf.RaggedTensor [[b'hello', b'world'], [b'this', b'is', b'a', b'test']]>
  
  By default, the `split` method simply delegates to `tokenize`.
class_docstr_tok: ["Base","class","for","tokenizer","implementations",".","A","Tokenizer","is","a","`","text.Splitter","`","that","splits","strings","into","*","tokens","*",".","Tokens","generally","correspond","to","short","substrings","of","the","source","string",".","Tokens","can","be","encoded","using","either","strings","or","integer","ids","(","where","integer","ids","could","be","created","by","hashing","strings","or","by","looking","them","up","in","a","fixed","vocabulary","table","that","maps","strings","to","ids",")",".","Each","Tokenizer","subclass","must","implement","a","`","tokenize","`","method",",","which","splits","each","string","in","a","Tensor","into","tokens",".","E.g",".",":",">",">",">","class","SimpleTokenizer","(","tf_text.Tokenizer",")",":","...","def","tokenize","(","self",",","input",")",":","...","return","tf.strings.split","(","input",")",">",">",">","print","(","SimpleTokenizer","(",")",".tokenize","(","[","``","hello","world","''",",","``","this","is","a","test","''","]",")",")","<","tf.RaggedTensor","[","[","b'hello","'",",","b'world","'","]",",","[","b'this","'",",","b'is","'",",","b","'","a","'",",","b'test","'","]","]",">","By","default",",","the","`","split","`","method","simply","delegates","to","`","tokenize","`","."]
docstr: Alias for `Tokenizer.tokenize`.
docstr_tok: ["Alias","for","`","Tokenizer.tokenize","`","."]
code: def split(self, input):  # pylint: disable=redefined-builtin
      """Alias for `Tokenizer.tokenize`."""
      return self.tokenize(input)
  
file_path: text/tensorflow_text/python/ops/tokenization.py
imports: from __future__ import absolute_import
  from __future__ import division
  from __future__ import print_function
  import abc
  from tensorflow.python.module import module
  from tensorflow_text.python.ops.splitter import Splitter
  from tensorflow_text.python.ops.splitter import SplitterWithOffsets
label_desc: Use image node_id 2 for calling the Tokenizer obj's underlying member method code with example usage: obj.split(input) and returns: self