class_docstr: Splits a string into tokens, and joins them back.
class_docstr_tok: ["Splits","a","string","into","tokens",",","and","joins","them","back","."]
docstr: Constructs a Tokenizer.
  
  Note that the Tokenizer is invertible if `alphanum_only=False`.
  i.e. `s == t.join(t.tokenize(s))`.
  
  Args:
    alphanum_only: `bool`, if `True`, only parse out alphanumeric tokens
      (non-alphanumeric characters are dropped); otherwise, keep all
      characters (individual tokens will still be either all alphanumeric or
      all non-alphanumeric).
    reserved_tokens: `list<str>`, a list of strings that, if any are in `s`,
      will be preserved as whole tokens, even if they contain mixed
      alphanumeric/non-alphanumeric characters.
code: def __init__(self, alphanum_only=True, reserved_tokens=None):
      """Constructs a Tokenizer.
  
      Note that the Tokenizer is invertible if `alphanum_only=False`.
      i.e. `s == t.join(t.tokenize(s))`.
  
      Args:
        alphanum_only: `bool`, if `True`, only parse out alphanumeric tokens
          (non-alphanumeric characters are dropped); otherwise, keep all
          characters (individual tokens will still be either all alphanumeric or
          all non-alphanumeric).
        reserved_tokens: `list<str>`, a list of strings that, if any are in `s`,
          will be preserved as whole tokens, even if they contain mixed
          alphanumeric/non-alphanumeric characters.
      """
      self._alphanum_only = alphanum_only
      (
          reserved_tokens,
          self._reserved_tokens_re,
      ) = _prepare_reserved_tokens(reserved_tokens)
      self._reserved_tokens = set(reserved_tokens)
  
imports: from __future__ import unicode_literals
  import abc
  import hashlib
  import json
  import re
  from tensorflow_datasets.core.utils.lazy_imports_utils import tensorflow
label_desc: Use image node_id 1 to create a new Tokenizer object from inherited base classes: object with example: obj = Tokenizer(alphanum_only, reserved_tokens)