class_docstr: Splits a string into tokens, and joins them back.
class_docstr_tok: ["Splits","a","string","into","tokens",",","and","joins","them","back","."]
docstr: Splits a string into tokens.
code: def tokenize(self, s):
      """Splits a string into tokens."""
      s = tf.compat.as_text(s)
  
      if self.reserved_tokens:
          # First split out the reserved tokens
          substrs = self._reserved_tokens_re.split(s)
      else:
          substrs = [s]
  
      toks = []
      for substr in substrs:
          if substr in self.reserved_tokens:
              toks.append(substr)
          elif self._alphanum_only:
              toks.extend(ALPHANUM_REGEX.split(substr))
          else:
              toks.extend(ALL_REGEX.split(substr))
  
      # Filter out empty strings
      toks = [t for t in toks if t]
      return toks
  
imports: from __future__ import unicode_literals
  import abc
  import hashlib
  import json
  import re
  from tensorflow_datasets.core.utils.lazy_imports_utils import tensorflow
label_desc: Use image node_id 4 for calling the Tokenizer obj's underlying member method code with example usage: obj.tokenize(s) and returns: toks