class_docstr: This class defines methods for loading and iterating over text datasets.
class_docstr_tok: ["This","class","defines","methods","for","loading","and","iterating","over","text","datasets","."]
docstr: Map string to a list of tokens.
  
  Arguments:
      string(str): String to be tokenized.
      token(object): Tokenizer object.
      tokenizer (function) : Tokenizer function.
  
  Returns:
      list : A list of tokens
docstr_tok: ["Map","string","to","a","list","of","tokens",".","Arguments",":","string","(","str",")",":","String","to","be","tokenized",".","token","(","object",")",":","Tokenizer","object",".","tokenizer","(","function",")",":","Tokenizer","function",".","Returns",":","list",":","A","list","of","tokens"]
code: def get_tokens(string, tokenizer=None):
      """
      Map string to a list of tokens.
  
      Arguments:
          string(str): String to be tokenized.
          token(object): Tokenizer object.
          tokenizer (function) : Tokenizer function.
  
      Returns:
          list : A list of tokens
      """
      # (if tokenizer is None, we have a list of characters)
      if tokenizer is None:
          return string
      else:
          return tokenizer(string)
  
imports: import logging
  import numpy
  import os
  import h5py
  from neon.data.dataiterator import NervanaDataIterator, ArrayIterator
  from neon.data.datasets import Dataset
  from neon.data.text_preprocessing import pad_sentences, pad_data
label_desc: Use image node_id 4 for calling the Text obj's underlying member method code with example usage: obj.get_tokens(string, tokenizer) and returns: string, tokenizer