class_docstr: A `Tokenizer` splits strings of text into tokens.  Typically, this either splits text into
  word tokens or character tokens, and those are the two tokenizer subclasses we have implemented
  here, though you could imagine wanting to do other kinds of tokenization for structured or
  other inputs.
  
  See the parameters to, e.g., :class:`~.SpacyTokenizer`, or whichever tokenizer
  you want to use.
  
  If the base input to your model is words, you should use a :class:`~.SpacyTokenizer`, even if
  you also want to have a character-level encoder to get an additional vector for each word
  token.  Splitting word tokens into character arrays is handled separately, in the
  :class:`..token_representations.TokenRepresentation` class.
docstr: Actually implements splitting words into tokens.
  
  # Returns
  
  tokens : `List[Token]`
docstr_tok: ["Actually","implements","splitting","words","into","tokens",".","#","Returns","tokens",":","`","List","[","Token","]","`"]
code: def tokenize(self, text: str) -> List[Token]:
      """
      Actually implements splitting words into tokens.
  
      # Returns
  
      tokens : `List[Token]`
      """
      raise NotImplementedError
  
file_path: allennlp/allennlp/data/tokenizers/tokenizer.py
imports: from typing import List, Optional
  import logging
  from allennlp.common import Registrable
  from allennlp.data.tokenizers.token_class import Token
label_desc: Use image node_id 2 for calling the Tokenizer obj's underlying member method code with example usage: obj.tokenize(text) without return types