class_docstr: Does really simple tokenization.  NLTK was too slow, so we wrote our own simple tokenizer
  instead.  This just does an initial split(), followed by some heuristic filtering of each
  whitespace-delimited token, separating contractions and punctuation.  We assume lower-cased,
  reasonably well-formed English sentences as input.
class_docstr_tok: ["Does","really","simple","tokenization",".","NLTK","was","too","slow",",","so","we","wrote","our","own","simple","tokenizer","instead",".","This","just","does","an","initial","split","(",")",",","followed","by","some","heuristic","filtering","of","each","whitespace-delimited","token",",","separating","contractions","and","punctuation",".","We","assume","lower-cased",",","reasonably","well-formed","English","sentences","as","input","."]
docstr: Splits a sentence into word tokens.  We handle four kinds of things: words with punctuation
  that should be ignored as a special case (Mr. Mrs., etc.), contractions/genitives (isn't,
  don't, Matt's), and beginning and ending punctuation ("antennagate", (parentheticals), and
  such.).
  
  The basic outline is to split on whitespace, then check each of these cases.  First, we
  strip off beginning punctuation, then strip off ending punctuation, then strip off
  contractions.  When we strip something off the beginning of a word, we can add it to the
  list of tokens immediately.  When we strip it off the end, we have to save it to be added
  to after the word itself has been added.  Before stripping off any part of a token, we
  first check to be sure the token isn't in our list of special cases.
docstr_tok: ["Splits","a","sentence","into","word","tokens",".","We","handle","four","kinds","of","things",":","words","with","punctuation","that","should","be","ignored","as","a","special","case","(","Mr.","Mrs.",",","etc",".",")",",","contractions\/genitives","(","is","n't",",","do","n't",",","Matt","'s",")",",","and","beginning","and","ending","punctuation","(","``","antennagate","''",",","(","parentheticals",")",",","and","such.",")",".","The","basic","outline","is","to","split","on","whitespace",",","then","check","each","of","these","cases",".","First",",","we","strip","off","beginning","punctuation",",","then","strip","off","ending","punctuation",",","then","strip","off","contractions",".","When","we","strip","something","off","the","beginning","of","a","word",",","we","can","add","it","to","the","list","of","tokens","immediately",".","When","we","strip","it","off","the","end",",","we","have","to","save","it","to","be","added","to","after","the","word","itself","has","been","added",".","Before","stripping","off","any","part","of","a","token",",","we","first","check","to","be","sure","the","token","is","n't","in","our","list","of","special","cases","."]
code: def split_words(self, sentence):
      """
      Splits a sentence into word tokens.  We handle four kinds of things: words with punctuation
      that should be ignored as a special case (Mr. Mrs., etc.), contractions/genitives (isn't,
      don't, Matt's), and beginning and ending punctuation ("antennagate", (parentheticals), and
      such.).
  
      The basic outline is to split on whitespace, then check each of these cases.  First, we
      strip off beginning punctuation, then strip off ending punctuation, then strip off
      contractions.  When we strip something off the beginning of a word, we can add it to the
      list of tokens immediately.  When we strip it off the end, we have to save it to be added
      to after the word itself has been added.  Before stripping off any part of a token, we
      first check to be sure the token isn't in our list of special cases.
      """
      fields = sentence.split()
      tokens = []
      for field in fields:
          add_at_end = []
          while (
              self._can_split(field)
              and field[0] in self.beginning_punctuation
          ):
              tokens.append(Token(field[0]))
              field = field[1:]
          while (
              self._can_split(field)
              and field[-1] in self.ending_punctuation
          ):
              add_at_end.insert(0, Token(field[-1]))
              field = field[:-1]
  
          # There could (rarely) be several contractions in a word, but we check contractions
          # sequentially, in a random order.  If we've removed one, we need to check again to be
          # sure there aren't others.
          remove_contractions = True
          while remove_contractions:
              remove_contractions = False
              for contraction in self.contractions:
                  if self._can_split(field) and field.lower().endswith(
                      contraction
                  ):
                      add_at_end.insert(
                          0, Token(field[-len(contraction) :])
                      )
                      field = field[: -len(contraction)]
                      remove_contractions = True
          if field:
              tokens.append(Token(field))
          tokens.extend(add_at_end)
      return tokens
  
imports: from __future__ import absolute_import
  import re
  import spacy
  from allennlp.common import Registrable
  from allennlp.common.util import get_spacy_model
  from allennlp.data.tokenizers.token import Token
label_desc: Use image node_id 2 for calling the SimpleWordSplitter obj's underlying member method code with example usage: obj.split_words(sentence) and returns: tokens