class_docstr: Does really simple tokenization.  NLTK was too slow, so we wrote our own simple tokenizer
  instead.  This just does an initial split(), followed by some heuristic filtering of each
  whitespace-delimited token, separating contractions and punctuation.  We assume lower-cased,
  reasonably well-formed English sentences as input.
class_docstr_tok: ["Does","really","simple","tokenization",".","NLTK","was","too","slow",",","so","we","wrote","our","own","simple","tokenizer","instead",".","This","just","does","an","initial","split","(",")",",","followed","by","some","heuristic","filtering","of","each","whitespace-delimited","token",",","separating","contractions","and","punctuation",".","We","assume","lower-cased",",","reasonably","well-formed","English","sentences","as","input","."]
code: def __init__(self):
      # These are certainly incomplete.  But at least it's a start.
      self.special_cases = set(
          ["mr.", "mrs.", "etc.", "e.g.", "cf.", "c.f.", "eg.", "al."]
      )
      self.contractions = set(
          ["n't", "'s", "'ve", "'re", "'ll", "'d", "'m"]
      )
      self.contractions |= set(
          [x.replace("'", "'") for x in self.contractions]
      )
      self.ending_punctuation = set(
          [
              '"',
              "'",
              ".",
              ",",
              ";",
              ")",
              "]",
              "}",
              ":",
              "!",
              "?",
              "%",
              '"',
              "'",
          ]
      )
      self.beginning_punctuation = set(
          ['"', "'", "(", "[", "{", "#", "$", '"', "'"]
      )
  
imports: from __future__ import absolute_import
  import re
  import spacy
  from allennlp.common import Registrable
  from allennlp.common.util import get_spacy_model
  from allennlp.data.tokenizers.token import Token
label_desc: Use image node_id 1 to create a new SimpleWordSplitter object from inherited base classes: WordSplitter with example: obj = SimpleWordSplitter()