class_docstr: NLTK's recommended word tokenizer (currently the TreeBankTokenizer).
  Uses regular expressions to tokenize text. Assumes text has already been
  segmented into sentences.
  
  Performs the following steps:
  
  * split standard contractions, e.g. don't -> do n't
  * split commas and single quotes
  * separate periods that appear at the end of line
class_docstr_tok: ["NLTK","'s","recommended","word","tokenizer","(","currently","the","TreeBankTokenizer",")",".","Uses","regular","expressions","to","tokenize","text",".","Assumes","text","has","already","been","segmented","into","sentences",".","Performs","the","following","steps",":","*","split","standard","contractions",",","e.g",".","do","n't","-",">","do","n't","*","split","commas","and","single","quotes","*","separate","periods","that","appear","at","the","end","of","line"]
docstr: Return a list of word tokens.
  
  :param text: string of text.
  :param include_punc: (optional) whether to include punctuation as separate tokens. Default to True.
docstr_tok: ["Return","a","list","of","word","tokens",".",":","param","text",":","string","of","text",".",":","param","include_punc",":","(","optional",")","whether","to","include","punctuation","as","separate","tokens",".","Default","to","True","."]
code: def tokenize(self, text, include_punc=True):
      """Return a list of word tokens.
  
      :param text: string of text.
      :param include_punc: (optional) whether to include punctuation as separate tokens. Default to True.
      """
      tokens = nltk.tokenize.word_tokenize(text)
      if include_punc:
          return tokens
      else:
          # Return each word token
          # Strips punctuation unless the word comes from a contraction
          # e.g. "Let's" => ["Let", "'s"]
          # e.g. "Can't" => ["Ca", "n't"]
          # e.g. "home." => ['home']
          return [
              word
              if word.startswith("'")
              else strip_punc(word, all=False)
              for word in tokens
              if strip_punc(word, all=False)
          ]
  
file_path: TextBlob/textblob/tokenizers.py
imports: from __future__ import absolute_import
  from itertools import chain
  import nltk
  from textblob.utils import strip_punc
  from textblob.base import BaseTokenizer
  from textblob.decorators import requires_nltk_corpus
label_desc: Use image node_id 1 for calling the WordTokenizer obj's underlying member method code with example usage: obj.tokenize(text, include_punc) and returns: tokens, unknown