Is there a library for splitting sentence into a list of words in it?
Question: I'm looking at nltk for python, but it splits(tokenize) `won't` as
`['wo',"n't"]`. Are there libraries that do this more robustly?

I know i can build a regex of some sort to solve this problem, but I'm looking
for a library/tool because it would be a more directed approach. For example,
after a basic regex with periods and commas, I realized words like 'Mr. ' will
break the system.

(@artsiom)

If the sentence was "you won't?", split() will give me ["you", "won't?"]. So
there's an extra '?' that I have to deal with. I'm looking for a tried and
tested method which do away with the kinks like the above mentioned and also
the lot many exceptions that I'm sure exist. Of course, I'll resort to a
split(regex) if I don't find any.


Answer: [The Natural Language Toolkit](http://www.nltk.org/) (NLTK) is probably what
you need.

    
    
    >>> from nltk.tokenize import word_tokenize
    >>> word_tokenize("'Hello. This is a test.  It works!")
    ["'Hello", '.', 'This', 'is', 'a', 'test', '.', 'It', 'works', '!']
    >>> word_tokenize("I won't fix your computer")
    ['I', 'wo', "n't", 'fix', 'your', 'computer']
    

[`nltk.tokenize.word_tokenize`](http://www.nltk.org/api/nltk.tokenize.html?highlight=word_tokenize#nltk.tokenize.word_tokenize)
by default use the
[`TreebankWordTokenizer`](http://www.nltk.org/api/nltk.tokenize.html?highlight=treebankwordtokenizer#nltk.tokenize.treebank.TreebankWordTokenizer),
a word tokenizer that tokenizes sentences with the [Penn
Treebank](http://www.cis.upenn.edu/~treebank/tokenization.html) conventions.

Note that this tokenizer assumes that the text has already been segmented into
sentences.

You can test some of the various tokenizers provided by NLTK (i.e.
[`WordPunctTokenizer`](http://www.nltk.org/api/nltk.tokenize.html?highlight=wordpuncttokenizer#nltk.tokenize.regexp.WordPunctTokenizer),
[`WhitespaceTokenizer`](http://www.nltk.org/api/nltk.tokenize.html?highlight=whitespacetokenizer#nltk.tokenize.regexp.WhitespaceTokenizer)...)
on this [page](http://text-processing.com/demo/tokenize/).

