NLTK set method prints characters, not words
Question: I'm new to NLTK (and python...) and I'm having two issues with one of its
basic methods: when I call

    
    
    sorted(set(<one of nltk's preloaded corpora>))
    

it prints a list of all the words in the text, but each word is preceded by
'u', like so: [u'yourselves', u'youth']. I thought I'd broken the tokenizer,
but I tried re-cloning the repo and re-installing.

The second, possibly related issue is that when I define a try using those
methods on a string I pass in myself, I get the individual characters, rather
than the words. Do I need to parse text that I pass in prior to using set()?


Answer: **`u'foo bar'`** is a just a string in `unicode`. both `str` and `unicode` are
considered as `basestring` (see <http://docs.python.org/2/howto/unicode.html>,
<http://docs.python.org/2/library/functions.html#basestring>)

    
    
    >>> x = u'foobar'
    >>> isinstance(x, str)
    False
    >>> isinstance(x,unicode)
    True
    >>> isinstance(x,basestring)
    True
    >>> print x
    foobar
    

When you try to access a corpus from NLTK's corpus readers, the default data
structure is a list of sentences where each sentence is a list of tokens and
each token is a basestring.

    
    
    >>> from nltk.corpus import brown
    >>> print brown.sents()
    [['The', 'Fulton', 'County', 'Grand', 'Jury', 'said', 'Friday', 'an', 'investigation', 'of', "Atlanta's", 'recent', 'primary', 'election', 'produced', '``', 'no', 'evidence', "''", 'that', 'any', 'irregularities', 'took', 'place', '.'], ['The', 'jury', 'further', 'said', 'in', 'term-end', 'presentments', 'that', 'the', 'City', 'Executive', 'Committee', ',', 'which', 'had', 'over-all', 'charge', 'of', 'the', 'election', ',', '``', 'deserves', 'the', 'praise', 'and', 'thanks', 'of', 'the', 'City', 'of', 'Atlanta', "''", 'for', 'the', 'manner', 'in', 'which', 'the', 'election', 'was', 'conducted', '.'], ...]
    

If you want a plain text version of the corpus, you can simply do:

    
    
    >>> for i in brown.sents():
    ...     print " ".join(i)
    ...     break
    ... 
    The Fulton County Grand Jury said Friday an investigation of Atlanta's recent primary election produced `` no evidence '' that any irregularities took place .
    

There are many internal magics in NLTK that makes the corpora work as it is
from the NLTK's modules but the simplest way to know what's in one of these
'preloaded' corpora (or more accurately 'precoded' corpus readers) is to use:

    
    
    >>> dir(brown)
    ['__class__', '__delattr__', '__dict__', '__doc__', '__format__', '__getattribute__', '__hash__', '__init__', '__module__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', '_add', '_c2f', '_delimiter', '_encoding', '_f2c', '_file', '_fileids', '_get_root', '_init', '_map', '_para_block_reader', '_pattern', '_resolve', '_root', '_sent_tokenizer', '_sep', '_tag_mapping_function', '_word_tokenizer', 'abspath', 'abspaths', 'categories', 'encoding', 'fileids', 'open', 'paras', 'raw', 'readme', 'root', 'sents', 'tagged_paras', 'tagged_sents', 'tagged_words', 'words']
    

