import itertools
import nltk
import string

nltk.data.path

nltk.data.path.append("../nltk_data")

nltk.data.path = ['../nltk_data']

"""
Explanation: 1. Intro to low level NLP - Tokenization, Stopwords, Frequencies, Bigrams
Lynn Cherny,  arnicas@gmail
End of explanation
"""


ls ../data/books

# the "U" here is for universal newline mode, because newlines on Mac are \r\n and on Windows are \n.

with open("../data/books/Austen_Emma.txt", "U") as handle:
    text = handle.read()

text[0:120]

"""
Explanation: Tokenization
Read in a file to use for practice.  The directory is one level above us now, in data/books.  You can add other files into the data directory if you want.
End of explanation
"""


## if you don't want the newlines in there - replace them all.
text = text.replace('\n', ' ')

text[0:120]

## Breaking it up by sentence!  Can be very useful for vis :)
nltk.sent_tokenize(text)[0:10]

tokens = nltk.word_tokenize(text)
tokens[70:85]  # Notice the punctuation:

# Notice the difference here:
nltk.wordpunct_tokenize(text)[70:85]

"""
Explanation: Before we go further, it might be worth saying that even the lines of a text can be interesting as a visual.  Here are a couple of books where every line is a line of pixels, and we've applied a simple search in JS to show lines of dialogue in pink.  (The entire analysis is done in the web file book_shape.html -- so it's a little slow to load.)
<img src="img/book_shape_dialog_emma_moby.png">
But usually you want to extract some sense of the content, which means crunching the text itself to get insights about the overall file.
End of explanation
"""


# run text2words on this book file at this location, pipe the output to the unix "head" command, showing 20 lines
!textkit text2words ../data/books/Austen_Emma.txt | head -n20

# Pipe the output through the lowercase textkit operation, before showing 20 lines again!
!textkit text2words ../data/books/Austen_Emma.txt | textkit lowercase | head -n20

"""
Explanation: There are other options for tokenization in NLTK.  You can test some out here: http://text-processing.com/demo/tokenize/
Doing it in textkit at the command line:
Thanks to the work of Bocoup.com, we have a library that will do some simple text analysis at the command line, wrapping up some of the python functions I'll be showing you. The library is at https://github.com/learntextvis/textkit.  Be aware it is under development!  Also, some of these commands will be slower than running the code in the notebook.
When I say you can run these at the command line, what I mean is that in your terminal window you can type the command you see here after the !.  The ! in the Jupyter notebook means this is a shell command.
The | is a "pipe."  This means take the output from the previous command and make it the input to the next command.
End of explanation
"""


!textkit text2words ../data/books/Austen_Emma.txt | textkit filterpunc | textkit tokens2counts > ../outputdata/simple_emma_counts.csv

!ls -al ../outputputdata/simple_emma_counts.csv

"""
Explanation: What if, at this point, we made a word cloud?  Let's say we strip out the punctuation and just count the words. I'll do it quickly just to show you... but we'll go a bit further.
End of explanation
"""


from nltk.corpus import stopwords
english_stops = stopwords.words('english')

# Notice they are lowercase.  This means we need to be sure we lowercase our text if we want to match against them.

english_stops

tokens = nltk.word_tokenize(text)
tokens[0:15]

# this is how many tokens we have:
len(tokens)

"""
Explanation: Using the html file simple_wordcloud.html and this data file, we can see something basically useless. You don't have to do this yourself, but if you want to, edit that file to point to the ../outputdata/simple_emma_counts.csv at the bottom.
<img src="img/emma_wc_nostops.png">
StopWords
"Stopwords" are words that are usually excluded because they are common connectors (or determiners, or short verbs) that are not considered to carry meaning. BEWARE hidden stopword filtering in libraries you use and always check stopword lists to see if you agree with their contents!
End of explanation
"""


# try this without .lower in the if-statement and check the size!
# We are using a python list comprehension to remove the tokens from Emma (after lowercasing them!) that are stopwords
tokens = [token.lower() for token in tokens if token.lower() not in english_stops]
len(tokens)

# now look at the first 15 words:
tokens[0:15]

"""
Explanation: We want to strip out stopwords - use a list comprehension.  Notice you need to lower case the words before you check for membership!
End of explanation
"""


import string
string.punctuation

# Now remove the punctuation and see how much smaller the token list is now:
tokens = [token for token in tokens if token not in string.punctuation]
len(tokens)

# But there's some awful stuff still in here:
sorted(tokens)[0:20]

"""
Explanation: Let's get rid of punctuation too, which isn't used in most bag-of-words analyses.  "Bag of words" means lists of words where the order doesn't matter.  That's how most NLP tasks are done!
End of explanation
"""


[token for token in tokens if len(token) <= 2][0:20]

# Let's define a small python function that's a pretty common one for text processing.

def clean_tokens(tokens):
    """ Lowercases, takes out punct and stopwords and short strings """
    return [token.lower() for token in tokens if (token not in string.punctuation) and 
                   (token.lower() not in english_stops) and len(token) > 2]

clean = clean_tokens(tokens)
clean[0:20]

len(clean)

"""
Explanation: The ugliness of some of those tokens! You have some possibilities now - add to your stopwords list the ones you want removed; or remove all very short words, which will get rid of our puntuation problem too.
End of explanation
"""


!textkit text2words ../data/books/Austen_Emma.txt | textkit filterpunc | textkit tokens2lower > ../outputdata/emma_lower.txt

!head -n5 ../outputdata/emma_lower.txt

!textkit text2words ../outputdata/emma_lower.txt | textkit filterwords | textkit filterlengths -m 3 > ../outputdata/emma_clean.txt

!head -n10 ../outputdata/emma_clean.txt

"""
Explanation: So now we've reduced our data set from 191739 to 72576, just by removing stopwords, punctuation, and short strings. If we're interested in "meaning", this is a useful removal of noise.
Using textkit at the commandline for filtering stopwords and punctuation and lowercase and short words:
(We are breaking these lines up with some intermediate output files (emma_lower.txt) because of how long these get.)
End of explanation
"""


from nltk import Text
cleantext = Text(clean)
cleantext.vocab().most_common()[0:20]

# if you want to know all the vocabular, without counts - you can remove the [0:10] which just shows the first 10:
cleantext.vocab().keys()[0:10]

# Another way to do this is with nltk.FreqDist, which creates an object with keys that are 
# the vocabulary, and values for the counts:

nltk.FreqDist(clean)['sashed']

"""
Explanation: Count Word Frequencies
The obvious thing you want to do next is count frequencies of words in texts - NLTK has you covered. (Or you can do it easily yourself using a Counter object.)
End of explanation
"""


wordpairs = cleantext.vocab().most_common()
with open("../outputdata/emma_word_counts.csv", "w") as handle:
    for pair in wordpairs:
        handle.write(pair[0] + "," + str(pair[1]) + "\n")

!head -n5 ../outputdata/emma_word_counts.csv

"""
Explanation: If you wanted to save the words and counts to a file to use, you can do it like this:
End of explanation
"""


!textkit text2words ../data/books/Austen_Emma.txt | textkit filterpunc | textkit tokens2lower > ../outputdata/emma_lower.txt

!textkit filterwords ../outputdata/emma_lower.txt | textkit filterlengths -m 3 > ../outputdata/emma_clean.txt

!head -n5 ../outputdata/emma_clean.txt

!textkit tokens2counts ../outputdata/emma_clean.txt > ../outputdata/emma_word_counts.csv

!head ../outputdata/emma_word_counts.csv

"""
Explanation: Using Textkit at the command line:
Let's save the output of the filtered, lowercase words into a file called cleantokens.txt:
End of explanation
"""


from nltk.collocations import *
bigram_measures = nltk.collocations.BigramAssocMeasures()

word_fd = nltk.FreqDist(clean) # all the words
bigram_fd = nltk.FreqDist(nltk.bigrams(clean))
finder = BigramCollocationFinder(word_fd, bigram_fd)
scored = finder.score_ngrams(bigram_measures.likelihood_ratio) # a good option here, there are others:
scored[0:50]

# Trigrams - using raw counts is much faster.

finder = TrigramCollocationFinder.from_words(clean,
    window_size = 15)
finder.apply_freq_filter(2)
#ignored_words = nltk.corpus.stopwords.words('english') # if you hadn't removed them...
# if you want to remove extra words, like character names, you can create the ignored_words list too:
#finder.apply_word_filter(lambda w: len(w) < 3 or w.lower() in ignored_words)

finder.nbest(trigram_measures.raw_freq, 20)

finder.score_ngrams(trigram_measures.raw_freq)[0:20]

## This is very slow!  Don't run unless you're serious :)

finder = TrigramCollocationFinder.from_words(clean,
    window_size = 20)
finder.apply_freq_filter(2)
#ignored_words = nltk.corpus.stopwords.words('english') # if you hadn't removed them...
# if you want to remove extra words, like character names, you can create the ignored_words list too:
#finder.apply_word_filter(lambda w: len(w) < 3 or w.lower() in ignored_words)
finder.apply_word_filter(lambda w: len(w) < 3)  # remove short words
finder.nbest(trigram_measures.likelihood_ratio, 10)

"""
Explanation: Now you are ready to make word clouds that are smarter than your average word cloud.  Move your counts file into a place where your html can find it.  Edit the file "simple_wordcloud.html" to use the name of your file, including the path! 
<img src=img/edit_file_name.png>
You may still see some words in here you don't love -- names, modal verbs (would, could):
<img src="img/emma_wc_before_more_stops.png">
We can actually edit those by hand in the html/js code if you want. Look for the list of stopwords. You can change the color, too, if you want. I've added a few more stops to see how it looks now:
<img src="img/emma_wc_after_more_stops.png">
You might want to keep going.
Finding Most Common Pairs of Words ("Bigrams")
Words occur in common sequences, sometimes.  We call word pairs "bigrams" (and word triples "trigrams").  We refer to N-grams when we mean "sequences of some length."
End of explanation
"""


with open("../data/movie_reviews/all_pos.txt", "U") as handle:
    text = handle.read()

tokens = nltk.word_tokenize(text)  # tokenize them - split into words and punct
clean_posrevs = clean_tokens(tokens)  # clean up stopwords and punct

clean_posrevs[0:10]

word_fd = nltk.FreqDist(clean_posrevs)
bigram_fd = nltk.FreqDist(nltk.bigrams(clean_posrevs))
finder = BigramCollocationFinder(word_fd, bigram_fd)
scored = finder.score_ngrams(bigram_measures.likelihood_ratio) # other options are 
scored[0:50]

"""
Explanation: Some more help is here: http://www.nltk.org/howto/collocations.html
What if we wanted to try non-fiction, to see if there are more interesting results?
We need to read and clean the text for another file.  Let's try positive movie reviews, located in data/movie_reviews/all_pos.txt.
End of explanation
"""


!textkit text2words ../data/books/Austen_Emma.txt | textkit filterpunc | textkit tokens2lower > ../outputdata/emma_lower.txt

!textkit filterwords ../outputdata/emma_lower.txt | textkit filterlengths -m 3 | textkit words2bigrams > ../outputdata/bigrams_emma.txt

!head -n10 ../outputdata/bigrams_emma.txt

"""
Explanation: To see more details about the NLTK Text object methods, read the code/doc here: http://www.nltk.org/_modules/nltk/text.html
Bigrams in Textkit at the command line:
Create a file with all the word pairs, after making everything lowercase and removing punctuation and basic stopwords:
End of explanation
"""


!textkit tokens2counts ../outputdata/bigrams_emma.txt > ../outputdata/bigrams_emma_counts.txt

!head -n20 ../outputdata/bigrams_emma_counts.txt

"""
Explanation: Then count them to get frequencies of the pairs.  This may reveal custom stopwords you want to filter out.
End of explanation
"""


!textkit filterwords --custom ../data/emma_customstops.txt ../outputdata/emma_lower.txt > ../outputdata/emma_custom_stops.txt

!textkit filterlengths -m 3 ../outputdata/emma_custom_stops.txt | textkit words2bigrams > ../outputdata/bigrams_emma.txt

!textkit tokens2counts ../outputdata/bigrams_emma.txt > ../outputdata/bigrams_emma_counts.txt

!head -n20 ../outputdata/bigrams_emma_counts.txt

"""
Explanation: Suppose you didn't want the names in there?  Custom stopwords can be created in a file, one per line, and added as an argument to the filterwords command:
End of explanation
"""


text = nltk.word_tokenize("And now I present your cat with something completely different.")
tagged = nltk.pos_tag(text)  # there are a few options for taggers, details in NLTK books
tagged

nltk.untag(tagged)

"""
Explanation: You could add more if you wanted.
Parts of Speech - Abbreviated POS
To do this, you need to make sure your nltk_data has the the MaxEnt Treebank POS tagger -- you can get it interactively with nltk.download() (on the models tab) - but we have it here already in the nltk_data directory.
End of explanation
"""


!textkit text2words ../data/books/Austen_Emma.txt | textkit tokens2pos | grep NNP | cut -d, -f1 > ../outputdata/emma_nouns.txt

!textkit tokens2counts ../outputdata/emma_nouns.txt > ../outputdata/emma_NNP_counts.csv

!head -n10 ../outputdata/emma_NNP_counts.csv

"""
Explanation: The Penn Treebank part of speech tags are these:
<img src="./img/TreebankPOSTags.png">
source: https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html
Parts of speech are used in anaysis that's "deeper" than bags-of-words approaches.  For instance, chunking (parsing for structure) may be used for entity identification and semantics.  See http://www.nltk.org/book/ch07.html for a little more info, and the 2 Perkins NLTK books.
Note also that "real linguists" parse a sentence into a syntactic structure, which is usually a tree form.
<img src="img/sentence_tree.png">
(Source)
For instance, try out the Stanford NLP parser visually at http://corenlp.run/.
In TextKit at the command line:
This requires more Unix-foo, since Textkit doesn't have the full capability yet to do just a count of certain POS. We'll use grep to search for all the NNPs (proper names, or characters) and cut to get the first column (the word).
End of explanation
"""


!textkit text2words ../data/books/Austen_Emma.txt | textkit tokens2pos | grep VB | cut -d, -f1 > ../outputdata/emma_verbs.txt

!textkit tokens2counts ../outputdata/emma_verbs.txt > ../outputdata/emma_VB_counts.csv

!head -n20 ../outputdata/emma_VB_counts.csv

"""
Explanation: That's all proper names.  Maybe not very interesting.
Let's look at the verbs now.
End of explanation
"""


!textkit text2words ../data/books/Austen_Emma.txt | textkit tokens2lower \
| textkit filterwords | textkit tokens2pos | grep VB | cut -d, -f1 > ../outputdata/emma_verbs.txt

!textkit tokens2counts ../outputdata/emma_verbs.txt > ../outputdata/emma_VB_counts.csv

!textkit text2words ../data/books/Austen_Pride.txt | textkit tokens2lower \
| textkit filterwords | textkit tokens2pos | grep VB | cut -d, -f1 > ../outputdata/pride_verbs.txt

!textkit tokens2counts ../outputdata/pride_verbs.txt > ../outputdata/pride_VB_counts.csv

"""
Explanation: Keep in mind that you can filter stopwords before you do this, too, although you should also lowercase things too. Also note that "grep VB" will also match other forms of VB, like VBP!
Suppose you want to make a word cloud of just the verbs... without stopwords, and you want to compare two books by the same author, say Emma and Pride and Prejudice.  Let's try it. (I'm using a \ to wrap the line here, so I don't need to use intermediate files for short commands.)
End of explanation
"""


# stemming removes affixes.  This is the default choice for stemming although other algorithms exist.
from nltk.stem import PorterStemmer
stemmer = PorterStemmer()
stemmer.stem('believes')

# lemmatizing transforms to root words using grammar rules. It is slower. Stemming is more common.
from nltk.stem import WordNetLemmatizer
lemmatizer = WordNetLemmatizer()
lemmatizer.lemmatize('said', pos='v')  # if you don't specify POS, you get zilch.

lemmatizer.lemmatize('cookbooks')

stemmer.stem('wicked')

lemmatizer.lemmatize("were", pos="v")  # lemmatizing would allow us to collapse all forms of "be" into one token

# an apparently recommended compression recipe in Perkins Python 3 NLTK book? Not sure I agree.
stemmer.stem(lemmatizer.lemmatize('buses'))

"""
Explanation: If you load those files into wc_clouds_bars.html (at the bottom!), and use the provided extra stopwords, you'll get this:
<img src="img/two_clouds_bars.png">
Underneath the word clouds are simple bar charts, to allow you to more precisely see the top words (it's cut off at 150).  This is one of the issues with word clouds, they lack precision in their display.
Another option for showing the difference more clearly is in analytic_wordlist.html.
<img src="img/analytic_wordlists.png">
Stemming / Lemmatizing
The goal is to merge data items that are the same at some "root" meaning level, and reduce the number of features in your data set.  "Cats" and "Cat" might want to be treated as the same thing, from a topic or summarization perspective. You can really see this in the word clouds above...so many forms of the same word!
End of explanation
"""


def make_verbs_lemmas(filename, outputfile):
    from collections import Counter
    with open(filename, 'U') as handle:
        emmav = handle.read()
    emmaverbs = emmav.split('\n')
    lemmaverbs = []
    for verb in emmaverbs:
        lemmaverbs.append(lemmatizer.lemmatize(verb, pos='v'))
    counts = Counter(lemmaverbs)
    with open(outputfile, 'w') as handle:
        for key, value in counts.items():
            if key:
                handle.write(key + "," + str(value) + "\n")
    print "wrote ", outputfile

make_verbs_lemmas("../outputdata/emma_verbs.txt", "../outputdata/emma_lemma_verbs.csv")

!head -n5 ../outputdata/emma_lemma_verbs.csv

make_verbs_lemmas("../outputdata/pride_verbs.txt", "../outputdata/pride_lemma_verbs.csv")

!head -n5 ../outputdata/pride_lemma_verbs.csv

"""
Explanation: Look at some of the clouds above.  How would this be useful, do you think?
End of explanation
"""


