from nltk.corpus import PlaintextCorpusReader

# identify the root folder where texts are stored
corpus_root = '/Users/asg/Dropbox/00-UCLA/Courses/DH150-TextAnalysis/plain_text'

# "Read" the list of files
filelist = PlaintextCorpusReader(corpus_root, '.*')

# list file names in root folder
filelist.fileids()

"""
Explanation: <h1>Reading in a Text Corpus</h1>
End of explanation
"""


# Use words function to create a words list from a single file
wordslist = filelist.words('1737_OldBailey_piracy_trial.txt')

# 'sents' is a method from PlaintextCorpusReader that reads in sentences from a document or corpus
# 'paras' is a method from the same that reads in paragraphs

# Begin to explore words in the text
# If you're interested in longer words that may carry more meaning, say words longer than three letters
filtered_words = [w for w in set(wordslist) if len(w) > 3]
#filtered_words

from nltk.probability import FreqDist

# Calculate the frequency distribution of all words
fdist = FreqDist(wordslist)
fdist

# Plot the frequency distribution of the top 30 words. 

fdist.plot(30)

"""
Explanation: <h1>Single Document Analysis</h1>

Let's begin by exploring just one document and its word frequencies.
End of explanation
"""


fdist_2 = FreqDist(word.lower() for word in set(filtered_words))

fdist_2.plot(30)


# Uncomment the following line by removing the hash symbol at the start of the line and run the code to read the documentation on frequency distributions.
# help(FreqDist)

"""
Explanation: <h2>Zipf's Law</h2>

Given some corpus of natural language utterances, the frequency of any word is inversely proportional to its rank on the frequency table.

This is why we want to remove stopwords. We are much more interested in words with much lower frequencies than "the", "and", or "to". Removing stopwords allows us to filter out this very high frequency words to focus on words that are likely to be more meaningful. 
One quick way to filter words without a stopwords list specified is to filter on length as we did above. Notice how the graph changes and which words appear when we filter on length.
<h3>Optional Extension: Explore Zipf's Law</h3>
<ol>
    <li>Read the [wikipedia article on Zipf’s Law](https://en.wikipedia.org/wiki/Zipf%27s_law)</li>
    <li>Generate a table of words and their frequencies in Voyant, export it as a tsv, and open it in Excel/Google Sheets <b>or</b> if you're familiar with pandas, create a data frame with this information using Python.
    <li>Add the following new columns to your spreadsheet/data frame:</li>
    <ul>
        <li>Word Rank</li>
        <li>log(rank)</li>
        <li>log(frequency)</li>
    </ul>
    <li>Graph log(rank) vs log(frequency) on a log-log graph</li>
    <li>Why might the log transformation be helpful in text analysis?</li>
</ol>
End of explanation
"""


#Read in your document with the open() function; the 'r' opens the file in "read" mode.

text = open('/Users/asg/Dropbox/00-UCLA/Courses/DH150-TextAnalysis/plain_text/1737_OldBailey_piracy_trial.txt', 'r').read()

# Uncomment the following line to see the document. Note that new line characters (\n) appear in the text.
# text.read()

"""
Explanation: <h1> Preprocessing text files for analysis</h1>

We're now starting to see why we need to use some sort of filtration method for our word list. In the last step, you will notice that we also use a method called "lower" to convert all of the text to lowercase so that all instances of a word will be grouped together whether that word is lower or upper case in the original raw text. This is generally one of the first steps in preprocessing a document before we begin analysis. Here is an outline of usual procedures to prepare our text:
<ol>
    <li>Tokenize text by paragraphs, sentences, or words (usually by words)</li>
    <li>Convert all tokens to lower case</li>
    <li>Remove stopwords</li>
    <li><em>Optional:</em> Use a "stemmer" to chop off word endings to group words with the same meaning together (flowery, flowering, flowers all reduce to flower, for example). <b>Or</b> use a "lemmatizer" to reduce tokens to their root words, rather than simply lopping off the ends of words. This is more precise, but it take much longer to run.</li>
</ol>

<b>In step 0 below, please note that the path to your document will be different than mine (shown in red below). See Working with Files for more tips on how to open files when working in Jupyter Labs.</b>
<h2>0. Read in your document</h2>
End of explanation
"""


# Tokenize text 
from nltk import word_tokenize
tokenlist = word_tokenize(text)

# Find out how many tokens there are in the document with the length (len()) method
print(len(tokenlist))

#Determine the type of object we've created with the tokenlist
type(tokenlist)

"""
Explanation: <h2>1. Tokenize text</h2>
End of explanation
"""


# We can sort the list of words with sorted()

vocab = sorted(set(tokenlist))

#Check the first four tokens in the sorted list. Note that punctuation appears first. We'll remove this in step 3.
vocab[0:4]

"""
Explanation: <h3>Strings and lists</h3>

<h4>The Difference between Lists and Strings:</h4>
Strings and lists are both kinds of sequence. We can pull them apart by indexing and slicing them, and we can join them together by concatenating them. However, we cannot join strings and lists.
<h4>Why does this matter?</h4>
When we open a file for reading into a Python program, we get a string corresponding to the contents of the whole file. If we use a for loop to process the elements of this string, all we can pick out are the individual characters — we don't get to choose the granularity. By contrast, the elements of a list can be as big or small as we like: for example, they could be paragraphs, sentences, phrases, words, characters. So lists have the advantage that we can be flexible about the elements they contain, and correspondingly flexible about any downstream processing. Consequently, one of the first things we are likely to do in a piece of NLP code is tokenize a string into a list of strings (3.7). Conversely, when we want to write our results to a file, or to a terminal, we will usually format them as a string (3.9). References point to NLTK Book
End of explanation
"""


# Convert text to lowercase using .lower() and a for loop, which will loop through every word in the token list
# and convert it lowercase.

words = [w.lower() for w in tokenlist]

#Check the first four words in our list
words[0:4]

"""
Explanation: <h2>2. Lowercase conversion</h2>
End of explanation
"""


from nltk.corpus import stopwords

#View stopwords in the English stopwords list
print(stopwords.words("english"))

"""
Explanation: <h2>3. Remove stopwords</h2>
End of explanation
"""


#Remove stopwords by selecting all words that are not in the stopwords list. 
stoplist = stopwords.words("english")

filtered = [w for w in words if w not in stoplist]

#View first 10 tokens in filtered list.
filtered[0:10]

#Uncomment the following line to download punkt tokenizer to remove punctuation (only need to do this step once)
#nltk.download("punkt")

#Keep only alpha-numeric tokens with word.isalnum() from punkt
words = [word for word in filtered if word.isalnum()]

"""
Explanation: <b>Notice that numbers and punctuation are not included in the stopwords list.</b>
End of explanation
"""


#Uncomment the following two lines and run this cell once to install the libraries.

#!pip3 install matplotlib
#!pip3 install wordcloud

#import the necessary packages

from matplotlib import pyplot as plt

# Begin by a exploring a new line chart of the word frequency distribution after our pre-processing steps
fd = FreqDist(words)
fd.plot(30)
plt.show()

"""
Explanation: <h2>4. Text Normalization</h2>

In this tutorial, we won't get into stemming and lemmatization, as this an optional step in text preparation. 
If you decide this would be helpful as you work with your documents, see: NLTK: A Beginners Hands-on Guide to Natural Language Processing. 
<hr/>

<h1>Word Frequency Analysis</h1>

To begin, we will need to install two new libraries if you don't already have them: 
<ul>
    <li>matplotlib is a data visualization library</li>
    <li>wordcloud is a wordcloud generator</li>
</ul>
End of explanation
"""


import pprint

#Convert word list to a single string
clean_words_string = " ".join(words)

#generate a basic wordcloud
from wordcloud import WordCloud
wordcloud = WordCloud(background_color="black").generate(clean_words_string)

#plot the wordcloud
plt.figure(figsize = (12, 12))
plt.imshow(wordcloud)

#to remove the axis value
plt.axis("off")
plt.show()

"""
Explanation: Now we have reduced our word list to just those that are likely to have more meaning. 
If the words that appear here are still not satisfactory, you can also use the extended stopwords list available in NLTK or add your own. 
To add your own, try the following code and replace the words inside the triple quotes with terms you want to remove.
more_stopwords = """with some your just have from it's /via &amp;amp; that they your there this into providing would can't"""
stoplist += more_stopwords.split()
End of explanation
"""


#To be able to use methods like concordance and collocations, convert the token list to nltk's "Text" type
textcontent = nltk.Text(tokenlist)
textcontent 

textcontent.concordance('piracy')

#Get a list of top collocations in the document
textcontent.collocations()

"""
Explanation: <h1>Simple Concordance and Collocations Methods</h1>

<h2>Converting from list to Text type in NLTK</h2>
End of explanation
"""


