# If you'd like to download it through the command line...
!curl -O http://www.cs.cornell.edu/home/llee/data/convote/convote_v1.1.tar.gz

# And then extract it through the command line...
!tar -zxf convote_v1.1.tar.gz

"""
Explanation: Homework 14 (or so): TF-IDF text analysis and clustering
Hooray, we kind of figured out how text analysis works! Some of it is still magic, but at least the TF and IDF parts make a little sense. Kind of. Somewhat.
No, just kidding, we're professionals now.
Investigating the Congressional Record
The Congressional Record is more or less what happened in Congress every single day. Speeches and all that. A good large source of text data, maybe?
Let's pretend it's totally secret but we just got it leaked to us in a data dump, and we need to check it out. It was leaked from this page here.
End of explanation
"""


# glob finds files matching a certain filename pattern
import glob

# Give me all the text files
paths = glob.glob('convote_v1.1/data_stage_one/development_set/*')
paths[:5]

len(paths)

"""
Explanation: You can explore the files if you'd like, but we're going to get the ones from convote_v1.1/data_stage_one/development_set/. It's a bunch of text files.
End of explanation
"""


speeches = []
for path in paths:
    with open(path) as speech_file:
        speech = {
            'pathname': path,
            'filename': path.split('/')[-1],
            'content': speech_file.read()
        }
    speeches.append(speech)
speeches_df = pd.DataFrame(speeches)
speeches_df.head()

"""
Explanation: So great, we have 702 of them. Now let's import them.
End of explanation
"""


speeches_df['content'].head(5)

"""
Explanation: In class we had the texts variable. For the homework can just do speeches_df['content'] to get the same sort of list of stuff.
Take a look at the contents of the first 5 speeches
End of explanation
"""


from sklearn.feature_extraction.text import CountVectorizer
count_vectorizer = CountVectorizer(max_features=100, stop_words='english')
from sklearn.feature_extraction.text import TfidfVectorizer
import re
from nltk.stem.porter import PorterStemmer

X = count_vectorizer.fit_transform(speeches_df['content'])

X.toarray()

"""
Explanation: Doing our analysis
Use the sklearn package and a plain boring CountVectorizer to get a list of all of the tokens used in the speeches. If it won't list them all, that's ok! Make a dataframe with those terms as columns.
Be sure to include English-language stopwords
End of explanation
"""


tophundred_df = pd.DataFrame(X.toarray(), columns=count_vectorizer.get_feature_names())

tophundred_df

"""
Explanation: Okay, it's far too big to even look at. Let's try to get a list of features from a new CountVectorizer that only takes the top 100 words.
End of explanation
"""


mrchairman_df = pd.DataFrame([tophundred_df['mr'], tophundred_df['chairman'], tophundred_df['mr'] + tophundred_df['chairman']], index=["mr", "chairman", "mr + chairman"]).T

mrchairman_df

num_speeches = len(mrchairman_df)

mrmention_df = mrchairman_df[(mrchairman_df['mr'] > 0)]
mr_mention = len(mrmention_df)

mrorchairmanmention_df = mrchairman_df[(mrchairman_df['mr + chairman'] > 0)]
mrorchair_mention = len(mrorchairmanmention_df)

print("There are",num_speeches,"speeches. Only", num_speeches - mr_mention, "do not mention mr and ", num_speeches - mrorchair_mention, "do not mention mr or chairman")

tophundred_df.columns

"""
Explanation: Now let's push all of that into a dataframe with nicely named columns.
Everyone seems to start their speeches with "mr chairman" - how many speeches are there total, and many don't mention "chairman" and how many mention neither "mr" nor "chairman"?
End of explanation
"""


tophundred_df['thank'].sort_values(ascending=False).head(1) # thank is not in the top 100 words unless you remove stop words

"""
Explanation: What is the index of the speech thank is the most thankful, a.k.a. includes the word 'thank' the most times?
End of explanation
"""


chinatrade_df = pd.DataFrame([tophundred_df['china'] + tophundred_df['trade']], index=["China + trade"]).T

chinatrade_df['China + trade'].sort_values(ascending=False).head(3)

"""
Explanation: If I'm searching for China and trade, what are the top 3 speeches to read according to the CountVectoriser?
End of explanation
"""


porter_stemmer = PorterStemmer()

def stemming_tokenizer(str_input):
    words = re.sub(r"[^A-Za-z0-9\-]", " ", str_input).lower().split()
    words = [porter_stemmer.stem(word) for word in words]
    return words

from sklearn.feature_extraction.text import TfidfVectorizer
tfidf_vectorizer = TfidfVectorizer(stop_words='english', tokenizer=stemming_tokenizer, use_idf=False, norm='l1', max_features = 100)
X = tfidf_vectorizer.fit_transform(speeches_df['content'])
chinatrade_tfidfpd = pd.DataFrame(X.toarray(), columns=tfidf_vectorizer.get_feature_names())

chinatrade_tfidfpd = pd.DataFrame([tophundred_df['china'] + tophundred_df['trade']], index=["China + trade"]).T

# chinatrade_tfidfpd

chinatrade_tfidfpd['China + trade'].sort_values(ascending=False).head(3)

"""
Explanation: Now what if I'm using a TfidfVectorizer?
End of explanation
"""


# index 0 is the first speech, which was the first one imported.
paths[0]

# Pass that into 'cat' using { } which lets you put variables in shell commands
# that way you can pass the path to cat
!cat {paths[0]}

"""
Explanation: What's the content of the speeches? Here's a way to get them:
End of explanation
"""


# Initialize a vectorizer
vectorizer = TfidfVectorizer(use_idf=True, tokenizer=stemming_tokenizer, stop_words='english', max_features =8)
X = vectorizer.fit_transform(speeches_df['content'])

X

pd.DataFrame(X.toarray())

from sklearn.cluster import KMeans
number_of_clusters = 8
km = KMeans(n_clusters=number_of_clusters)
km.fit(X)

print("Top terms per cluster:")
order_centroids = km.cluster_centers_.argsort()[:, ::-1]
terms = vectorizer.get_feature_names()
for i in range(number_of_clusters):
    top_ten_words = [terms[ind] for ind in order_centroids[i, :5]]
    print("Cluster {}: {}".format(i, ' '.join(top_ten_words)))

km.labels_

speeches_df['content']

results = pd.DataFrame()
results['content'] = speeches_df['content']
results['category'] = km.labels_
results

vectorizer.get_feature_names()

df = pd.DataFrame(X.toarray(), columns=vectorizer.get_feature_names())
df


"""
Explanation: Now search for something else! Another two terms that might show up. elections and chaos? Whatever you thnik might be interesting.
Enough of this garbage, let's cluster
Using a simple counting vectorizer, cluster the documents into eight categories, telling me what the top terms are per category.
Using a term frequency vectorizer, cluster the documents into eight categories, telling me what the top terms are per category.
Using a term frequency inverse document frequency vectorizer, cluster the documents into eight categories, telling me what the top terms are per category.
End of explanation
"""


!curl -LO https://github.com/ledeprogram/courses/raw/master/algorithms/data/hp.zip

"""
Explanation: Which one do you think works the best?
Harry Potter time
I have a scraped collection of Harry Potter fanfiction at https://github.com/ledeprogram/courses/raw/master/algorithms/data/hp.zip.
I want you to read them in, vectorize them and cluster them. Use this process to find out the two types of Harry Potter fanfiction. What is your hypothesis?
End of explanation
"""


