%matplotlib inline
import pandas as pd
import matplotlib.pyplot as plt


"""
Explanation: Computing In Context
Social Sciences Track
Lecture 4--topics, trends, and dimensional scaling
Matthew L. Jones
End of explanation
"""


from IPython.display import Image
Image("http://journalofdigitalhumanities.org/wp-content/uploads/2013/02/blei_lda_illustration.png")

import textmining_blackboxes as tm


"""
Explanation: Reading at scale: 
Martha Ballard's Diary
http://dohistory.org/diary/index.html
http://www.cameronblevins.org/posts/topic-modeling-martha-ballards-diary/
Richmond Dispatch
https://dsl.richmond.edu/dispatch/pages/home
Source: http://dlxs.richmond.edu/d/ddr/index.html
End of explanation
"""


#see if package imported correctly
tm.icantbelieve("butter")

"""
Explanation: IMPORTANT: tm is our temporarily helper, not a standard python package!!
download it from my github:
https://github.com/matthewljones/computingincontext
End of explanation
"""


title_info["Date"].str.replace("[^0-9]", "") #use regular expressions to clean up

title_info["Date"]=title_info["Date"].str.replace("\-\?", "5")
title_info["Date"]=title_info["Date"].str.replace("[^0-9]", "") # what assumptions have I made about the data?

title_info["Date"]=pd.to_datetime(title_info["Date"], coerce=True)

title_info["Date"]<pd.datetime(1800,1,1)

title_info[title_info["Date"]<pd.datetime(1800,1,1)]

"""
Explanation: Let's keep using the remarkable narratives available from Documenting the American South (http://docsouth.unc.edu/docsouthdata/)
Assuming that you are storing your data in a directory in the same place as your iPython notebook.
Put the slave narratives texts within a data directory in the same place as this notebook
End of explanation
"""


#Let's use a brittle thing for reading in a directory of pure txt files.
our_texts=tm.readtextfiles('data/na-slave-narratives/data/texts')
#again, this is not a std python package
#returns a simple list of the document as very long strings

#note if you want the following notebook will work on any directory of text files.

"""
Explanation: back to boolean indexing!
End of explanation
"""


our_texts, names=tm.readtextfiles("data/british-fiction-corpus")

names

"""
Explanation: For now, we'll play with the cool scientists and use the powerful and fast scikit learn package.
Let's look at Victorian novels for a little while
End of explanation
"""


our_texts=tm.data_cleanse(our_texts)

#more necessary when have messy text
#eliminate escaped characters

"""
Explanation: Our Zero-ith tool: cleaning up the text
I've included a little utility function in tm that takes a list of strings and cleans it up a bit
check out the code on your own time later
End of explanation
"""


from sklearn.feature_extraction.text import TfidfVectorizer

vectorizer=TfidfVectorizer(min_df=0.5, stop_words='english', use_idf=True)

document_term_matrix=vectorizer.fit_transform(our_texts)

# now let's get our vocabulary--the names corresponding to the rows
vocab=vectorizer.get_feature_names()


len(vocab)

document_term_matrix.shape

document_term_matrix_dense=document_term_matrix.toarray()


dtmdf=pd.DataFrame(document_term_matrix_dense, columns=vocab)

dtmdf

"""
Explanation: back to vectorizer from scikit learn
End of explanation
"""


#easy to program, but let's use a robust version from sklearn!
from sklearn.metrics.pairwise import cosine_similarity

similarity=cosine_similarity(document_term_matrix)

#Note here that the `cosine_similiary` can take 
#an entire matrix as its argument

pd.DataFrame(similarity, index=names, columns=names)

"""
Explanation: While this data frame is lovely to look at and useful to think with, it's tough on your computer's memory
End of explanation
"""


similarity_df.ix[1].order(ascending=False)

"""
Explanation: that is a symmetrical matrix relating each of the texts (rows) to another text (row)
End of explanation
"""


#here's the blackbox
from sklearn.manifold import MDS
mds = MDS(n_components=2, dissimilarity="precomputed", random_state=1)
positions= mds.fit_transform(1-similarity)

positions.shape

"""
Explanation: can do lots of things with similarity matrix
you've already seen hierarchical clustering
Multidimension scaling

Technique to visualize distances in high dimensional spaces in ways we can cognize.
Keep distances but reduce dimensionality
End of explanation
"""


#let's plot it: I've set up a black box
tm.plot_mds(positions,names)

names=[name.replace(".txt", "") for name in names]

tm.plot_mds(positions,names)

"""
Explanation: It's an 11 by 2 matrix
OR
simply an (x,y) coordinate pair for each of our texts
End of explanation
"""


our_texts, names=tm.readtextfiles("Data/PCCIPtext")

our_texts=tm.data_cleanse(our_texts)

#improved stoplist--may be too complete
stop=[]
with open('data/stoplist-multilingual') as f:
    stop=f.readlines()
    stop=[word.strip('\n') for word in stop]

texts = [[word for word in document.lower().split() if word not in stop] for document in our_texts] #gensim requires list of list of words in documents

from gensim import corpora, models, similarities, matutils
"""gensim includes its own vectorizing tools"""
dictionary = corpora.Dictionary(texts)
corpus = [dictionary.doc2bow(text) for text in texts]

#doc2bow just means `doc`uments to `b`ag `o`f `w`ords
#ok, this has just vectorized our texts; it's another form

"""
Explanation: What has this got us?
It suggests that even this crude measure of similarity is able to capture something significant.
Note: the axes don't really mean anything
interesting but what does it mean?
topic modeling
unsupervised algorithm for finding the major topics of texts
unlike hierarchical clustering, assumes texts spring from multiple sets of topics
the big thing in much text modeling, from humanities, to Facebook, to NSA
many variations
fantastic python package gensim
"corpora" = a collection of documents or texts
gensim likes its documents to be a list of lists of words, not a list of strings
Get the stoplist in the data directory in my github.
End of explanation
"""


number_topics=40
model = models.LdaModel(corpus, id2word=dictionary, num_topics=number_topics, passes=10) #use gensim multicore LDA

model.show_topics()

topics_indexed=[[b for (a,b) in topics] for topics in model.show_topics(number_topics,10,formatted=False)]
topics_indexed=pd.DataFrame(topics_indexed)

topics_indexed

"""
Explanation: Now we are going to call the topic modeling black box
the key parameter is how many distinct topics we want the computer to find
this will take a while
End of explanation
"""


model[dictionary.doc2bow(texts[1])]

"""
Explanation: So which topics most significant for each document?
End of explanation
"""


