import re
import urllib2
import graphlab as gl

BASE_DIR = "/Users/pablo/fromHomeMac/sherlock/data" # NOTE: Update BASE_DIR to your own directory path

books_url = "http://sherlock-holm.es/ascii/"
re_books_links = re.compile("\"piwik_download\"\s+href=\"(?P<link>.*?)\">(?P<title>.*?)</a>", re.MULTILINE)
html = urllib2.urlopen(books_url).read()
books_list = [m.groupdict() for m in re_books_links.finditer(html)]
print books_list

"""
Explanation: Elementary, My Dear Watson!
Text Analytics Tutorial using Sherlock Holmes Stories
In this notebook, I will demonstrate several text analytics techniques that can be used to analyze various text corpora in order to extract various interesting insights from the text. 
Namely, throughout this notebook, I will use The Sherlock Holmes Books Collection to show how to (a) calculate various textual statistics; (b) create the social network among entities that appear in the books; (c) use a topic model to discover abstract topics in the text; and (d) utilize Word2Vec to find connections among various section of the text, and do some predictions.
To perform the text analytics presented in this notebook, we will use NLTK Python package, GraphLab Create's SFrame and SGraph objects, as well as GraphLab's text analytics toolkit, and
Word2Vec deep learning inspired model implemented in the Gensim Python package.
The notebook is divided to the following sections:
- <a href="#setup">Setup</a>
- <a href="#prepare">Preparing and validating the dataset</a>
- <a href="#stat">Calculating Various Statistics</a>
- <a href="#sn">Constructing Social Networks</a>
- <a href="#topic">Topic Models</a>
Each section can be executed independently. So feel free to skip ahead, just remember  to import all the required packages, and define all the needed functions.
Required Python Packages:
- GraphLab Create - for classification, data.
- NLTK (including downloading punkt).
- Stanford Named Entity Recognizer - for named entity recognition. 
- Gensim - for Word2Vec deep learning.
 engineering, and evaluation.
- pyLDAvis - for topic model visualziation. 
Let's do some text analytics!
<a id="setup"></a>0. Setup
Before we begin, make sure you have installed all the required Python packages. (The instructions below use pip. You can use easy_install, too.) Also, consider using virtualenv for a cleaner installation experience instead of sudo. I also recommend to run the code via IPython Notebook.
<pre>
$ sudo pip install --upgrade gensim
$ sudo pip install --upgrade nltk
$ sudo pip install --upgrade graphlab-create
$ sudo pip install --upgrade pyldavis
</pre>

You will need a product key for GraphLab Create, and to make Stanford Named Entity Recognizer work with Pyhton NLTK.
After installing NLTK and from an interactive shell download the punkt model by importing nltk and running nltk.download(). From the resulting interactive window navigate to the model tab and select punkt and download to your system. 
To prepare the Stanford Named Entity Recognizer to work in your system make sure you read the following links : 
[1 - How to] (http://textminingonline.com/how-to-use-stanford-named-entity-recognizer-ner-in-python-nltk-and-other-programming-languages)
[2 - API] (http://www.nltk.org/api/nltk.tag.html#module-nltk.tag.stanford)
[3 - Stanford Parser FAQ] (http://nlp.stanford.edu/software/parser-faq.shtml)
[4 - download] (http://nlp.stanford.edu/software/CRF-NER.html)
5 - In the extracted directory it seems to help to run the stanford-ner.jar file in some computer configurations. 
Take note of what directory you downloaded and extracted the files to as you will need the location to pass the classifier and Stanford NER as arguments later in this notebook.
Also in case you haven’t already, make sure you are running the latest [Java JDK] (http://www.oracle.com/technetwork/java/javase/downloads/index.html)
For a quick test you can open an ipython shell and try the following:
from nltk.tag import StanfordNERTagger
st = StanfordNERTagger(‘[path_to_your_downloaded_package_classifiers_directory]/english.all.3class.distsim.crf.ser.gz', '[path_to_your_downloaded_package_root_directory]/stanford-ner.jar')
 st.tag('When we turned him over, the Boots recognized him at once as being the same gentleman who had engaged the room under the name of Joseph Stangerson.'.split())
<a id="prepare"></a>1. Preparing the Dataset
1.1 Constructing the Dataset
<pre>
<i>"Data! Data! Data!" he cried impatiently. "I can't make bricks without clay."</i>
                                                -The Adventure of the Copper Beeches
</pre>

Throughout this notebook, we will be analyzing Sherlock Holmes's stories collection. 
So first, we will download the stories in ASCII format from the sherlock-holm.es website. 
sherlock-holm.es website contains over sixty downloadable stories, we will use the following code to download the stories and insert them into a SFrame object.
<b> <u>Important Note:</u></b> in some countries, such as the U.S., few of Sherlock Holmes's books & stories are still under copyright restrictions. For more information, please advise the following website, and read the guidelines that appear in the end of sherlock-holm.es ASCII download page.
End of explanation
"""


# Filter books due to copyright issues. In this code, we filtered "The Complete Canon", “Case-Book of Sherlock Holmes” books, and
# "The Canon — U.S. edition" book (For more information please read the note above).
filtered_books = set(["The Complete Canon", "The Case-Book of Sherlock Holmes", "The Canon — U.S. edition" ])
books_list = filter(lambda d: d['title'] not in filtered_books, books_list )

#Download books' texts (to not overload the website we download the text in batch and not in parallel)
for d in books_list:
    d['text'] = urllib2.urlopen("http://sherlock-holm.es" + d['link']).read().strip()

"""
Explanation: We got the books' titles and links, now let's download the books' texts.
End of explanation
"""


sf = gl.SFrame(books_list).unpack("X1", column_name_prefix="")
sf.save("%s/books.sframe" % BASE_DIR)
sf.head(3)

"""
Explanation: Let's load the dict list into a SFrame object.
End of explanation
"""


import graphlab as gl
import re

BASE_DIR = "/Users/pablo/fromHomeMac/sherlock/data" # NOTE: Update BASE_DIR to your own directory path
gl.canvas.set_target('ipynb')
sf = gl.load_sframe("%s\\books.sframe" % BASE_DIR)

"""
Explanation: <a id="stat"></a>2. Calculating Various Statistics
In this section, I will demonstrate  how it is very straight-forward to utilize GraphLab Create SFrame object to calculate & visualize various statistics.
In previous section, we created a SFrame object which consists of 64 texts. Let us first load the SFrame object.
End of explanation
"""


sf['chars_num'] = sf['text'].apply(lambda t: len(t))
sf.head(3)

"""
Explanation: Using Python, it is very easy to calculate the number of characters  in a text, we just need to use the built-in len function. Let's calculate the number of characters  in each downloaded text using the the <i>len</i> function and SArray's <i>apply</i> function (notice that each column in a SFrame object is a SArray object).
End of explanation
"""


sf['chars_num'].show()

"""
Explanation: Let's use the show function to visualize  the distribution  of text length in each one of our downloaded text.
End of explanation
"""


text = """I think that you know me well enough, Watson, to understand that I am by no means a nervous man. At the same time,
it is stupidity rather than courage to refuse to recognize danger when it is close upon you."""

#using the split function
print text.split()

#Using NLTK 
#Note: Remember to download the NLTK's punkt package by running nltk.download() from the Interactive Python Shell
import nltk
print nltk.word_tokenize(text)

"""
Explanation: We can see that the mean characters  number in the download stories is 95020.42, and the maximal number of characters  in a story is 662,242 characters.
Let's also calculate  the number of words in each text. Calculating the number of words in a text is little trickier and there are several methods to perform this task. Using one of the following methods:
End of explanation
"""


sf['words_num'] = sf['text'].apply(lambda t: len(re_words_split.findall(t)))

"""
Explanation: You can see that both using the split function, or using the regular expression work pretty-well. 
However, it is important to notice, that the first regular expression can mistakenly split words, 
such as "S.H." into two words, while the split function doesn't remove punctuation. 
Therefore, if we want to be precise, we can use the NLTK's tokenize package and remove punctuation from the results.
Nevertheless, for our case, it is good enough to use the regular expression method to count words.
End of explanation
"""


tokenizer = nltk.data.load('tokenizers/punkt/english.pickle')
def txt2sentences(txt, remove_none_english_chars=True):
    """
    Split the English text into sentences using NLTK
    :param txt: input text.    
    :param remove_none_english_chars: if True then remove none English chars from text
    :return: string in which each line consists of single sentence from the original input text.
    :rtype: str
    """        
    # decode to utf8 to avoid encoding problems - if someone has better idea how to solve encoding 
    # problem I will love to learn about it.     
    txt = txt.decode("utf8") 
    # split text into sentences using NLTK package
    for s in tokenizer.tokenize(txt):
        if remove_none_english_chars:
            #remove none English chars
            s = re.sub("[^a-zA-Z]", " ", s)
        yield s

sf['sentences_num'] = sf['text'].apply(lambda t: len(list(txt2sentences(t))))

sf[['chars_num','words_num','sentences_num']].show()

"""
Explanation: We can also use NLTK to count the number of sentences in each story.
End of explanation
"""


sf['words_count'] = gl.text_analytics.count_words(sf['text'], to_lower=True)
sf['sherlock_count'] = sf['words_count'].apply(lambda d: d.get('sherlock',0))
sf['watson_count'] = sf['words_count'].apply(lambda d: d.get('watson',0))
sf['elementary_count'] = sf['words_count'].apply(lambda d: d.get('elementary',0))
sf[['sherlock_count', 'watson_count', 'elementary_count']].show()

"""
Explanation: Until now I calculated very basic text statistics. Let's try to do something more complicated like count the number of time the words 'Sherlock',
'Watson', and 'Elementary' appeared in each story. We will do it using GraphLab's text_analytics.count_words toolkit.
<u>Note</u>: To count the frequency a word appears in a text, one can also consider using the collection.Counter function.
End of explanation
"""


linear_reg = gl.linear_regression.create(sf, target='sherlock_count', features=['watson_count'])
linear_reg.show()

"""
Explanation: It is nice to see that the mean number of times the word 'Sherlock' appear in the stories is 9.609 times while the mean the word 'Watson' appear is only 1.734.
Moreover, there are stories, such as The Adventure of the Lion's Mane that the word 'Sherlock' doesn't appear even once. 
Let's try to use simple linear regression to predict the number of times the word 'Sherlock' appear in a text based on the number of time the word 'Watson' appear in the text.
End of explanation
"""


import graphlab as gl
import re,nltk
gl.canvas.set_target('ipynb')

tokenizer = nltk.data.load('tokenizers/punkt/english.pickle')
def txt2sentences(txt, remove_none_english_chars=True):
    """
    Split the English text into sentences using NLTK
    :param txt: input text.    
    :param remove_none_english_chars: if True then remove none English chars from text
    :return: string in which each line consists of single sentence from the original input text.
    :rtype: str
    """        
    txt = txt.decode("utf8") 
    # split text into sentences using nltk packages
    for s in tokenizer.tokenize(txt):
        if remove_none_english_chars:
            #remove none English chars
            s = re.sub("[^a-zA-Z]", " ", s)
        yield s
        
sf = gl.load_sframe("%s/books.sframe" % BASE_DIR)
sf['sentences'] = sf['text'].apply(lambda t: list(txt2sentences(t)))

sf_sentences = sf.flat_map(['title', 'text'], lambda t: [[t['title'],s.strip()] for s in txt2sentences(t['text'])])
sf_sentences = sf_sentences.rename({'text': 'sentence'})
re_words_split = re.compile("(\w+)")

#split each sentence into words
sf_sentences['words'] = sf_sentences['sentence'].apply(lambda s:re_words_split.findall(s))
sf_sentences.save("%s/sentences.sframe" % BASE_DIR)
sf_sentences.head(3)

"""
Explanation: According to the simple linear regression, we have the following equation:
$$sherlock_{count} = 2.1404*watson_{count} + 5.8972$$
There are a lot of other really interesting insights that one can discover using similar methodology. I leave the reader to discover these insights by themselves. Let's move to the next section
and create some nice graphs using various entity extraction tools.
<a id="sn"></a>3. Constructing Social Networks
<pre>
<i>"Listen, what I said before John, I meant it. I don’t have friends; I’ve just got one."
                                               -Sherlock, The Hounds of Baskerville, 2012 </i>
</pre>

One of my main fields of interest are social networks. I love to study and visualize graphs of various types of networks. One of the nice studies that I read not long ago showes that it is possible to create the social network of book characters. For example, Miranda et al. built and analyzed a social network utilizing the Odyssey of Homer. 
To manually create a precise social network between Sherlock Holmes characters, we can read the stories and whenever two characters have a conversation, or appear in the same scene, we add to the network nodes with the two characters names (if there are not in the network already), and create a link between the two characters. In case, we want to create a weighted social network, we can also add a weight to each link with the number of times each two characters talked to each other. 
When processing a large text corpus, manually using this process to construct a social network can very time-consuming. Therefore, we would like to perform this process automatically. One of the ways to consturct the social network is by using various NLP algorithms that analyze the text and "understand" the relationships between two entities. However, I am not familiar with open source tools that can analyze a text corpus and infer the connections between two entities with high precision. 
In this section, I will demonstrate some very simple techniques that can be utilized to study the social connections among characters in Sherlock Holmes stories. These techniques won't create the most precise social network. However, the created network is sufficient to observe some interesting insights about the relationships among the stories' characters.
3.1 Constructing Social Network using Names List
Using this techniques, we will split the downloaded Sherlock Holmes stories into sentences, and using a predefined list of names of book characters we will create a social network with links among the stories characters by adding a link between each two characters that appear in the same sentence. Let start constructing the social network by splitting the stories into sentences.
End of explanation
"""


main_characters_set = set(["Irene","Mycroft","Lestrade","Sherlock","Moran","Moriarty","Watson" ])
sf_sentences['characters'] = sf_sentences['words'].apply(lambda w: list(set(w) & main_characters_set))

"""
Explanation: We created a SFrame named <i>sf_sentences</i> in which each row contains a single sentence. Now let's find out which two or more characters from the following link appear in the same sentences. Notice that we only use the characters unique names so we don't mix up between characters with similar names. For example, the name Holmes can represent both Sherlock Holmes and Mycroft Holmes.
End of explanation
"""


import itertools
from collections import Counter
from graphlab import SGraph, Vertex, Edge

def get_characters_graph(sf, min_edge_strength=1):
    """
    Constructs a social network from the an input SFrame. In the social network the verticies are the characters
    and the edges are only between characters that appear in the same sentence at least min_edge_strength times
    :param sf: input SFrame object that contains 'characters' column   
    :param min_edge_strength: minimal connetion strength between two characters.  
    :return: SGraph object constructed from the input SFrame. The graph only contains edges with 
        the at least the input minimal strength between between the characters.
    :rtype: gl.SGraph
    """
    #filter sentences with less than two characters
    sf['characters_num'] = sf['characters'].apply(lambda l: len(l))
    sf = sf_sentences[sf['characters_num'] > 1]
    characters_links = []
    for l in sf['characters']:    
        # if there are more than two characters in the same sentences. Create all link combinations between
        # all the characters (order doesn't matter)
        characters_links += itertools.combinations(l,2)

    #calculating the connections strength between each two characters
    c = Counter(characters_links)
    g = SGraph()

    edges_list = []
    for l,s in c.iteritems():    
        if s < min_edge_strength:
            # filter out connections that appear less than min_edge_strength
            continue
        edges_list.append(Edge(l[0], l[1], attr={'strength':s}))

    g = g.add_edges(edges_list)
    return g

g = get_characters_graph(sf_sentences)
g.show(vlabel="__id", elabel="strength", node_size=200)

"""
Explanation: Now, the <i>'characters'</i> column contain the names of the main characters that appear in the same sentences together. Let's use this information to create the characters social network by constructing a SGraph object.
End of explanation
"""


minor_characters_set = set(["Irene","Mycroft","Lestrade","Sherlock","Moran","Moriarty","Watson","Baynes","Billy","Bradstreet","Gregson"
                            ,"Hopkins","Hudson","Shinwell","Athelney","Mary","Langdale","Toby","Wiggins"])

sf_sentences['characters'] = sf_sentences['words'].apply(lambda w: list(set(w) & minor_characters_set))
sf_sentences['characters_num'] = sf_sentences['characters'].apply(lambda l: len(l))
sf_sentences = sf_sentences[sf_sentences['characters_num'] > 1]

g = get_characters_graph(sf_sentences)
g.show(vlabel="__id", elabel="strength", node_size=200)

"""
Explanation: According to Sherlock's social network, it can be noticed that Sherlock has two main social circles. The first one is circle of friends that include Mycroft and Lestrade. Additionally, he has a circle of enemies that include Moriaty and Moran. Additionally, we can notice Watson is strongly connected to Sherlock and Sherlock's nemesis Moriaty. <br>Let's repeat the experiments only this time we also add minor characters from the following link.
End of explanation
"""


from nltk.tag import StanfordNERTagger

sf_books =  gl.load_sframe("%s/books.sframe" % BASE_DIR)

#IMPORTANT: The directory that include the Stanford Named Entity Recognizer files it need to be updated according 
# to the local installation directory
#STANFORD_DIR = BASE_DIR + "/stanford-ner-2015-06-16/"

#need to insert as parameters the stanford-ner.jar and the type of classifier we want to use
st = StanfordNERTagger('/Users/pablo/fromHomeMac/sherlock/stanford-ner-2015-04-20/classifiers/english.all.3class.distsim.crf.ser.gz',
                       '/Users/pablo/Downloads/stanford-ner-2015-04-20/stanford-ner.jar')
st.java_options = "-Xmx4096m"

sf_books['sentences'] = sf_books['text'].apply(lambda t: list(txt2sentences(t)))
sf_books['words'] = sf_books['sentences'].apply(lambda l: [re_words_split.findall(s) for s in l])
sf_books['NER'] = sf_books['words'].apply(lambda w: st.tag_sents(w))
sf_books['person'] = sf_books['NER'].apply(lambda n: [e[0] for s in n for e in s if e[1] == 'PERSON'])

person_list = []
for p in sf_books['person']:
    person_list += p
    
print len(set(person_list))

from collections import Counter
c = Counter(person_list)
# We are removing some mistken classified words, too common names, and etc. to make the constructed social network
# more readable.
characters_set = set(i[0] for i in c.most_common(200)) - set(['the', 'You', 'Mrs', 'He', 'Dr', 'me','did', 'Mr', 
                                      'Now', 'My', 'Miss', 'of', 'Sir', 'Here', 'All', 'Our', 'sir',
                                      'man', 'father', 'What', 'There', 'When', 'no', 'Lord', 'you', 'St',
                                      'John', 'James',  'Holmes', 'Arthur', 'Conan', 'Doyle', 'Lady'])

sf_sentences = gl.load_sframe("%s/sentences.sframe" % BASE_DIR)
sf_sentences['characters'] = sf_sentences['words'].apply(lambda w: list(set(w) & characters_set))
sf_sentences['characters_num'] = sf_sentences['characters'].apply(lambda l: len(l))
sf_sentences = sf_sentences[sf_sentences['characters_num'] > 1]
g = get_characters_graph(sf_sentences, min_edge_strength=3)
print g.summary()
g.show(vlabel="__id", elabel="strength", node_size=200)


# adding a function to clean the graph as in some cases the Stanford NER maps 'I' as a person.
def clean_graph(g, remove_entities_set):
    vertices = g.vertices[g.vertices["__id"].apply(lambda v: v not in remove_entities_set)] 
    edges = g.edges[g.edges.apply(lambda e: e["__src_id"] not in remove_entities_set and e["__dst_id"] not in remove_entities_set)]
    return gl.SGraph(vertices, edges)

#cleaning the graph and displaying it again
g = clean_graph(g, {"I"})
g.show(vlabel="__id", elabel="strength", node_size=200)

"""
Explanation: We got a more complex social network with the additional minor characters.
I believe this social network graph can be improved by increasing the scope of characters search from single sentence to multiple sentences, or by using characters additional names and nick names. I leave the reader to try to improve the graph by themselves.
3.2 Constructing Social Network using Named Entity Recognition
One of the disadvantages of the above method is that you need a predefined list of names to create the social network. However, in many cases this list is unavailable. Therefore, we need another method to find entities in the text. One common method to achieve this is using Named Entity Recognition (NER). By using NER algorithms, we can classify elements in the text into pre-defined categories, such as the names of persons, organizations, and locations. There are many tools that can perform NER, such as OpenNLP, Stanford Named Entity Recognizer, Rosette Entity Extractor. In this notebook, we will use the Stanford Named Entity Recognizer via NLTK. We will use NER algorithms to automatically  construct an entity list of the most common characters of the book.
Please note that making NLTK run Stanford Named Entity Recognizer can be non-trivial. For more details, on how to make NLTK work with Stanford Named Entity Recognizer please read the information provided in the following links 1,2, & 3. 
NOTE: running the next code section can take several minutes.
End of explanation
"""


def get_graph_largest_compnent(g):
    """
    Returns a graph with the largest component of the input graph
    :param g: input graph (SGraph object)
    :return: a graph of the largest component in the input object
    :rtype: gl.SGraph
    """
    
    cc = gl.connected_components.create(g)    
    #add each vertices its component id
    g.vertices['component_id'] = cc['graph'].vertices['component_id']
    # calculate the component id of the largest component
    largest_component_id = cc['component_size'].sort('Count', ascending=False)[0]['component_id']
    largest_component_verticies = g.vertices.filter_by(largest_component_id, 'component_id')['__id']
    h = g.get_neighborhood(largest_component_verticies, 1)
    return h

h = get_graph_largest_compnent(g)  
h.show(vlabel="__id", elabel="strength", node_size=300)

h = clean_graph(g, {"I"})
h.show(vlabel="__id", elabel="strength", node_size=300)

"""
Explanation: The NER algorithm did pretty good job, and most of the names of the identified entities looks logical (at least to me).
Additionally, we can understand the link between the various book characters. We can also notice that in many of the graph's components that have only two vertices  the connection is between each characfter first and it's last names. Let use GraphLab graph_analytics toolkit and focus on the social network's largest component.
End of explanation
"""


import graphlab as gl
import re

sf =  gl.load_sframe("%s/books.sframe" % BASE_DIR)
sf_paragraphs = sf.flat_map(['title', 'text'], lambda t: [[t['title'],p.strip()] for p in t['text'].split("\n\n")])
sf_paragraphs = sf_paragraphs.rename({'text': 'paragraph'})

"""
Explanation: According the above graph that can be created almost automatically. We can easily identify the main characters in the stories. Additionally, we can observe that the strongest connection is between Sherlock and Watson.
Moreover, we can see various connections among the main and minor characters of the book. However, from only looking at the graph, it is non-trivial to understand the various communities and their relationships.
Using similar methods, we can learn more on each character by finding connections among person and location and person and organization. I leave the reader to find additional insights on the various characters on their own.
<a id="topic"></a> 4. Topic Model
<pre>
<i>"I have known him for some time," said I, "but I never knew him do anything yet without a very good reason, and with that our conversation drifted off on to other topics.</i>
                                                                 -Memoirs of Sherlock Holmes
</pre>

According to Wikipedia article, topic model is a type of statistical model for discovering the abstract "topics" that occur in a collection of documents. I personally find topic models an interesting tool to explore a large text corpus. In this section, we are going to demonstrate how it is possible to utilze GraphLab's topic model toolkit with the pyLDAvis package to uncover topics in a set of documents. Namely, we will use GraphLab's topic model toolkit to analyze paragraphs in Sherlock Holmes stories. 
We will start by separating each story into paragraphs.
End of explanation
"""


re_words_split = re.compile("(\w+)")
sf_paragraphs['paragraph_words_number'] = sf_paragraphs['paragraph'].apply(lambda p: len(re_words_split.findall(p)) )
sf_paragraphs = sf_paragraphs[sf_paragraphs['paragraph_words_number'] >=25]

"""
Explanation: Let's calculate the number of words in each paragraph, and filter the paragraph that have less than 25 words.
End of explanation
"""


docs =  gl.text_analytics.count_ngrams(sf_paragraphs['paragraph'], n=1)
stopwords = gl.text_analytics.stopwords()
# adding some additional stopwords to make the topic model more clear
stopwords |= set(['man', 'mr', 'sir', 'make', 'made', 'll', 'door', 'long', 'day', 'small']) 
docs = docs.dict_trim_by_keys(stopwords, exclude=True)	
docs = docs.dropna()
topic_model = gl.topic_model.create(docs, num_topics=10)

"""
Explanation: Using the stories' paragraphs as documents, we can utilize GraphLab's topic model toolkit to discover topics that appear in these paragraph.
We create a topic model with 10 topics to learn.
Note: the topic model results may be different in each run.
End of explanation
"""


topic_model.get_topics().print_rows(100)
topic_model.save("%s/topic_model" % BASE_DIR)

"""
Explanation: Let's view the most common word in each topic
End of explanation
"""


import pyLDAvis
import pyLDAvis.graphlab
pyLDAvis.enable_notebook()
pyLDAvis.graphlab.prepare(topic_model, docs)

"""
Explanation: Reading the above table, we can understand some of the topics. However, it still hard to get good overall overview.
Therefore, we will use the excellent pyLDAvis package, developed by Ben Mabey,
to better the various topics in the books.
End of explanation
"""


import graphlab as gl
import urllib2
import gensim
import nltk
import re

txt = urllib2.urlopen("https://sherlock-holm.es/stories/plain-text/cnus.txt").read()

re_words_split = re.compile("(\w+)")
tokenizer = nltk.data.load('tokenizers/punkt/english.pickle')
def txt2words(s):
    s = re.sub("[^a-zA-Z]", " ", s).lower()
    return re_words_split.findall(s)

class MySentences(object):
        def __init__(self, txt):
            self._txt = txt.decode("utf8") 
            
        def __iter__(self):
            """
            Split the English text into sentences and then to words using NLTK
            :param txt: input text.    
            :param remove_none_english_chars: if True then remove none English chars from text
            :return: list of words in which each list consists of single sentence's words from the original input text.
            :rtype: str
            """                
            # split text into sentences using NLTK package
            for s in tokenizer.tokenize(self._txt):                                    
                yield txt2words(s)

sentences = MySentences(txt)
model = gensim.models.Word2Vec(sentences, size=100, window=5, min_count=3, workers=4)

"""
Explanation: From the above visualization, we can observe that the algorithm returned pretty interesting results. For example, one identified topic is related to Watson, locations (room, street, house, etc.), and time (days, hours, etc.). While, another topic is related to Holmes, men, and murder. For me these are pretty interesting results. I recommend the reader to try investigate the results by themselves. Moreover, I think that running the topic model algorithm on other text corpus can help to better understand this algorithms advantages.
<a id="word2vec"></a> 5. Finding SImilar Paragraphs using Word2Vec
<pre>
"I have notes of several similar cases, though none, as I remarked before, which were quite as prompt. My whole examination served to 
turn my conjecture into a certainty. Circumstantial evidence is occasionally very convincing, as when you find a trout in the milk, to 
quote Thoreau's example."
                                                                                                 -The Adventure of the Noble Bachelor
</pre>

These days, no NLP related post can be complete without including the words "deep learning." Therefore, in this section I will demonstrate how to use Word2Vec deep learning inspired algorithm to search for paragraphs that have similar text or writing style. 
First, let's build a Word2Vec model using Sherlock's stories. We will construct the Word2Vec model using the Gensim package and a similar method to the one presented in Word2vec Tutorial and in my previous post.
End of explanation
"""


print model.most_similar("watson")
print model.most_similar("holmes")

"""
Explanation: We now have a trained Word2Vec model, let's see if it gives reasonable results:
End of explanation
"""


import graphlab as gl
import re
import numpy as np

BASE_DIR = r"/Users/pablo/fromHomeMac/sherlock/data" # NOTE: Update BASE_DIR to your own directory path
sf =  gl.load_sframe("%s/books.sframe" % BASE_DIR)
sf_paragraphs = sf.flat_map(['title', 'text'], lambda t: [[t['title'],p.strip()] for p in t['text'].split("\n\n")])
sf_paragraphs = sf_paragraphs.rename({'text': 'paragraph'})
sf_paragraphs['paragraph_words_number'] = sf_paragraphs['paragraph'].apply(lambda p: len(re_words_split.findall(p)) )
sf_paragraphs = sf_paragraphs[sf_paragraphs['paragraph_words_number'] >=25]

def txt2avg_vector(txt, w2v_model):
    words = [w for w in txt2words(txt.lower()) if w in w2v_model]
    v = np.mean([w2v_model[w] for w in words],axis=0)    
    return v

sf_paragraphs['mean_vector'] = sf_paragraphs['paragraph'].apply(lambda p: txt2avg_vector(p, model))

"""
Explanation: We got that the most similar word to Watson is Mortimer and the most similar word to Holmes is Lestrade. These results sound logical enough. Let us calculate the average vector of each paragraph.
End of explanation
"""


#construncting nearest neighbors model
nn_model = gl.nearest_neighbors.create(sf_paragraphs, features=['mean_vector'])

#calaculating the two nearest neighbors of each paragraph from all the paragraphs 
r = nn_model.query(sf_paragraphs, k=2)
r.head(10)

"""
Explanation: Now we have the mean vector value of each paragraph. Let's utilize GraphLab Create nearest neighbors toolkit to identify paragraphs that have similar text or  writing style. We will acheive that by calaculating the nearest neighbor to each the mean vector of each paragraph.
End of explanation
"""


#filter out paragraphs that are exactly exactly the same
r = r[r['distance'] != 0]

#filter out paragraphs that are with distance >= 0.1
r = r[r['distance'] < 0.08]
r

"""
Explanation: Of course the nearest neighbors to each paragraph is the paragraph itself. Therefore, let us filter out paragraph that are with a distance of zero from each other. Additionally, let's look only on two near paragraphs that have small distance from each other (distance < 0.08)
End of explanation
"""


sf_paragraphs = sf_paragraphs.add_row_number('query_label')
sf_paragraphs = sf_paragraphs.add_row_number('reference_label')
sf_similar = r.join(sf_paragraphs, on="query_label").join(sf_paragraphs, on="reference_label")

sf_similar[['paragraph','title', 'title.1', 'paragraph.1', 'distance']]

"""
Explanation: Now, let's use join to match between each query_label and reference_label values and their actual paragraphs.
End of explanation
"""


print sf_similar[1]['paragraph']
print "-"*100
print sf_similar[1]['paragraph.1']

"""
Explanation: Let's look at some of the similar paragraphs.
End of explanation
"""


