import nltk
#nltk.download('webtext')
document = nltk.corpus.webtext.open('grail.txt').read()

"""
Explanation: Text Data
Pre-introduction
We'll be spending a lot of time today manipulating text. Make sure you remember how to split, join, and search strings.
Introduction
We've spent a lot of time in python dealing with text data, and that's because text data is everywhere. It is the primary form of communication between persons and persons, persons and computers, and computers and computers. The kind of inferential methods that we apply to text data, however, are different from those applied to tabular data. 
This is partly because documents are typically specified in a way that expresses both structure and content using text (i.e. the document object model).
Largely, however, it's because text is difficult to turn into numbers in a way that preserves the information in the document. Today, we'll talk about dominant language model in NLP and the basics of how to implement it in Python.
The term-document model
This is also sometimes referred to as "bag-of-words" by those who don't think very highly of it. The term document model looks at language as individual communicative efforts that contain one or more tokens. The kind and number of the tokens in a document tells you something about what is attempting to be communicated, and the order of those tokens is ignored.
To start with, let's load a document.
End of explanation
"""


len(document.split('\n'))

document.split('\n')[0:10]

"""
Explanation: Let's see what's in this document
End of explanation
"""


import re
snippet = "I fart in your general direction! Your mother was a hamster, and your father smelt of elderberries!"
re.search(r'mother', snippet)

"""
Explanation: It looks like we've gotten ourselves a bit of the script from Monty Python and the Holy Grail. Note that when we are looking at the text, part of the structure of the document is written in tokens. For example, stage directions have been placed in brackets, and the names of the person speaking are in all caps.
Regular expressions
If we wanted to read out all of the stage directions for analysis, or just King Arthur's lines, doing so in base python string processing will be very difficult. Instead, we are going to use regular expressions. Regular expressions are a method for string manipulation that match patterns instead of bytes.
End of explanation
"""


re.search(r'[a-z]', snippet)

"""
Explanation: Just like with str.find, we can search for plain text. But re also gives us the option for searching for patterns of bytes - like only alphabetic characters.
End of explanation
"""


re.search(r'[a-z]!', snippet)

"""
Explanation: In this case, we've told re to search for the first sequence of bytes that is only composed of lowercase letters between a and z. We could get the letters at the end of each sentence by including a bang at the end of the pattern.
End of explanation
"""


re.findall(r'[a-zA-Z]', document)[0:10]

"""
Explanation: If we wanted to pull out just the stage directions from the screenplay, we might try a pattern like this:
End of explanation
"""


re.findall(r'\[[a-zA-Z]+\]', document)[0:10]

"""
Explanation: So that's obviously no good. There are two things happening here:

[ and ] do not mean 'bracket'; they are special characters which mean 'any thing of this class'
we've only matched one letter each

A better regular expression, then, would wrap this in escaped brackets, and include a command saying more than one letter.
Re is flexible about how you specify numbers - you can match none, some, a range, or all repetitions of a sequence or character class.
character | meaning
----------|--------
{x}     | exactly x repetitions
{x,y}   | between x and y repetitions
?       | 0 or 1 repetition
*       | 0 or many repetitions
+       | 1 or many repetitions
End of explanation
"""


snippet = 'This is [cough cough] and example of a [really] greedy operator'
re.findall(r'\[.+\]', snippet)

"""
Explanation: This is better, but it's missing that [clop clop clop] we saw above. This is because we told the regex engine to match any alphabetic character, but we did not specify whitespaces, commas, etc. to match these, we'll use the dot operator, which will match anything expect a newline.
Part of the power of regular expressions are their special characters. Common ones that you'll see are:
character | meaning
----------|--------
.       | match anything except a newline
^       | match the start of a line
$       | match the end of a line
\s      | matches any whitespace or newline
Finally, we need to fix this + character. It is a 'greedy' operator, which means it will match as much of the string as possible. To see why this is a problem, try:
End of explanation
"""


p = re.compile(r'\[.+?\]')
re.findall(p, document)[0:10]

"""
Explanation: Since the operator is greedy, it is matching everything inbetween the first open and the last close bracket. To make + consume the least possible amount of string, we'll add a ?.
End of explanation
"""


p = re.compile(r'(?:ARTHUR: )(.+)')
re.findall(p, document)[0:10]

"""
Explanation: What if we wanted to grab all of Arthur's speech? This one is a little trickier, since:

It is not conveniently bracketed; and,
We want to match on ARTHUR, but not to capture it

If we wanted to do this using base string manipulation, we would need to do something like:
split the document into lines
create a new list of just lines that start with ARTHUR
create a newer list with ARTHUR removed from the front of each element
Regex gives us a way of doing this in one line, by using something called groups. Groups are pieces of a pattern that can be ignored, negated, or given names for later retrieval.
character | meaning
----------|--------
(x)     | match x
(?:x)   | match x but don't capture it
(?P&lt;x&gt;) | match something and give it name x
(?=x)   | match only if string is followed by x
(?!x)   | match only if string is not followed by x
End of explanation
"""


p = re.compile(r'(?P<name>[A-Z ]+)(?::)(?P<line>.+)')
match = re.search(p, document)
match

match.group('name'), match.group('line')

"""
Explanation: Because we are using findall, the regex engine is capturing and returning the normal groups, but not the non-capturing group. For complicated, multi-piece regular expressions, you may need to pull groups out separately. You can do this with names.
End of explanation
"""


p = re.compile(r'(?:ARTHUR: )(.+)')
arthur = ' '.join(re.findall(p, document))
arthur[0:100]

"""
Explanation: Now let's try a small challenge!
To check that you've understood something about regular expressions, we're going to have you do a small test challenge. Partner up with the person next to you - we're going to do this as a pair coding exercise - and choose which computer you are going to use.
Then, navigate to challenges/03_analysis/ and read through challenge A. When you think you've completed it successfully, run py.test test_A.py .
Tokenizing
Let's grab Arthur's speech from above, and see what we can learn about Arthur from it.
End of explanation
"""


p = re.compile(r'\w+', flags=re.I)
re.findall(p, arthur)[0:10]

"""
Explanation: In our model for natural language, we're interested in words. The document is currently a continuous string of bytes, which isn't ideal. You might be tempted to separate this into words using your newfound regex knowledge:
End of explanation
"""


re.findall(p, "It isn't Dav's cheesecake that I'm worried about")

"""
Explanation: But this is problematic for languages that make extensive use of punctuation. For example, see what happens with:
End of explanation
"""


from nltk import word_tokenize
word_tokenize("It isn't Dav's cheesecake that I'm worried about")

"""
Explanation: The practice of pulling apart a continuous string into units is called "tokenizing", and it creates "tokens". NLTK, the canonical library for NLP in Python, has a couple of implementations for tokenizing a string into words.
End of explanation
"""


tokens = word_tokenize(arthur)
tokens[0:10]

"""
Explanation: The distinction here is subtle, but look at what happened to "isn't". It's been separated into "IS" and "N'T", which is more in keeping with the way contractions work in English.
End of explanation
"""


len(tokens), len(set(tokens))

"""
Explanation: At this point, we can start asking questions like what are the most common words, and what words tend to occur together.
End of explanation
"""


from nltk import collocations
fd = collocations.FreqDist(tokens)
fd.most_common()[:10]

measures = collocations.BigramAssocMeasures()
c = collocations.BigramCollocationFinder.from_words(tokens)
c.nbest(measures.pmi, 10)

c.nbest(measures.likelihood_ratio, 10)

"""
Explanation: So we can see right away that Arthur is using the same words a whole bunch - on average, each unique word is used four times. This is typical of natural language. 

Not necessarily the value, but that the number of unique words in any corpus increases much more slowly than the total number of words.
A corpus with 100M tokens, for example, probably only has 100,000 unique tokens in it.

For more complicated metrics, it's easier to use NLTK's classes and methods.
End of explanation
"""


snowball = nltk.SnowballStemmer('english')

"""
Explanation: We see here that the collocation finder is pulling out some things that have face validity. When Arthur is talking about peasants, he calls them "bloody" more often than not. However, collocations like "Brother Maynard" and "BLACK KNIGHT" are less informative to us, because we know that they are proper names.
If you were interested in collocations in particular, what step do you think you would have to take during the tokenizing process?
Stemming
This has gotten us as far identical tokens, but in language processing, it is often the case that the specific form of the word is not as important as the idea to which it refers. For example, if you are trying to identify the topic of a document, counting 'running', 'runs', 'ran', and 'run' as four separate words is not useful. Reducing words to their stems is a process called stemming.
A popular stemming implementation is the Snowball Stemmer, which is based on the Porter Stemmer. It's algorithm looks at word forms and does things like drop final 's's, 'ed's, and 'ing's.
Just like the tokenizers, we first have to create a stemmer object with the language we are using.
End of explanation
"""


snowball.stem('running')

snowball.stem('eats')

snowball.stem('embarassed')

"""
Explanation: Now, we can try stemming some words
End of explanation
"""


snowball.stem('cylinder'), snowball.stem('cylindrical')

"""
Explanation: Snowball is a very fast algorithm, but it has a lot of edge cases. In some cases, words with the same stem are reduced to two different stems.
End of explanation
"""


snowball.stem('vacation'), snowball.stem('vacate')

snowball.stem('organization'), snowball.stem('organ')

snowball.stem('iron'), snowball.stem('ironic')

snowball.stem('vertical'), snowball.stem('vertices')

"""
Explanation: In other cases, two different words are reduced to the same stem.

This is sometimes referred to as a 'collision'
End of explanation
"""


# nltk.download('wordnet')
wordnet = nltk.WordNetLemmatizer()

wordnet.lemmatize('iron'), wordnet.lemmatize('ironic')

wordnet.lemmatize('vacation'), wordnet.lemmatize('vacate')

"""
Explanation: A more accurate approach is to use an English word bank like WordNet to call dictionary lookups on word forms, in a process called lemmatization.
End of explanation
"""


%timeit wordnet.lemmatize('table')

4.45 * 5.12

%timeit snowball.stem('table')

"""
Explanation: Nothing comes for free, and you've probably noticed already that the lemmatizer is slower. We can see how much slower with one of IPYthon's magic functions.
End of explanation
"""


from textblob import TextBlob

blob = TextBlob(arthur)

for sentence in blob.sentences[10:25]:
    print(sentence.sentiment.polarity, sentence)

"""
Explanation: Time for another small challenge!
Switch computers for this one, so that you are using your partner's computer, and try your hand at challenge B!
Sentiment
Frequently, we are interested in text to learn something about the person who is speaking. One of these things we've talked about already - linguistic diversity. A similar metric was used a couple of years ago to settle the question of who has the largest vocabulary in Hip Hop.

Unsurprisingly, top spots go to Canibus, Aesop Rock, and the Wu Tang Clan. E-40 is also in the top 20, but mostly because he makes up a lot of words; as are OutKast, who print their lyrics with words slurred in the actual typography

Another thing we can learn is about how the speaker is feeling, with a process called sentiment analysis. Before we start, be forewarned that this is not a robust method by any stretch of the imagination. Sentiment classifiers are often trained on product reviews, which limits their ecological validity.
We're going to use TextBlob's built-in sentiment classifier, because it is super easy.
End of explanation
"""


from gensim import corpora, models, similarities

"""
Explanation: Semantic distance
Another common NLP task is to look for semantic distance between documents. This is used by search engines like Google (along with other things like PageRank) to decide which websites to show you when you search for things like 'bike' versus 'motorcycle'.
It is also used to cluster documents into topics, in a process called topic modeling. The math behind this is beyond the scope of this course, but the basic strategy is to represent each document as a one-dimensional array, where the indices correspond to integer ids of tokens in the document. Then, some measure of semantic similarity, like the cosine of the angle between unitized versions of the document vectors, is calculated.
Luckily for us there is another python library that takes care of the heavy lifting for us.
End of explanation
"""


p = re.compile(r'(?:GALAHAD: )(.+)')
galahad = ' '.join(re.findall(p, document))
arthur_tokens = tokens
galahad_tokens = word_tokenize(galahad)

"""
Explanation: We already have a document for Arthur, but let's grab the text from someone else to compare it with.
End of explanation
"""


dictionary = corpora.Dictionary([arthur_tokens, galahad_tokens])
corpus = [dictionary.doc2bow(doc) for doc in [arthur_tokens, galahad_tokens]]
tfidf = models.TfidfModel(corpus, id2word=dictionary)

"""
Explanation: Now, we use gensim to create vectors from these tokenized documents:
End of explanation
"""


query = tfidf[dictionary.doc2bow(['peasant'])]
index = similarities.MatrixSimilarity(tfidf[corpus])

"""
Explanation: Then, we create matrix models of our corpus and query
End of explanation
"""


list(enumerate(index[query]))

"""
Explanation: And finally, we can test our query, "peasant" on the two documents in our corpus
End of explanation
"""


import pandas as pd

"""
Explanation: So we see here that "peasant" does not match Galahad very well (a really bad match would have a negative value), and is more similar to the kind of speach output that we see from King Arthur.
Tabular data
In data storage, data visualization, inferential statistics, and machine learning, the most common way to pass data between applications is in the form of tables (these are called tabular, structured, or rectangular data). These are convenient in that, when used correctly, they store data in a DRY and easily queryable way, and are also easily turned into matrices for numeric processing.

note - it is sometimes tempting to refer to N-dimensional matrices as arrays, following the numpy naming convention, but these are not the same as arrays in C++ or Java, which may cause confusion

It is common in enterprise applications to store tabular data in a SQL database. In the sciences, data is typically passed around as comma separated value files (.csv), which you have already been dealing with over the course of the last two days.
For this brief introduction to analyzing tabular data, we'll be using the scipy stack, which includes numpy, pandas, scipy, and "scikits" like sk-learn and sk-image.
End of explanation
"""


table = pd.DataFrame({'id': [1,2,3], 'name':['dillon','juan','andrew'], 'age':[47,27,23]})
print(table)

"""
Explanation: You might not have seen this as convention yet. It is just telling python that when we import pandas, we don't want to access it in the namespace as pandas but as pd instead.
Pandas basics
We'll start by making a small table to practice on. Tables in pandas are called data frames, so we'll start by making an instance of class DataFrame, and initialize it with some data.

note - pandas and R use the same name for their tables, but their behavior is often very different
End of explanation
"""


table['name'], type(table['name'])

"""
Explanation: Variables in pandas are represented by a pandas-specific data structure, called a Series. You can grab a Series out of a DataFrame by using the slicing operator with the name of the variable that you want to pull.
End of explanation
"""


table['fingers'] = [9, 10, None]

"""
Explanation: We could have made each variable a Series, and then put it into the DataFrame object, but it's easier in this instance to pass in a dictionary where the keys are variable names and the values are lists. You can also modify a data frame in place using similar syntax:
End of explanation
"""


table['fingers']

"""
Explanation: If you try to run that code without the None there, pandas will return an error. In a table (in any language) each column must have the same number of rows.
We've entered None, base python's missingness indicator, but pandas is going to swap this out with something else:
End of explanation
"""


table['fingers'][2] == None

table['fingers'][2] == 'NaN'

type(table['fingers'][2]) == str

"""
Explanation: You might be tempted to write your own control structures around these missing values (which are variably called NaN, nan, and NA), but this is always a bad idea:
End of explanation
"""


pd.isnull(table['fingers'])

"""
Explanation: None of this works because the pandas NaN is a subclass of numpy's double precision floating point number. However, for ambiguous reasons, even numpy.nan does not evaluate as being equal to itself.
To handle missing data, you'll need to use the pandas method isnull.
End of explanation
"""


table[:1]

"""
Explanation: In the same way that we've been pulling out columns by name, you can pull out rows by index. If I want to grab the first row, I can use:
End of explanation
"""


table.sort_values('age')

"""
Explanation: Recall that indices in python start at zero, and that selecting by a range does not include the final value (i.e. [ , )).
Unlike other software languages (R, I'm looking at you here), row indices in pandas are immutable. So, if I rearrange my data, the index also get shuffled.
End of explanation
"""


table[table['age'] < 40]

"""
Explanation: Because of this, it's common to set the index to be something like a timestamp or UUID.
We can select parts of a DataFrame with conditional statements:
End of explanation
"""


other_table = pd.DataFrame({
        'name':['dav', 'juan', 'dillon'], 
        'languages':['python','python','python']})

table.merge(other_table, on='name')

"""
Explanation: Merging tables
As you might expect, tables in pandas can also be merged by keys. So, if we make a new dataset that shares an attribute in common:
End of explanation
"""


table.merge(other_table, on='name', how='outer')

"""
Explanation: Note that we have done an "inner join" here, which means we are only getting the intersection of the two tables. If we want the union, we can specify that we want an outer join:
End of explanation
"""


table.merge(other_table, on='name', how='left')

"""
Explanation: Or maybe we want all of the data from table, but not other_table
End of explanation
"""


wide_table = pd.DataFrame({'name' : ['dillon', 'juan', 'dav'],
                           'city1' : ['williamsburg', 'berkeley', 'cambridge'],
                           'city2' : ['berkeley', 'berkeley', 'berkeley'],
                           'population' : [110, 110, 110]
                          })
wide_table

"""
Explanation: Reshaping
To make analysis easier, you may have to reshape your data. It's easiest to deal with data when each table meets the follwing criteria:

Each row is exactly one observation
Each column is exactly one kind of data
The table expresses one and only one relationship between observations and variables

This kind of format is easy to work with, because:

It's easy to update when every piece of data exists in one and only one place
It's easy to subset conditionally across rows
It's easy to test across columns

To make this more concrete, let's take an example table.
name   | city1 | city2 | population
-------|-------|-------|-----------
dillon | williamsburg | berkeley | 110
juan   | berkeley | berkeley | 110
dav    | cambridge | berkeley | 110
This table violates all three of the rules above. Specifically, it:

each row is about two observations
two columns are about the same kind of date (city), while another datatype (time) has been hidden in the column names
it expresses the relationship between people and where they live; and, cities and their population

In this particular example, our data is too wide. If we create that dataframe in pandas
End of explanation
"""


long_table = pd.melt(wide_table, id_vars = ['name'])
long_table

"""
Explanation: We can make this longer in pandas using the melt function
End of explanation
"""


long_table.pivot(columns='variable')

"""
Explanation: We can make the table wider using the pivot method

side note - this kind of inconsistency between melt and pivot is un-pythonic and should not be emulated
End of explanation
"""


long_table.pivot(columns='variable', values='value')

"""
Explanation: WHOA
One of the really cool things about pandas is that it allows you to have multiple indexes for rows and columns. Since pandas couldn't figure out what do with two kinds of value variables, it doubled up our column index. We can fix this by specifying that we only want the 'values' values
End of explanation
"""


table['fingers'].mean()

table['fingers'].std()

table['fingers'].quantile(.25)

table['fingers'].kurtosis()

"""
Explanation: Challenge time!
Switch computers again so that you are working on the first computer of the day, and have a look at challenge C. This will have you practice reading and merging tables. Again, when you are finished, check your work by running py.test test_C in a shell.
Descriptive statistics
Single descriptives have their own method calls in the Series class.
End of explanation
"""


table.describe()

"""
Explanation: You can call several of these at once with the describe method
End of explanation
"""


from scipy import stats
data = pd.read_csv('../data/03_feedback.csv')

"""
Explanation: Inferential statistics
pandas does not have statistical functions baked in, so we are going to call them from the scipy.stats library and the statmodels scikit.
We are also going to load in an actual dataset, as stats examples aren't very interesting with tiny bits of fake data.
End of explanation
"""


i = data['inside.barriers'].dropna()
o = data['outside.barriers'].dropna()
stats.ttest_ind(i, o)

"""
Explanation: Using what you've learned so far about manipulating pandas objects, how would you find out the names of the variables in this dataset? Their datatypes? The distribution of their values?
Comparisons of group means
A common statistical procedure is to look for differences between groups of values. Typically, the values are grouped by a variable of interest, like sex or age. Here, we are going to compare the barriers of access to technology that people experience in the D-Lab compared to the world outside.
If you only have two groups in your sample, you can use a t-test:
End of explanation
"""


m = data[data.gender == "Male/Man"]['outside.barriers'].dropna()
f = data[data.gender == "Female/Woman"]['outside.barriers'].dropna()
q = data[data.gender == "Genderqueer/Gender non-conforming"]['outside.barriers'].dropna()
stats.f_oneway(m, f, q)

"""
Explanation: Notice that here, we are passing in two whole columns, but we could also be subsetting by some other factor.
If you have more than two groups (or levels) that you would like to compare, you'll have to use something like an ANOVA:
End of explanation
"""


intermediate = data.dropna(subset=['inside.barriers', 'outside.barriers'])
stats.pearsonr(intermediate['outside.barriers'], intermediate['inside.barriers'])

"""
Explanation: Linear relationships
Another common task is to establish if/how two variables are related across linear space. This could be something, for example, like relating shoe size to height. Here, we are going to ask whether barriers to access to technology inside and outside of the D-Lab are related.
One implementation of linear relationships is correlation testing:
End of explanation
"""


import statsmodels.formula.api as smf

"""
Explanation: At this point, we're going to pivot to using statsmodels
End of explanation
"""


model_1 = smf.ols("inside_barriers ~ outside_barriers", data=data).fit()
model_1

"""
Explanation: The formulas module in statsmodels lets us work with pandas dataframes, and linear model specifications that are similar to R and other variants of statistical software, e.g.:
outcome ~ var1 + var2
End of explanation
"""


model_1.summary()

"""
Explanation: To get a summary of the test results, call the model's summary method
End of explanation
"""


model_1.params['outside_barriers']

"""
Explanation: Since Python does not have private data or hidden attributes, you can pull out just about any intermediate information you want, including coefficients, residuals, and eigenvalues

Raymond Hettinger would say that Python is a "consenting adult language"
End of explanation
"""


model_1.get_influence().summary_frame()

"""
Explanation: statsmodels also exposes methods for validity checking your regressions, like looking for outliers by influence statistics
End of explanation
"""


model_1.get_robustcov_results().summary()

"""
Explanation: If, at this stage, you suspect that one or more outliers is unduly influencing your model fit, you can transform your results into robust OLS with a method call:
End of explanation
"""


smf.ols("inside_barriers ~ outside_barriers + gender", data=data).fit().summary()

"""
Explanation: This isn't very different, so we're probably okay.
If you want to add more predictors to your model, you can do so inside the function string:
End of explanation
"""


smf.ols("inside_barriers ~ outside_barriers * gender", data=data).fit().summary()

"""
Explanation: Note that our categorical/factor variable has been automatically one-hot encoded as treatment conditions. There's not way to change this within statsmodels, but you can specify your contrasts indirectly using a library called (Patsy)[http://statsmodels.sourceforge.net/stable/contrasts.html].
To add interactions to your model, you can use :, or * [for full factorial]
End of explanation
"""


