import os

open(os.path.join('data_a', 'archeological', '10.2307_104838.txt')).read()

"""
Explanation: Introduction to Classification
The goal of a classification task is to predict whether a given observation in a dataset (e.g. a text in a collection of text files) possesses some particular property or attribute (e.g. was written by a woman). To make these predictions, we generally measure the attributes of several labelled data observations, then compare new unlabelled observations to those measurements.
For example, let's examine a small corpus of text files from the Philosophical Transactions, a scientific periodical published in England since 1665. Each of our samples from this corpus is either an archeological or medical text. Let's look at a sample archeological file:
End of explanation
"""


import os

open(os.path.join('data_a', 'medical', '10.2307_105060.txt')).read()

"""
Explanation: Next let's look at a sample medical file:
End of explanation
"""


import os
import glob
import collections

# specify the files to process - nb: * matches all files and directories
path = os.path.join('data_a', '*', '*.txt')

# iterate over each file matched above
for i in glob.glob(path):
  
  # read the current file
  text = open(i).read()
  
  # split the current file into a list of words
  words = text.split()
  
  # count the number of times each word in the file occurs
  counts = collections.Counter(words)
  
  # classify this file as a mathematical or medical text
  prediction = 'archeological' if counts['bones'] > 5 else 'medical'
  
  # print the prediction
  print(prediction, '--', i)

"""
Explanation: As we can see above, our archeological files are stored in data_a/archeological while our medical files are stored in data_a/medical. Let's keep this in mind below.
Given this data, we might build a classifier to determine whether a given text is from the collection is from the archeological or medical genre. Our first approach to this task will use a simple method--we will simply count the number of times the word "bones" occurs in the text. If that word occurs more than 5 times, we will classify the text as an archeological text; otherwise, we'll classify the text as medical:
End of explanation
"""


%matplotlib inline
import matplotlib.pyplot as plt

x_values = []
y_values = []
colors = []

# specify the files to process - nb: * matches all files and directories
path = os.path.join('data_a', '*', '*.txt')

# iterate over each file matched above
for i in glob.glob(path):
  
  # read the current file
  text = open(i).read()
  
  # add this text's count of the first keyword to the x axis values
  x_values.append( text.count('bones') )
  
  # use the same y axis value for all texts
  y_values.append(0)
  
  # make the dot blue if it's a medical dot, else red if it's a mathematical dot
  colors.append('blue' if 'medical' in i else 'red')
  
# plot the 1D distribution
plt.title('Samples from the Philosophical Transactions')
plt.xlabel('Count of the word "bones"')
plt.ylabel('')
plt.scatter(x_values, y_values, c=colors)

"""
Explanation: These results show our simple classification model does a pretty good job of classifying a text as archeological or medical! As we recall, however, this classifier is based on counts of a single word in each text file. As you might guess, increasing the number of words that the model uses will increase the power of our model. Let's see how to use multiple word counts in a classifier model below.
Working with Multiple Dimensions
In the example above, we used the counts of a single word to classify whether a document was a mathematical or medical text. Another way of stating that fact is to say we used just a single "dimension" of data to perform our classification--where "dimension" here refers to the count of a single word. 
We can plot that single dimension of data as follows:
End of explanation
"""


x_values = []
y_values = []
colors = []

# specify the files to process - nb: * matches all files and directories
path = os.path.join('data_a', '*', '*.txt')

# iterate over each file matched above
for i in glob.glob(path):
  
  # read the current file
  text = open(i).read()
  
  # add this text's count of the first keyword to the x axis values
  x_values.append( text.count('bones') )
  
  # add this text's count of the second keyword to the y axis values
  y_values.append( text.count('fossil') )
  
  # make the dot blue if it's a medical dot, else red if it's a mathematical dot
  colors.append('blue' if 'medical' in i else 'red')
  
# plot the 1D distribution
plt.title('Samples from the Philosophical Transactions')
plt.xlabel('Count of the word "bones"')
plt.ylabel('Count of the word "fossil"')
plt.scatter(x_values, y_values, c=colors)

"""
Explanation: As we can see, the archeological texts tend to have higher counts of the word "bones" than medical texts. If we add the counts of the word "fossil" on the y axis of our plot, we can display a 2D data model:
End of explanation
"""


# type your code here

"""
Explanation: Comparing this plot to the one above it, we can see adding a second "dimension" of data increases the separation between our two document types. This is good, from a model-building perspective, as the greater the separation between our classes, the easier it will be for an algorithm to classify the observations from either class. We'll continue to use this 2D data model as we explore K-Nearest Neighbors classification below.
<h1 style='color:green'>Reviewing Multidimensional Classification</h1>

Using the approach we discussed above, see if you can plot the counts of the words "parallax" and "angle" in data_b/*/*.txt. That folder contains texts that are either astronomical or geometrical. Color the astronomical texts blue and the non-astronomical texts red.
Hint: once again, you may find it helpful to copy the code block we used above and modify that code block!
End of explanation
"""


from sklearn.neighbors import KNeighborsClassifier
import numpy as np
import glob
import os

# create the 2D dataset we used above
counts = []
labels = []

# identify the data we will use for our analysis
path = os.path.join('data_a', '*', '*.txt')

# iterate over each file
for i in glob.glob(path):
  
  # get the text content for this file
  text = open(i).read()
  
  # add the counts to our data
  counts.append([ 
    text.count('bones'), 
    text.count('fossil'),
  ])
  
  # add the label for this text to our data
  labels.append('archeological' if 'archeological' in i else 'medical')

# create a KNN classifier using 3 as the value of K
classifier = KNeighborsClassifier(3)

# "fit" the classifier by showing it our labelled data
classifier.fit(counts, labels)

"""
Explanation: <details>
  <summary>Solution</summary>

  We can create this plot by modifying the code above slightly:

  ```
  x_values = []
  y_values = []
  colors = []

  # specify the files to process - nb: * matches all files and directories
  path = os.path.join('data_b', '*', '*.txt')

  # iterate over each file matched above
  for i in glob.glob(path):

    # read the current file
    text = open(i).read()

    # add this text's count of the first keyword to the x axis values
    x_values.append( text.count('parallax') )

    # add this text's count of the second keyword to the x axis values
    y_values.append( text.count('angle') )

    # make the dot blue if it's a medical dot, else red if it's a mathematical dot
    colors.append('blue' if 'astronomical' in i else 'red')

  # plot the 1D distribution
  plt.title('Samples from the Philosophical Transactions')
  plt.xlabel('Count of the word "parallax"')
  plt.ylabel('Count of the word "angle"')
  plt.scatter(x_values, y_values, c=colors)
  ```
</details>

Classification using K-Nearest Neighbors
In our initial classification experiment, we classified texts as "archeological" if they included the word "bones" more than five times. That algorithm worked pretty well, but it required us to choose a keyword and then specify the threshold (>5) counts of our selected word, which is pretty clumsy. 
As it turns out, one can use algorithmic classification techniques that don't require one to specify any thresholds. There are many such classification algorithms, but we will focus on just one of them: K-Nearest Neighbors.
With a K-Nearest Neighbors Classifier, we start with a labelled dataset (e.g. a collection of texts with genre labels). We then add new, unlabelled observations to the dataset. For each of these new observations, we consult the K labelled observations to which the unlabelled observation is closest, where K is an odd integer we use for all classifications. We then find the most common label among those K observations (the "K nearest neighbors") and give the new observation that label.
The following diagram shows this scenario. Our new observation (represented by the question mark) has some points near it that are labelled with a triangle or star. Suppose we have chosen to use 3 for our value of K. In that case, we consult the 3 nearest labelled points near the question mark. Those 3 nearest neighbors have labels: star, triangle, triangle. Using a majority vote, we give the question mark a triangle label.
<img src='./images/knn.gif'>
Examining the plot above, we can see that if K were set to 1, we would classify the question mark as a star, but if K is 3 or 5, we would classify the question mark as a triangle. That is to say, K is an important parameter in a K Nearest Neighbors classifier.
To show how to execute this classification in Python, let's show how we can use our labelled book data to classify an unlabelled text record:
End of explanation
"""


classifier.predict([[5, 1]])

"""
Explanation: The classifier above takes as input the counts of the words "bones" and "fossil" for a single text, then predicts the genre label for that text based on those counts. Stated more precisely, the classifier.fit() method takes a 2D list counts and a 1D list labels. The counts 2D list is a list of lists, with one sublist for each text in the corpus. Each of those sublists contains a sequence of numbers that represent the number of times a given words occurs in the given text file. Note that the order of the word counts in these sublists is very important--each sublist must include the counts of each word being counted in the same order.
Now that we have trained our classifier, we can now ask this classifier to classify text records for which we don't have genre labels. Let's suppose we have a text file that includes the word "bones" five times and the word "fossil" once. We can predict the genre of this text with the following method:
End of explanation
"""


# type your code here

"""
Explanation: Given a text with 5 counts of the word "bones" and 1 count of the word "fossil", the model classifies that text as a "medical" rather than "mathematical" text. That's all it takes to classify texts with Python!
<h1 style='color:green'>Reviewing KNN Classification</h1>

Challenge 1:
Using the approach we discussed above, see if you can train your own classifier. To train this classifier, use the counts of the words "parallax" and "angle" within files in data_b/*/*.txt. Each text in the astronomical directory should have a class label of astronomical, and each text in the geometrical directory should have a class label geometrical. 
Challenge 2:
Once you have trained your classifier, see if you can predict the genre of a text that contains 10 instances of the word "parallax" and 1 instance of the word "angle".
Hint: once again, you may find it helpful to copy the code block we used above and modify that code block!
End of explanation
"""


%matplotlib inline
import matplotlib.pyplot as plt
import helpers

# create a KNN classifier using 3 as the value of K
classifier = KNeighborsClassifier(3)

# "fit" the classifier by showing it our labelled data
classifier.fit(counts, labels)

# use a helper function to plot the trained classifier's decision boundary
helpers.plot_decision_boundary(classifier, counts, labels)
plt.title('K-Nearest Neighbors: Classifying the Philosophical Transactions')
plt.xlabel('occurrences of word bones')
plt.ylabel('occurrences of word fossil')
plt.show()

"""
Explanation: <details>
    <summary>Solution</summary>

    We can train this classifier with the following code:

```
from sklearn.neighbors import KNeighborsClassifier
import numpy as np
import glob
import os

# create the 2D dataset we used above
counts = []
labels = []

# identify the data we will use for our analysis
path = os.path.join('data_b', '*', '*.txt')

# iterate over each file
for i in glob.glob(path):

  # get the text content for this file
  text = open(i).read()

  # add the counts to our data
  counts.append([ 
    text.count('parallax'), 
    text.count('angle'),
  ])

  # add the label for this text to our data
  labels.append('astronomical' if 'astronomical' in i else 'geometrical')

# create a KNN classifier using 3 as the value of K
classifier = KNeighborsClassifier(3)

# "fit" the classifier by showing it our labelled data
classifier.fit(counts, labels)

# predict the genre label for a text that contains 10 instances of parallax and 1 instance of angle
classifier.predict([[10, 1]])
```
</details>

Decision Boundaries
The classification example above shows how we can classify just a single point in space, but suppose we want to analyze the way a classifier would classify each possible point in some space. To do so, we can transform our space into a grid of units, then classify each point in that grid. Analyzing a space in this way is known as identifying a classifier's decision boundary, because this analysis shows one the boundaries between different classification outcomes in the space. This kind of analysis is very helpful in training machine learning models, because studying a classifier's decision boundary can help one see how to improve the classifier.
Let's plot our classifier's decision boundary below:
End of explanation
"""


from sklearn.feature_extraction.text import CountVectorizer

# use a "vectorizer" from sklearn to count our word occurrences
vectorizer = CountVectorizer()

# specify a list of strings to process -- each string represents a document
corpus = [
  'This is the first document.',
  'This document is the second document.',
  'And this is the third one.',
  'Is this the first document?',
]

# process the corpus into counts, which has one row per document and one column per unique word
counts = vectorizer.fit_transform(corpus).toarray()

# obtain the words that correspond to the columns in `counts`
words = vectorizer.get_feature_names()

# print the words
print(words)

# print the word counts in each row
print(counts)

"""
Explanation: For each pixel in the plot above, we retrieve the 3 closest points with known labels. We then use a majority vote of those labels to assign the label of the pixel. This is exactly analogous to predicting a label for unlabelled point&mdash;in both cases, we take a majority vote of the 3 closest points with known labels. Working in this way, we can use labelled data to classify unlabelled data. That's all there is to K-Nearest Neighbors classification!
It's worth noting that K-Nearest Neighbors is only one of many popular classification algorithms. From a high-level point of view, each classification algorithm works in a similar way&mdash;each requires a certain number of observations with known labels, and each uses those labelled observations to classify unlabelled observations. However, different classification algorithms use different logic to assign unlabelled observations to groups, which means different classification algorithms have very different decision boundaries. In the chart below [source], each row plots the decision boundaries several classifiers give the same dataset. Notice how some classifiers work better with certain data shapes:
<img src='images/scikit_decision_boundaries.png'>
For an intuitive introduction to many of these classifiers, including Support Vector Machines, Decision Trees, Neural Networks, and Naive Bayes classifiers, see Luis Serrano's introduction to machine learning video.
Term Document Matrices and Vector Space Models
In our simple classifier above, we used the count of a single word to predict the genre of each text in our sample corpus. As you can imagine, however, using more words allows us to make better predictions. Let's see how this works below.
To create a classifier from multiple words, we typically make what is known as a "Term Document Matrix" (TDM). A Term Document Matrix records the number of times each unique word type in a corpus occurs in each document within that corpus:
<img src='./images/term-document-matrix.png'>
The TDM above indicates that document "D1" includes the word "complexity" 2 times, the word "algorithm" 3 times, and so on. This TDM could be an Excel spreadsheet, with one row for each unique term within the corpus and one column for each document within the corpus. Since we are working in Python, however, we will build our Term Document Matrix with code:
End of explanation
"""


from sklearn.feature_extraction.text import CountVectorizer

# use a "vectorizer" from sklearn to count our word occurrences
vectorizer = CountVectorizer()

# create a list that will hold the text content of each file
corpus = []

# create a list that will hold the class label for each file
labels = []

# loop over the documents to process
for i in glob.glob( os.path.join('data_a', '*', '*.txt') ):
  
  # add the words from this document to the corpus
  corpus.append( open(i).read() )
  
  # add the class label for this file
  labels.append('archeological' if 'archeological' in i else 'medical')

# process the corpus into X, which has one row per document and one column per unique word
counts = vectorizer.fit_transform(corpus).toarray()

# identify the column labels
words = vectorizer.get_feature_names()

"""
Explanation: counts above is a list with the words contained in our Term Document Matrix. Note that the order matters! Just as we saw in our example above, counts is an example of a 2D array, or a list of lists. counts contains one sublist for each input document, and each of those sublists has a count for each word in labels. In this way, counts and labels work together to express the Term Document Matrix for our input data.
Once we have the Term Document Matrix counts, we can use that matrix to create a classifier that predicts the labels of unlabelled observations. Let's see how this looks in practice:
End of explanation
"""


# print a sample of the column labels
print(words[5000:5050])

# print a sample of the word counts in each row
print(counts[:10])

"""
Explanation: That's all it takes to create a Term Document Matrix of the word counts in each of our documents. We can investigate that Term Document Matrix by printing counts and words:
End of explanation
"""


# create a KNN classifier using 3 as the value of K
classifier = KNeighborsClassifier(3)

# "fit" the classifier by showing it our labelled data
classifier.fit(counts, labels)

"""
Explanation: As we can see, our list of words contains a large number of terms, only a sample of which is printed above. We can also see that our word counts matrix has many rows and columns (signified by the ellipses above), most of which contain 0-value cells (as most words don't occur in most documents).
We can now use the same syntax we used above to train a KNN classifier on that corpus:
End of explanation
"""


# identify the files to process
path = os.path.join('data_c', '*', '*.txt')

# iterate over each file
for i in glob.glob(path):
  
  # read the current file
  text = open(i).read()
  
  # count the words in this file
  word_counts = collections.Counter(text.split())
  
  # get the word counts for this file
  text_counts = [word_counts.get(i, 0) for i in words]
  
  # predict the genre for this file
  prediction = classifier.predict([text_counts])
  
  # print the file path and the prediction
  print(prediction, '--', i)

"""
Explanation: Just as before, we can then use our classifier to predict the genre of certain text files. In the example below, we'll ask the classifier to predict the genre of each file in data_c.
End of explanation
"""


import glob
import os

# create a list that will hold the text content of each file
corpus = []

# create a list that will hold the class labels for each file
labels = []

# identify the files to process
path = os.path.join('federalist-papers', '*', '*.txt')

# iterate over each file
for i in glob.glob(path):
  
  # determine the label for this file
  if   'hamilton' in i: label = 'Hamilton'
  elif 'madison' in i: label = 'Madison'
  elif 'jay' in i: label = 'Jay'
  else: continue # skip the disputed papers
    
  # store the text content for this file
  corpus.append( open(i).read() )
  
  # store the label for this file
  labels.append(label)

"""
Explanation: The results look pretty good! We can see our classifier correctly identifies the genre of our four new texts, despite the fact that the classifier had never seen those files before. This means our classifier has "learned" some of the features that diffentiate archeological and medical texts!
<h1 style='color:green'>Reviewing Classification: Classifying the Federalist Papers</h1>

We've now covered a lot of ground on classification. We've discussed:
1) How to transform a list of files into a Term Document Matrix
2) How to identify the words that correspond to each column in teh Term Document Matrix
3) How to train a KNN classifier on that Term Document Matrix
4) How to use that trained classifier to predict the class label for unlabelled observations
Now's our chance to put all of this together in a mini project challenge. Our goal is to files from The Federalist Papers to predict the author of the anonymously published Federalist Papers. 
The Federalist Files text files are available in: federalist-papers. If you look inside that directory, you'll see there are four subfolders: hamilton, jay, madison, and disputed. The first three of these contain the Federalist papers of known authorship, while disputed contains files of uncertain authorship. Just as we did in our examples above, we'll use the labelled observations to predict the unlabelled observations. Let's get started!
Getting Started
Our first step will be to create our word count matrix and our list of class labels for labelled text files.
End of explanation
"""


from sklearn.feature_extraction.text import CountVectorizer
from sklearn.neighbors import KNeighborsClassifier

vectorizer = CountVectorizer()
classifier = KNeighborsClassifier(3)

"""
Explanation: <h4 style='color:green'>Challenge One: Training the Classifier</h4>

See if you can use the vectorizer below to obtain a variable counts that indicates the number of times each word occurs in each document. Then see if you can use that variable counts to train the classifier defined below:
End of explanation
"""


# count the word occurrences in each document
counts = vectorizer.fit_transform(corpus).toarray()

# fit the classifier on the counts and labels
classifier.fit(counts, labels)

# obtain a list of the words in this TDM
words = vectorizer.get_feature_names()

# specify the path to the disputed papers
path = os.path.join('federalist-papers', 'disputed', '*')

# iterate over each disputed paper
for i in glob.glob(path):
  
  # read the file
  text = open(i).read()
  
  # count the words in the file
  word_counts = collections.Counter(text.split())
  
  # get the word counts for the file
  text_counts = [word_counts.get(i, 0) for i in words]
  
  # type your code here...

"""
Explanation: <details>
  <summary>Solution</summary>

  We can accomplish these two goals with the following:

```
counts = vectorizer.fit_transform(corpus).toarray()
classifier.fit(counts, labels)
```

</details>

<h4 style='color:green'>Challenge Two: Classifying Disputed Papers</h4>

We now have a trained KNN classifier. To predict the class label ("Hamilton", "Madison", or "Jay") for our so-called disputed texts, we now need to read in each of those texts, create a list of word counts using the word order in words, and then ask the classifier to predict the label for that sequence of word counts.
See if you can update the code below to predict the class label for each disputed document. Hint: you'll need to use the classifier and the text_counts variables!
End of explanation
"""


