import pandas as pd
from pandas import Series,DataFrame

import matplotlib.pyplot as plt
import seaborn as sns

# Gaussian Naive Bayes
from sklearn import datasets
from sklearn import metrics
from sklearn.naive_bayes import GaussianNB

"""
Explanation: Naive Bayes Classifiers
In this lecture we will learn how to use Naive Bayes Classifier to perform a Multi Class Classification on a data set we are already familiar with: the Iris Data Set. 
 This Lecture will consist of 7 main parts:

Part 1: Note on Notation and Math Terms
Part 2: Bayes' Theorem
Part 3: Introduction to Naive Bayes
Part 4: Naive Bayes Classifier Mathematics Overview
Part 5: Constructing a classifier from the probability model
Part 6: Gaussian Naive Bayes
Part 7: Gaussian Naive Bayes with SciKit Learn

Part 1: Note on Notation and Math Terms
There are a few more advanced notations and mathematical terms used during the explanation of naive Bayes Classification.
You should be familiar with the following:
Product of Sequence
The product of a sequence of terms can be written with the product symbol, which derives from the capital letter Π (Pi) in the Greek alphabet. The meaning of this notation is given by:
 $$\prod_{i=1}^4 i = 1\cdot 2\cdot 3\cdot 4,  $$
that is
 $$\prod_{i=1}^4 i = 24. $$
Arg Max
In mathematics, the argument of the maximum (abbreviated arg max or argmax) is the set of points of the given argument for which the given function attains its maximum value. In contrast to global maximums, which refer to a function's largest outputs, the arg max refers to the inputs which create those maximum outputs.
The arg max is defined by
$$\operatorname*{arg\,max}_x  f(x) := {x \mid \forall y : f(y) \le f(x)}$$
In other words, it is the set of points x for which f(x) attains its largest value. This set may be empty, have one element, or have multiple elements. For example, if f(x) is 1−|x|, then it attains its maximum value of 1 at x = 0 and only there, so
$$\operatorname*{arg\,max}_x (1-|x|) = {0}$$
Part 2: Bayes' Theorem
First, for a quick introduction to Bayes' Theorem, check out the Bayes' Theorem Lecture in the statistics appendix portion of this course, in order ot fully understand Naive Bayes, you'll need a complete understanding of the Bayes' Theorem.
Part 3: Introduction to Naive Bayes
Naive Bayes is probably one of the practical machine learning algorithms. Despite its name, it is actually performs very well considering its classification performance. It proves to be quite robust to irrelevant features, which it  ignores. It learns and predicts very fast and it does not require lots of storage. So, why is it then called naive?
The naive was added to the account for one assumption that is required for Bayes to work optimally: all features must be independent of each other. In reality, this is usually not the case, however, it still returns very good accuracy in practice even when the independent assumption does not hold.
Naive Bayes classifiers have worked quite well in many real-world situations, famously document classification and spam filtering. We will be working with the Iris Flower data set in this lecture.
Part 4: Naive Bayes Classifier Mathematics Overview
Naive Bayes methods are a set of supervised learning algorithms based on applying Bayes’ theorem with the “naive” assumption of independence between every pair of features. Given a class variable y and a dependent feature vector x<sub>1</sub> through x<sub>n</sub>, Bayes’ theorem states the following relationship:
$$P(y \mid x_1, \dots, x_n) = \frac{P(y) P(x_1, \dots x_n \mid y)}
                                 {P(x_1, \dots, x_n)}$$
Using the naive independence assumption that
$$P(x_i | y, x_1, \dots, x_{i-1}, x_{i+1}, \dots, x_n) = P(x_i | y)$$
for all i, this relationship is simplified to:
$$P(y \mid x_1, \dots, x_n) = \frac{P(y) \prod_{i=1}^{n} P(x_i \mid y)}
                                 {P(x_1, \dots, x_n)}$$
We now have a relationship between the target and the features using Bayes Theorem along with a Naive Assumption that all features are independent.
Part 5: Constructing a classifier from the probability model
So far we have derived the independent feature model, the Naive Bayes probability model. The Naive Bayes classifier combines this model with a decision rule, this decision rule will decide which hypothesis is most probable, in our example case this will be which class of flower is most probable.
Picking the hypothesis that is most probable  is known as the maximum a posteriori or MAP decision rule. The corresponding classifier, a Bayes classifier, is the function that assigns a class label to y as follows:
Since P(x<sub>1</sub>, ..., x<sub>n</sub>) is constant given the input, we can use the following classification rule:
$$P(y \mid x_1, \dots, x_n) \propto P(y) \prod_{i=1}^{n} P(x_i \mid y)$$
$$\Downarrow$$
$$\hat{y} = \arg\max_y P(y) \prod_{i=1}^{n} P(x_i \mid y),$$
and we can use Maximum A Posteriori (MAP) estimation to estimate P(y) and P(x<sub>i</sub> | y); the former is then the relative frequency of class y in the training set.
There are different naive Bayes classifiers that differ mainly by the assumptions they make regarding the distribution of P(x<sub>i</sub> | y).
Part 6: Gaussian Naive Bayes
When dealing with continuous data, a typical assumption is that the continuous values associated with each class are distributed according to a Gaussian distribution. Go back to the normal distribution lecture to review the formulas for the Gaussian/Normal Distribution.
For example of using the Gaussian Distribution, suppose the training data contain a continuous attribute, x. We first segment the data by the class, and then compute the mean and variance of x in each class. Let  μ<sub>c</sub> be the mean of the values in x associated with class c, and let  σ<sup>2</sup><sub>c</sub> be the variance of the values in x associated with class c. Then, the probability distribution of some value given a class, p(x=v|c), can be computed by plugging v into the equation for a Normal distribution parameterized by μ<sub>c</sub> and  σ<sup>2</sup><sub>c</sub>. That is:
$$p(x=v|c)=\frac{1}{\sqrt{2\pi\sigma^2_c}}\,e^{ -\frac{(v-\mu_c)^2}{2\sigma^2_c} }$$
The key to Naive Bayes is making the (rather large) assumption that the presences (or absences) of
each data feature are independent of one another, conditional on a data having a certain label.
Part 7: Gaussian Naive Bayes with SciKit Learn
Quick note we will actually only use the SciKit Learn Library in this lecture:
End of explanation
"""


# load the iris datasets
iris = datasets.load_iris()

# Grab features (X) and the Target (Y)
X = iris.data

Y = iris.target

# Show the Built-in Data Description
print iris.DESCR

"""
Explanation: Now that we have our module imports, let's go ahead and import the Iris Data Set. We have previously worked with this dataset, so go ahead and look at Lectures on MultiClass Classification for a complete breakdown on this data set!
End of explanation
"""


# Fit a Naive Bayes model to the data
model = GaussianNB()

"""
Explanation: Since we have already done a general analysis of this data in earlier lectures, let's go ahead and move on to using the Naive Bayes Method to seperate this data set into multiple classes.
First we create and fit the model
End of explanation
"""


from sklearn.cross_validation import train_test_split
# Split the data into Trainging and Testing sets
X_train, X_test, Y_train, Y_test = train_test_split(X, Y)

"""
Explanation: Now that we have our model, we will continue by seperating into training and testing sets:
End of explanation
"""


# Fit the training model
model.fit(X_train,Y_train)

"""
Explanation: Now we fit our model using the training data set:
End of explanation
"""


# Predicted outcomes
predicted = model.predict(X_test)

# Actual Expected Outvomes
expected = Y_test

"""
Explanation: Now we predict the outcomes from the Testing Set:
End of explanation
"""


print metrics.accuracy_score(expected, predicted)

"""
Explanation: Finally we can see the metrics for performance:
End of explanation
"""


