import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
plt.style.use('fivethirtyeight')

df = pd.read_csv("data/creditRisk.csv")

df.head()

df.dtypes

"""
Explanation: Learning from Data
Decision Trees are a non-parametric supervised learning method used for classification and regression. The goal is to create a model that predicts the value of a target variable by learning simple decision rules inferred from the data features.
Paper Exercise
Let us start with a simple exercise in classifying credit risk. 
We have the following features in our dataset. 
- Risk - ordinal (label)
- Income - continuous
- Credit History - ordinal
We want to find out the rules that would help us classify the three risk type - This is a paper and pen exercise first!!
End of explanation
"""


import seaborn as sns

sns.stripplot(data = df, x = "Income", y = "Credit History", hue = "Risk", size = 10)

"""
Explanation: Plotting the Data
End of explanation
"""


df.Risk.unique()

Risk_mapping = {
       'High': 2,
       'Moderate': 1,
       'Low': 0}

df['Risk'] = df['Risk'].map(Risk_mapping)

df['Credit History'].unique()

Credit_mapping = {
        'Unknown': 0,
        'Bad': -1,
        'Good': 1}

df['Credit History'] = df['Credit History'].map(Credit_mapping)

df.head()

sns.stripplot(data = df, x = "Income", y = "Credit History", hue = "Risk", size = 10)

"""
Explanation: Preparing Data
We have one ordinal variable (Risk) and one nominal variable (Credit History)
Lets use a dictionary for encoding nominal variable
End of explanation
"""


data = df.iloc[:,0:2]
target = df.iloc[:,2:3]

from sklearn import tree

clf = tree.DecisionTreeClassifier()

clf

clf = clf.fit(data, target)

"""
Explanation: Decision Tree Classifier
End of explanation
"""


import pydotplus 
from IPython.display import Image

data.columns

target.columns

dot_data = tree.export_graphviz(clf, out_file='tree.dot', feature_names=data.columns,
                                class_names=['Low', 'Moderate', 'High'], filled=True, 
                                rounded=True, special_characters=True)

graph = pydotplus.graph_from_dot_file('tree.dot')  

Image(graph.create_png()) 

"""
Explanation: Visualise the Tree
End of explanation
"""


x_min, x_max = data.ix[:, 0].min() - 2000, data.ix[:, 0].max() + 2000
y_min, y_max = data.ix[:, 1].min() - 1, data.ix[:, 1].max() + 1
xx, yy = np.meshgrid(np.arange(x_min, x_max, (x_max - x_min)/100), np.arange(y_min, y_max, (y_max - y_min)/100))

Z = clf.predict(np.c_[xx.ravel(), yy.ravel()])
Z = Z.reshape(xx.shape)
cs = plt.contourf(xx, yy, Z, cmap=plt.cm.viridis, alpha = 0.5)
plt.scatter(x = data.ix[:,0], y = data.ix[:,1], c = target, s = 100, cmap=plt.cm.magma)

"""
Explanation: Understanding how the Decision Tree works
Terminology
- Each root node represents a single input variable (x) and a split point on that variable.
- The leaf nodes of the tree contain an output variable (y) which is used to make a prediction.
Growing the tree

The first choice we have is how many branches we split the trees. And we choose Binary Tree because otherwise it will explode due to combinatorial explosion. So BINARY TREES is a practical consideration.
The second decision is to choose which variable and where to split it. We need to have an objective function to do this

One objective function is to maximize the information gain (IG) at each split:
$$ IG(D_p,f)= I(D_p) - \frac{N_{right}}{N} I(D_{right}) - \frac{N_{left}}{N} I(D_{left}) $$
where: 
- f is the feature to perform the split
- $D_p$, $D_{left}$, and $D_{right}$ are the datasets of the parent, left and right child node, respectively
- I is the impurity measure
- N is the total number of samples
- $N_{left}$ and $N_{right}$ is the number of samples in the left and right child node. 
Now we need to first define an Impurity measure. The three popular impurity measures are:
    - Gini Impurity
    - Entropy
    - Classification Error
Gini Impurity and Entropy lead to similiar results when growing the tree, while Classification error is not as useful for growing the tree (but for pruning the tree) - See example here http://sebastianraschka.com/faq/docs/decision-tree-binary.html
Lets understand Gini Impurity a little better. Gini impurity is a measure of how often a randomly chosen element from the set would be incorrectly labeled if it was randomly labeled according to the distribution of labels in the subset,  Gini impurity can be computed by summing the probability $t_{i} $ of an item with label $i$ being chosen times the probability $ 1-t_{i}$ of a mistake in categorizing that item.
$$ I_{G}(f)=\sum {i=1}^{J}t{i}(1-t_{i})=\sum {i=1}^{J}(t{i}-{t_{i}}^{2})=\sum {i=1}^{J}t{i}-\sum {i=1}^{J}{t{i}}^{2}=1-\sum {i=1}^{J}{t{i}}^{2} $$
Lets calculate the Gini for the overall data set:
Low - 4, Moderate - 6, High - 8 and total observations are 18 
$$ I_G(t) = 1 - \left(\frac{6}{18}\right)^2 - \left(\frac{4}{18}\right)^2 - \left(\frac{8}{18}\right)^2 = 1 - \frac{116}{256} = 0.642 $$
scikit-learn uses an optimized CART algorithm, which will use a greedy approach.  A greedy approach is used to divide the space called recursive binary splitting. This is a numerical procedure where all the values are lined up and different split points are tried and tested using a objective cost function. The split with the best cost (lowest cost because we minimize cost) is selected.
Another way to think of this is that a learned binary tree is actually a partitioning of the input space. You can think of each input variable as a dimension on an p-dimensional space. The decision tree split this up into rectangles (when p=2 input variables) or some kind of hyper-rectangles with more inputs. 
We can draw these partitions for our dataset
End of explanation
"""


