#Load the libraries
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

#Default Variables
%matplotlib inline
plt.rcParams['figure.figsize'] = (8,6)
plt.style.use('ggplot')
pd.set_option('display.float_format', lambda x: '%.2f' % x)

#Load the training dataset
df = pd.read_csv("../data/historical_loan.csv")

#View the first few rows of training dataset
df.head()

#View the columns of the train dataset
df.columns

#View the data types of the train dataset
df.dtypes

#View the number of records in the data
df.shape

#View summary of raw data 
df.describe()

"""
Explanation: Machine Learning Model
Business Problem
Startup XYZ is in the business of giving personal loans, structured as non-recourse loans. The defaults on their loans are much higher than their competitors. Also, the underlying collaterals lose their value way too quicky and has resulted in huge losses for Bank XYZ.
Alice was recently appointed as the Senior VP of the Risk Organization. She comes from a strong analytics background and wants to leverage data science to identify customer's risk before approving loan.
She's appointed you as a consultant to help her and the team solve this problem.
Note: This case study was inspired by the bank marketing case study. The data is a modified version of what is available in that site
Brainstorming
1. Frame
The first step is to convert the business problem into an analytics problem
Alice wants to know customer's risk. Let's try to predict the propensity of a customer to default, given the details he/she has entered on the loan application form
2. Acquire
After discussions with the IT team of Startup XYZ, you have obtained some historical data from the bank. It has the following columns
Application Attributes:
- years: Number of years the applicant has been employed
- ownership: Whether the applicant owns a house or not
- income:  Annual income of the applicant
- age: Age of the applicant
- amount : Amount of Loan requested by the applicant  
Behavioural Attributes:
- grade:  Credit grade of the applicant
Outcome Variable:

default : Whether the applicant has defaulted or not 

Load the data
End of explanation
"""


# Find if df has missing values.
df.isnull().head()

# In a large dataset, this is hard to find if there are any missing values or not.
# We can chain operators on the output. Let's use sum()

df.isnull().sum()

"""
Explanation: 3. Refine
Lets check the dataset for compeleteness - by checking for missing values
Missing values
End of explanation
"""


# Let's replace missing values with mean
# There's a fillna function
df.years = df.years.fillna(np.mean(df.years))

#Finding unique values of years 
pd.unique(df.years)

"""
Explanation: So, we see that years have missing values. The column is numeric. We have three options for dealing with missing values
Options to treat Missing Values

REMOVE - NAN rows
IMPUTATION - Replace them with something??
Mean
Median
Fixed Number - Domain Relevant
High Number (999) - Issue with modelling


BINNING - Categorical variable and "Missing becomes a number
DOMAIN SPECIFIC - Entry error, pipeline, etc.
End of explanation
"""


# Create histogram for target variable - default
df.default.plot.hist()

# Explore grade
df.grade.value_counts().plot.barh()

# Explore age
df.age.plot.hist(bins=50)

"""
Explanation: We also need to check for quality - by checking for outliers in the data. For this workshop, we will skip doing that. But remember to check for outliers when doing in real-life
4. Explore
The goal is to build some intuition around the data
 Single Variable Exploration - Univariate Analysis
End of explanation
"""


# Explore the impact of age with income
df.plot.scatter(x='age', y='income', alpha=0.7)

"""
Explanation: Dual Variable Exploration - Bivariate Analysis
End of explanation
"""


df.plot.scatter(x='age', y='income', c='default', alpha=0.25, cmap='viridis')

# Explore the relationship between age, income and default
df.plot.scatter(x='age', y='income', c='default', logx=True, logy=True, alpha=0.25, cmap='viridis')

"""
Explanation: EXERCISE
 Three Variables Exploration 
Explore the relationship between age, income and defualt
End of explanation
"""


# Let's again revisit the data types in the dataset
df.dtypes

"""
Explanation: 5. Transform
End of explanation
"""


from sklearn.preprocessing import LabelEncoder

# Let's not modify the original dataset. 
# Let's transform it in another dataset
df_encoded = df.copy()

# instantiate label encoder
le_grade = LabelEncoder()

# fit label encoder
le_grade = le_grade.fit(df.grade)

df_encoded.grade = le_grade.transform(df_encoded.grade)

df_encoded.head()

"""
Explanation: Two of the columns are categorical in nature - grade and ownership.
To build models, we need all of the features to be numeric. There exists a number of ways to convert categorical variables to numeric values.
We will use one of the popular options: LabelEncoding
End of explanation
"""


df.ownership.unique()

le_ownership = LabelEncoder()
le_ownership = le_ownership.fit(df["ownership"])

df_encoded.ownership = le_ownership.transform(df_encoded.ownership)

df_encoded.head()

"""
Explanation: EXERCISE
Do label encoding on ownership
End of explanation
"""


X_2 = df_encoded.loc[:,('age', 'amount')]
y = df_encoded.loc[:,'default']

"""
Explanation: 6. Model
Common approaches:

Linear models
Tree-based models
Neural Networks
... 

Some choices to consider:

Interpretability
Run-time
Model complexity
Scalability

For the purpose of this workshop, we will use tree-based models.
We will do the following two:

Decision Tree
Random Forest

Decision Trees
Decision Trees are a non-parametric supervised learning method used for classification and regression. The goal is to create a model that predicts the value of a target variable by learning simple decision rules inferred from the data features.
Let's first build a model using just two features to build some intuition around decision trees
Step 1 - Create features matrix and target vector
End of explanation
"""


from sklearn import tree

# instantiate the decision tree object
clf_dt_2 = tree.DecisionTreeClassifier(max_depth=2)

# fit the decision tree model
clf_dt_2 = clf_dt_2.fit(X_2, y)

"""
Explanation: Step 2 - Build decision tree model
End of explanation
"""


import pydotplus 
from IPython.display import Image

dot_data = tree.export_graphviz(clf_dt_2, out_file='tree.dot', feature_names=X_2.columns,
                                class_names=['no', 'yes'], filled=True, 
                                rounded=True, special_characters=True)

# Incase you don't have graphviz installed
# txt = open("tree.dot").read().replace("\\n", "\n  ").replace(";", ";\n")
# print(txt)

graph = pydotplus.graph_from_dot_file('tree.dot')  

Image(graph.create_png()) 

"""
Explanation: Step 3 - Visualize the decision tree
End of explanation
"""


def plot_boundaries(X2, clf):
    x_min, x_max = X2.iloc[:, 0].min() - 1, X2.iloc[:, 0].max() + 1
    y_min, y_max = X2.iloc[:, 1].min() - 1, X2.iloc[:, 1].max() + 1
    xx, yy = np.meshgrid(np.arange(x_min, x_max, (x_max - x_min)/100), 
                         np.arange(y_min, y_max, (y_max - y_min)/100))
    Z = clf.predict_proba(np.c_[xx.ravel(), yy.ravel()])[:,1]
    Z = Z.reshape(xx.shape)
    target = clf.predict(X2)
    plt.scatter(x = X2.iloc[:,0], y = X2.iloc[:,1], c = y, s = 20, cmap=plt.cm.magma)
    cs = plt.contourf(xx, yy, Z, cmap=plt.cm.viridis, alpha = 0.4)

plot_boundaries(X_2, clf_dt_2)

"""
Explanation: Let's see the decision boundaries
End of explanation
"""


clf_dt_10 = tree.DecisionTreeClassifier(max_depth=10).fit(X_2,y)

plot_boundaries(X_2, clf_dt_10)

"""
Explanation: EXERCISE
Change the depth of the Decision Tree classifier to 10 and plot the decision boundaries again.
End of explanation
"""


pred_class = clf_dt_10.predict(X_2)
pred_proba = clf_dt_10.predict_proba(X_2)

plt.hist(pred_class);

import seaborn as sns
sns.kdeplot(pred_proba[:,1], shade=True)

"""
Explanation: Lets understand first just the difference between Class prediction and Class Probabilities
End of explanation
"""


from sklearn.metrics import confusion_matrix
from sklearn.metrics import roc_auc_score
from sklearn.metrics import roc_curve

X = df_encoded.iloc[:,1:]
y = df_encoded.iloc[:,0]

clf_dt = tree.DecisionTreeClassifier(max_depth=5)

def pred_df(clf, X, y):
    clf = clf.fit(X,y)
    y_pred = clf.predict(X)
    y_proba = clf.predict_proba(X)[:,1]
    pred_df = pd.DataFrame({"actual": np.array(y), "predicted": y_pred, "probability": y_proba})
    return pred_df

pred_dt = pred_df(clf_dt, X,y)
pred_dt.head()

pd.crosstab(pred_dt.predicted, pred_dt.actual)

confusion_matrix(pred_dt.predicted, pred_dt.actual)

def plot_prediction(pred_df):
    pred_df_0 = pred_df[pred_df.actual == 0]
    pred_df_1 = pred_df[pred_df.actual == 1]
    sns.kdeplot(pred_df_0.probability, shade=True, label="no default")
    sns.kdeplot(pred_df_1.probability, shade=True, label="default")    

plot_prediction(pred_dt)

def plot_roc_auc(pred_df):
    fpr, tpr, thresholds = roc_curve(pred_df.actual, pred_df.probability)
    auc_score = roc_auc_score(pred_df.actual,pred_df.probability)
    plt.plot(fpr, tpr, label='AUC = %0.2f' % auc_score)
    plt.xlabel("False Positive Rate")
    plt.ylabel("True Positive Rate")
    plt.legend(loc="lower right")
    return print("AUC = %0.2f" % auc_score)

plot_roc_auc(pred_dt)

"""
Explanation: Model Validation
While we have created the model, we still don't have a measure of how good the model is. We need to measure some accuracy metric of the model and have confidence that it will generalize well. We should be confident that when we put the model in production (real-life), the accuracy we get from the model results should mirror the metrics we obtained when we built the model.
Selecting the right accuracy metric for the model is important. 
This wiki has a good overview of some of the common metrics.
We will use a metric - Area Under the Curve
Area Under the Curve
In a Receiver Operating Characteristic (ROC) curve the true positive rate (Sensitivity) is plotted in function of the false positive rate (100-Specificity) for different cut-off points. Each point on the ROC curve represents a sensitivity/specificity pair corresponding to a particular decision threshold. A test with perfect discrimination (no overlap in the two distributions) has a ROC curve that passes through the upper left corner (100% sensitivity, 100% specificity). Therefore the closer the ROC curve is to the upper left corner, the higher the overall accuracy of the test 
(source)
End of explanation
"""


clf_dt_10 = tree.DecisionTreeClassifier(max_depth=10)

pred_dt_10 = pred_df(clf_dt_10, X,y)

confusion_matrix(pred_dt_10.predicted, pred_dt_10.actual)

plot_prediction(pred_dt_10)

plot_roc_auc(pred_dt_10)

"""
Explanation: EXERCISE
Build a decison tree classifier with max_depth = 10 and plot confusion_matrix & auc
End of explanation
"""


from sklearn.model_selection import StratifiedKFold

def cross_val(clf, k):
    
    # Instantiate stratified k fold.
    kf = StratifiedKFold(n_splits=k)
    
    # Let's use an array to store the results of cross-validation
    kfold_auc_score = []

    # Run kfold CV
    for train_index, test_index in kf.split(X,y):
        clf = clf.fit(X.iloc[train_index], y.iloc[train_index])
        proba = clf.predict_proba(X.iloc[test_index])[:,1]
        auc_score = roc_auc_score(y.iloc[test_index],proba)
        print(auc_score)
        kfold_auc_score.append(auc_score)
    
    print("Mean K Fold CV:", np.mean(kfold_auc_score))

cross_val(clf_dt, 3)

"""
Explanation: Cross-validation
Now that we have chosen the error metric, how do we find the generalization error?
We do this using cross-validation. ([source]
(https://en.wikipedia.org/wiki/Cross-validation_(statistics))
From wiki: 

One round of cross-validation involves partitioning a sample of data into complementary subsets, performing the analysis on one subset (called the training set), and validating the analysis on the other subset (called the validation set or testing set). To reduce variability, multiple rounds of cross-validation are performed using different partitions, and the validation results are averaged over the rounds.


We will use StratifiedKFold.
This ensures that in each fold, the proportion of positive class and negative class remain similar to the original dataset
This is the process we will follow to get the mean cv-score

Generate k-fold
Train the model using k-1 fold
Predict for the kth fold 
Find the accuracy.
Append it to the array
Repeat 2-5 for different validation folds
Report the mean cross validation score
End of explanation
"""


clf_dt_10 = tree.DecisionTreeClassifier(max_depth=10)

cross_val(clf_dt_10, 5)

"""
Explanation: EXERCISE
Build a classifier with max_depth = 10 and run a 5-fold CV to get the auc score.
End of explanation
"""


clf_dt_20 = tree.DecisionTreeClassifier(max_depth=20)

cross_val(clf_dt_20, 5 )

"""
Explanation: Build a classifier with max_depth = 20 and run a 5-fold CV to get the auc score.
End of explanation
"""


from sklearn.ensemble import RandomForestClassifier

clf_rf = RandomForestClassifier(n_estimators=10)

cross_val(clf_rf, 5)

"""
Explanation: Bagging
Decision trees in general have low bias and high variance. We can think about it like this: given a training set, we can keep asking questions until we are able to distinguish between ALL examples in the data set. We could keep asking questions until there is only a single example in each leaf. Since this allows us to correctly classify all elements in the training set, the tree is unbiased. However, there are many possible trees that could distinguish between all elements, which means higher variance.
How do we reduce variance?
In order to reduce the variance of a single error tree, we usually place a restriction on the number of questions asked in a tree. This is true  for single decision trees which we have seen in previous notebooks.
Along with this other method to do reduce variance is to ensemble models of decision trees. The goal of ensemble methods is to combine the predictions of several base estimators built with a given learning algorithm in order to improve generalizability / robustness over a single estimator.
How to ensemble?


Averaging: Build several estimators independently and then average their predictions. On average, the combined estimator is usually better than any of the single base estimator because its variance is reduced. Examples:

Bagging
Random Forest
Extremely Randomized Trees



Boosting: Build base estimators sequentially and then try to reduce the bias of the combined estimator. The motivation is to combine several weak models to produce a powerful ensemble.

AdaBoost
Gradient Boosting (e.g. xgboost)



Random Forest
In random forests, each tree in the ensemble is built from a sample drawn with replacement (i.e., a bootstrap sample) from the training set. In addition, when splitting a node during the construction of the tree, the split that is chosen is no longer the best split among all features. Instead, the split that is picked is the best split among a random subset of the features. 
As a result of this randomness, the bias of the forest usually slightly increases (with respect to the bias of a single non-random tree) but, due to averaging, its variance also decreases, usually more than compensating for the increase in bias, hence yielding an overall better model.
Random Forest Model
The advantage of the scikit-learn API is that the syntax remains fairly consistent across all the classifiers.
If we change the DecisionTreeClassifier to RandomForestClassifier in the above code, we should be good to go :-)
End of explanation
"""


clf_rf_100 = RandomForestClassifier(n_estimators=100)

cross_val(clf_rf_100, 5)

"""
Explanation: EXERCISE
Change the number of trees from 10 to 100 and make it 5-fold. And report the cross-validation error (Hint: You should get ~ 0.74. )
End of explanation
"""


df_encoded.head()

clf_rf_100.fit(X,y)

pred_rf_100 = pred_df(clf_rf_100, X,y)

plot_prediction(pred_rf_100)

confusion_matrix(pred_dt.predicted, pred_dt.actual)

y_pred = clf_rf_100.predict(X)
y_proba = clf_rf_100.predict_proba(X)[:,1]

y_out = pd.DataFrame({"y_pred":y_pred, "y_proba": y_proba})

y_out.head()

df_out = pd.concat([df, y_out], axis =1)

df_out.head()

df_out.to_csv( "../data/default.csv",index_label=False)

final_model = RandomForestClassifier(n_estimators=100)
final_model = final_model.fit(X, y)

"""
Explanation: A more detailed version of bagging and random forest can be found in the speakers' introductory machine learning workshop material
bagging
random forest
Model Selection
We choose the model and its hyper-parameters that has the best cross-validation score on the chosen error metric.
In our case, it is random forest.
Now - how do we get the model?
We need to run the model with the chosen hyper-parameters on all of the train data. And serialize it.
End of explanation
"""


from sklearn.externals import joblib

joblib.dump(final_model, "model.pkl")
joblib.dump(le_grade, "le_grade.pkl")
joblib.dump(le_ownership, "le_ownership.pkl");

"""
Explanation: Model serialization
We need to serialize the model and the label encoders.
End of explanation
"""


