"""
## Explainabilty for Credit Card Fraud Detection
"""
"""
This notebook shows the use of a machine learning model explainability library called LIME to explain credit card fraud detection.
> This notebook goes through all of the steps leading to building a classfier for fraud prediction. Then shows an overview of how to use LIME(Locally Interpretable Model-Agnostic Explanations) to understand why a transaction was deemed fraudulent or not by the LightGBM Classifier. 

![explain](https://media.giphy.com/media/WhTC5v5qQP4yAUvGKz/giphy.gif)
"""
"""
### Loading Required Libraries
"""
import numpy as np 
import pandas as pd 
import os
import lightgbm as lgb
import sklearn
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report
"""
### Data Understanding and Feature Engineering
"""
"""
To ensure things run smoothly on kaggle, lets subset the data and load one less than 20 million rows in a dataframe
"""
df = pd.read_csv("../input/credit-card-transactions/credit_card_transactions-ibm_v2.csv",nrows=19999999)
df.head()
"""
From the data sneak peak, I change the data-types to match the data description and do fill missing values since that is necessary for the explainability component.
"""
df['Zip'] = df['Zip'].fillna(0)
df['Amount'] = df['Amount'].apply(lambda value: float(value.split("$")[1]))
df['Hour'] = df['Time'].apply(lambda value: int(value.split(":")[0]))
df['Minutes'] = df['Time'].apply(lambda value: int(value.split(":")[1]))
df.drop(['Time'], axis=1, inplace=True)
df['Merchant Name'] = df['Merchant Name'].astype("object")
df['Card'] = df['Card'].astype("object")
df['Use Chip'] = df['Use Chip'].astype("object")
df['MCC'] = df['MCC'].astype("object")
df['Zip'] = df['Zip'].astype("object")
for col in df.columns:
    col_type = df[col].dtype
    if col_type == 'object':
        df[col] = df[col].fillna("")
        
"""
Separating out the label and the features from the dataset
"""
#Label: Binary: 0/1
y = df['Is Fraud?']
#Features
X = df.drop(['Is Fraud?'],axis=1)
"""
Identifying the categorical column names which will be needed as a parameter when training a LightGBM model
"""
categorical_column_names = []
categorical_cols = []
for idx,col in enumerate(X.columns):
    col_type = X[col].dtype
    if col_type == 'object' or col_type.name == 'category':
        categorical_column_names.append(col)
        categorical_cols.append(idx)
categorical_column_names.append("Zip")
categorical_column_names.append("MCC")
categorical_column_names.append("Card")
categorical_column_names.append("Merchant Name")

"""
Even though LightGBM can convert categorical variables into appropriate representations conducive to training, LIME requires categorical data to be encoded in one hot encoded/label encoded form. Since LightGBM is a tree based method label encoding will not add to an artifical ranking and a viable and less data exppanding technique
"""
categorical_names = {}
for feature in categorical_column_names:
    le = sklearn.preprocessing.LabelEncoder()
    le.fit(X.loc[:, feature])
    X.loc[:, feature] = le.transform(X.loc[:, feature])
    categorical_names[feature] = le.classes_

"""
### Creating Training and Test Sets

Train and test sets are created with the stratification option to maintain the proportion of fraudulent to successful true transactions in both train and test.
"""
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.3, random_state = 0, stratify=y)
"""
### Training a LightGBM Classifier
"""
model = lgb.LGBMClassifier()
model.fit(X_train, y_train, feature_name='auto', categorical_feature = categorical_column_names, verbose=50)
y_pred=model.predict(X_test)
"""
### Looking at the Classification Report

The dataset is imbalanced and LightGBM performs better than random predictions and can be further tuned to enhance performance.
"""
print(classification_report(y_test, y_pred))
"""
### LightGBM Model Importance

The LightGBM model importance plot shows the count by which a feature was used for a split in all weak learners. From the feature importance figure we can see that LightGBM can probably identify users which are likely to cause fraudulent transactions along with numeric columns like Amount, Time of the day and so on
"""
from lightgbm import plot_importance
plot_importance(model)
"""
## LIME (Locally Interpretable Model-Agnostic Explanations)

This framework tries to explain individual results of different classifiers/regressors by approximating it with a locally interpretable model. LIME assumes that within the vicinity of an example an interpretable model like a linear classifier can approximate any black box model. 
"""
#Import the library
import lime.lime_tabular
"""
LIME takes in the prediction function as it creates perturbed examples in the vicinity of a given input.
"""
predict_fn = lambda x: model.predict_proba(x)
"""
The LIME explainer:

1. Creates samples similar to the seed instance 
2. Generates predictions for these samples using the predict_fn
3. Use a distance measure to weight the samples with respect to the seed instance
4. Train a local interpretable model using these samples and explain the result based on the model
"""
explainer = lime.lime_tabular.LimeTabularExplainer(X_train.values ,feature_names = list(X_train.columns), class_names=['No','Yes'],
                                                   categorical_features=categorical_cols, 
                                                   categorical_names=categorical_column_names, kernel_width=3)
"""
The explain instance shows the contribution of the features for the input example
"""
exp = explainer.explain_instance(X_test.values[1], predict_fn)
exp.show_in_notebook()
"""
The true class is similar to the predicted class
"""
y_test.values[1]