"""
# Bank Marketing Data - Building EasyEnsemble Model
"""
"""
### Introduction

__The main goal of this project is to predict if the client will subscribe a term deposit (variable y).__ I will show how to approach developing a model to predict client's subscriptions using an imbalanced dataset. 

The dataset is provided by [UCI-Machine Learning Repository](https://archive.ics.uci.edu/ml/datasets/bank+marketing). The data contains direct marketing campaigns (phone calls) of a Portuguese banking institution. Often, more than one contact to the same client was required, in order to access if the product (bank term deposit) would be 'yes' or 'no' subscribed. 


### Feature Engineering
The original dataset has 10 numerical variables (9 continuous and 1 discrete), 10 categorical variables. I created 2 categorical variables `age_group` and `pdays_group` by using `age` and `pdays`, and removed the original variables. 
Also, I removed `duration` variable due to the given information: this attribute highly affects the output target (e.g., if duration=0 then y='no'). Yet, the duration is not known before a call is performed. Also, after the end of the call y is obviously known. Thus, this input should only be included for benchmark purposes and should be discarded if the intention is to have a realistic predictive model.

So, the final dataset has 11 numerical variables and 8 categorical variables. 

Before building a model, we need to encode categorical variables as numerical values. So, I will use integer encoding by using feature engine package. Also, I will use StandardScaler from sklearn to scale all numerical variables. 

I'll show the feature engineering process in this notebook with less detail, so if you want to see more, please take a look at [Bank_Marketing_EDA.ipynb](https://github.com/yejiseoung/BankMarketing/blob/develop/Bank_Marketing_EDA.ipynb) and [Bank_Marketing_FeatureEngineering.ipynb](https://github.com/yejiseoung/BankMarketing/blob/develop/Bank_Marketing_FeatureEngineering.ipynb) in my Github. Or, you can find [Simple EDA, Feature Engineering, and Visualization](https://www.kaggle.com/yejiseoung/simple-eda-feature-engineering-and-visualization) in Kaggle. You can find the visualizations and more detail in those notebooks. 




### Imbalanced data
We have a classification problem. The target is a binary variable (no-yes), and it shows 89% of no and 11% of yes responses. This dataset is a typical imbalanced data which means that datasets have many more instances of certain classes (no - 89%) than of others (yes - 11%). Since most machine learning algorithms assume balanced distributions, samples from the minority class (in this case, yes label) are likely misclassified. So, we need to deal with this very carefully. Through the notebook, I will implement EasyEnsemble technique which is an ensemble algorithms that were designed to work with imbalanced datasets. 

You might wonder why I won't use under- or over-sampling techniques. I tried all possible techinques to balance imbalanced dataset and none of under- or over-sampling techinques did not improve the model performance. You can find the whole process in [Bank_Marketing_BalancingSampling.ipynb](https://github.com/yejiseoung/BankMarketing/blob/develop/Bank_Marketing_BalancingSampling.ipynb)


- __Reference__: [Moro et al., 2014] S. Moro, P. Cortez and P. Rita. A Data-Driven Approach to Predict the Success of Bank Telemarketing. Decision Support Systems, Elsevier, 62:22-31, June 2014


"""
# This Python 3 environment comes with many helpful analytics libraries installed
# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python
# For example, here's several helpful packages to load

import numpy as np # linear algebra
import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)

# Input data files are available in the read-only "../input/" directory
# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory

import os
for dirname, _, filenames in os.walk('/kaggle/input'):
    for filename in filenames:
        print(os.path.join(dirname, filename))

# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using "Save & Run All" 
# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session
"""
### Import
"""
# we need feature_engine library for categorical encoding
!pip install feature_engine
import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
from scipy import stats

import warnings
warnings.filterwarnings('ignore')

from collections import Counter

pd.set_option('display.max_columns', 100)
pd.set_option('display.max_rows', 100)

# for the model 
from sklearn.model_selection import train_test_split, RandomizedSearchCV
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import (
    GradientBoostingClassifier,
    RandomForestClassifier)

from sklearn.metrics import roc_auc_score

from sklearn.preprocessing import StandardScaler

from sklearn.pipeline import Pipeline

# for feature engineering
from feature_engine import encoding as ce

# for ensemble sampling
from imblearn.ensemble import EasyEnsembleClassifier
"""
### Load data
"""
data = pd.read_csv('/kaggle/input/bank-marketing-data-set/bank-additional-full.csv',
                  delimiter=';')
print(data.shape)
data.head()
# we have imbalanced data
data['y'].value_counts() / len(data['y'])
"""
## Data Clean
"""
"""
Before building a model, we need to engineer categorical variables and scale numerical variables. 


You can see more detail in [Bank_Marketing_EDA.ipynb](https://github.com/yejiseoung/BankMarketing/blob/develop/Bank_Marketing_EDA.ipynb) and [Bank_Marketing_FeatureEngineering.ipynb](https://github.com/yejiseoung/BankMarketing/blob/develop/Bank_Marketing_FeatureEngineering.ipynb) in my Github. Or, you can find [Simple EDA, Feature Engineering, and Visualization](https://www.kaggle.com/yejiseoung/simple-eda-feature-engineering-and-visualization) in Kaggle. 
"""
"""
### Feature Engineering
"""
"""
Here, I will drop `duration` and create `age_group` and `pdays_group` by using `age` and `pdays` variables, and drop the original variables. 
"""
data.drop('duration', axis=1, inplace=True)
data.shape
## create age_group
# let's divide age into the buckets 
buckets = [16, 30, 40, 50, 60, 100]

# bucket labels
labels = ['17-30', '31-40', '41-50', '51-60', '>60']

# discretization
data['age_group'] = pd.cut(data['age'], bins=buckets, labels=labels, include_lowest=True)
data['age_group'] = data['age_group'].astype('O') # change dtype

# drop the original age column
data.drop('age', axis=1, inplace=True)
## create pdays_group
# pdays has 999 values which mean that 
# client was not previous contacted. 
# I will change 999 to -1 
data['pdays'] = data['pdays'].replace(999, -1)

# let's divide pdays into 3 groups
bins = [0, 7, 14, 30]

# labels
labels = ['1w', '2w', '>2w']

data['pdays_group'] = pd.cut(data['pdays'], bins=bins, labels=labels, 
                             include_lowest=False) # False makes -1 value to missing values

# fill missing values to Not contacted
data['pdays_group'] = data['pdays_group'].astype('O')
data['pdays_group'].fillna('Not contacted', inplace=True)

# drop the original column
data.drop('pdays', axis=1, inplace=True)
# change binary value of y to numerical values
data['y'] = data['y'].map({'no':0, 'yes': 1})
"""
### Separate train and test data
"""
# Create lists for categorical and numerical variables

cat_vars = [var for var in data.columns if var != 'y' and data[var].dtype=='O']
num_vars = [var for var in data.columns if var != 'y' and data[var].dtype!='O']

print('Number of Categorical variables: {}'.format(len(cat_vars)))
print('Number of Numerical variables: {}'.format(len(num_vars)))
# separate train and test set
X_train, X_test, y_train, y_test = train_test_split(
    data.drop('y', axis=1),
    data['y'],
    test_size=0.3,
    random_state=0
)

X_train.shape, X_test.shape
"""
### Encoding Categorical variables
"""
# using OrdinalEncoder from feature-engine

ordinal_enc = ce.OrdinalEncoder(
    encoding_method='arbitrary',
    variables=cat_vars)

ordinal_enc.fit(X_train) 
X_train = ordinal_enc.transform(X_train)
X_test = ordinal_enc.transform(X_test)
"""
## Feature Scaling - StandardScaler
"""
# set up scaler
scaler = StandardScaler()

# fit the scaler to the train set, it will learn the parameters
scaler.fit(X_train)

# transform train and test set
X_train = scaler.transform(X_train)
X_test = scaler.transform(X_test)
"""
## Building a model

EasyEnsemble involves creating balanced samples of the training datset by selecting all examples from the minority class and a subset from the majority class. Rather than using pruned decision trees, boosted decision trees are used on each subset, specifically the AdaBoost algorithm.

The EasyEnsembleClassifier class from the imbalanced-learn library provides an implmentation of the easy ensemble technique. I will use EasyEnsembleClassifier in this notebook, because it shows the best performance. I compared other balancing sampling techniques such over- and under-sampling techniques, so if you want to see more detail, please see [Bank_Marketing_BalancingSampling.ipynb](https://github.com/yejiseoung/BankMarketing/blob/main/Bank_Marketing_BalancingSampling.ipynb)

We can choose the classifiers to build the best model, so I found the best hyperparameters and estimators for this problem. I will skip this process because it takes a long time to find best hyperparatmers. You can see the entire process [Bank_Marketing_Model.ipynb](https://github.com/yejiseoung/BankMarketing/blob/main/Bank_Marketing_Model.ipynb)
"""
def run_RFs(X_train, X_test, y_train, y_test):
    """ Function to evaluate random forests performance"""
    rf = RandomForestClassifier(n_estimators=20)
    rf.fit(X_train, y_train)
    
    print('Train set')
    pred = rf.predict_proba(X_train)
    print('RF roc-auc: {}'.format(roc_auc_score(y_train, pred[:,1])))
    
    print('Test set')
    pred = rf.predict_proba(X_test)
    print('RF roc-auc: {}'.format(roc_auc_score(y_test, pred[:, 1])))
def run_easy(X_train, X_test, y_train, y_test):
    """Function to evaluate EasyEnsemble performance"""
    easy = EasyEnsembleClassifier(
        n_estimators=20,
        sampling_strategy='auto', 
        n_jobs=1,
        random_state=42)
    
    easy.fit(X_train, y_train)
    
    print('Train set')
    pred = easy.predict_proba(X_train)
    print('EasyEnsemble roc-auc: {}'.format(roc_auc_score(y_train, pred[:,1])))
    
    print('Test set')
    pred = easy.predict_proba(X_test)
    print('EasyEnsemble roc-auc: {}'.format(roc_auc_score(y_test, pred[:,1])))
# base model for RF
run_RFs(X_train, X_test, y_train, y_test)
"""
Random forest shows over-fitting, and we can solve this problem by tuning the hyperparamters, but I won't do it here because I wanted to create base model. 
"""
# base model
run_easy(X_train, X_test, y_train, y_test)
# Default estimator for easyensemble is AdaBoost
# however, I found gbm shows better performance for this dataset
# so, I will use gbm as base_estimator 

# set up gbm
gbm = GradientBoostingClassifier()

# set up classifier
easy = EasyEnsembleClassifier(n_estimators=98,
                             sampling_strategy='auto',
                             base_estimator=gbm,
                             n_jobs=1,
                             random_state=42)
easy.fit(X_train, y_train)
X_train_preds = easy.predict_proba(X_train)[:, 1]
X_test_preds = easy.predict_proba(X_test)[:, 1]

print('Train roc-auc: ', roc_auc_score(y_train, X_train_preds))
print('Test roc-auc: ', roc_auc_score(y_test, X_test_preds))
"""
By tuning hyperparameters of EasyEnsembleClassifier, we got slightly improved performance of the model.
"""