"""
### Dataset Information:

**Name: Bank_Personal_Loan_Modelling.xlsx**

**Domain: Banking**

**Aim: To classify in order to predict the likelihood of a liability bank customer for buying personal loan.**
"""
"""
## Learning Outcomes:

* Exploratory Data Analysis
* Preparing the data to train a model
* Training and making predictions using a classification model
* Model evaluation
"""
"""
## About the dataset, column attributes (independent variables) and target variable (dependent variable)
The dataset is related to the Banking sector where a bank (namely Thera bank) having information of about 5000 liability customers wants to benefit from the information to explore ways of converting its liability customers to personal loan customers (while retaining them as depositors). The data includes customer demographic information (age, income, etc.), the customer's relationship with the bank (mortgage, securities account, etc.), and the customer response to the last personal loan campaign (Personal Loan) which saw a healthy conversion rate of 9.6% i.e., 480 out of 5000 liablility customers opted for personal loans. 

Now, the retail department of the bank wants to build a machine learning model in order devise campaigns with better target marketing to increase the success ratio with minimal budget. The model should help them identify the potential customers who have a higher probability of purchasing the loan which will eventually increase their success ratio at the reduced campaign cost. 

**There are total 14 attributes in the dataset and in the context of the given problem, the target (or dependent) attribute is "Personal Loan" whereas the remaining are independent attributes.**

**Attribute Information:**
* ID : Customer ID
* Age : Customer's age in completed years
* Experience : #years of professional experience
* Income : Annual income of the customer (\$000)
* ZIP Code : Home Address ZIP code.
* Family : Family size of the customer
* CCAvg : Avg. spending on credit cards per month (\$000)
* Education : Education Level. 1: Undergrad; 2: Graduate; 3: Advanced/Professional
* Mortgage : Value of house mortgage if any. (\$000)
* Securities Account : Does the customer have a securities account with the bank?
* CD Account : Does the customer have a certificate of deposit (CD) account with the bank?
* Online : Does the customer use internet banking facilities?
* Credit card : Does the customer use a credit card issued by UniversalBank?
* Personal Loan : Did this customer accept the personal loan offered in the last campaign? (**Target Attribute**)
"""
"""
## 1) Analyzing the data distribution of each attribute and listing the relevant findings
**In this section, we will use Exploratory Data Analysis (EDA) tools and Applied Statistics concepts to verify the significance of each independent attribute towards predicting probability of a cutomer for buying the personal loan.**
"""
# Importing the necessary libraries
import numpy as np #importing numpy library
import pandas as pd  # To read the dataset as dataframe
import seaborn as sns # For Data Visualization 
import matplotlib.pyplot as plt # Necessary module for plotting purpose
%matplotlib inline
from sklearn.model_selection import train_test_split # For train-test split
# getting methods for confusion matrix, F1 score, Accuracy Score
from sklearn.metrics import confusion_matrix,f1_score,accuracy_score, classification_report
from sklearn.linear_model import LogisticRegression # For logistic Regression
from sklearn.naive_bayes import GaussianNB # For Naive Bayes classifier
from sklearn.neighbors import KNeighborsClassifier # For K-NN Classifier
from sklearn.svm import SVC # For support vector machine based classifier
# Reading the data as a data frame
df_orig = pd.read_excel('../input/bank-loan-modelling/Bank_Personal_Loan_Modelling.xlsx',sheet_name = 'Data')
df_orig.head()
# Creating copy of original dataframe 
df = df_orig.copy()
# For our convinience, let us make target attribute "Personal Loan" as the last column 
# of our dataframe.
df.drop('Personal Loan',axis=1,inplace=True)
df['Personal Loan'] = df_orig['Personal Loan']
df.head()
# Shape of dataframe
df.shape
"""
**There are 5000 rows and 14 columns in this dataset.**
"""
# More info about columns
df.info()
"""
**By closely observing the data and description given about each column attribute we can say that:**
* **Numeric data columns (Interval or Ratio) are: Age, Experience, Income, Mortgage and CCAvg**
* **Ordinal Categorical columns are: Family and Education**
* **Nominal Categorical columns are: ID, ZIP Code, Securities Account, CD Account, Online, CreditCard, Personal Loan**
"""
# The column (attribute) names are:
for name in df.columns:
    print(name)
# Let us see datatypes of the column attributes
df.dtypes
"""
**Data type of column 'CCAvg' is float64, whereas remaining attributes are of type int64**
"""
# Number of unique datatypes and their value count
df.dtypes.value_counts()
# Let us check the dataset for missing values
df.isnull().sum()
"""
**One can see that, there are no missing values in any of the column attributes**
"""
# Let us see the 5-point summary of for the attributes
df.describe()
"""
**Observations:**
* **'Experience' column shows presence of negative entries (min=-3) which is wrong and needs to be handled appropriately.**
* Remaining numeric columns such as Age, Income, CCAvg, Mortgage seem to be fine and no data cleaning is required
* The customer ages are in the range of 23 to 67 with average age as 45. Quarter of the customer population is below 35.
* Columns with binary information such as Securities Account, CD Account, Online, CreditCard, Personal Loan are also clean.
"""
# Let us see how many negative entries are there in the Experience column
df[df['Experience']<0].Experience.count()
"""
**There are 52 negative entries in the Experience column**
"""
# Let us see how many unique negative entries are there?
df[df['Experience']<0].Experience.value_counts()
"""
There there unique negative entries -1,-2 and -3 in the experience column.

**Let us clean the Experience column by removing the negative entries with appropriate values**

In order to get the appropriate values, let take a cue from correlation of Experience attribute with other attributes
"""
df.corr()
# Above table represented more elegently using heatmap
plt.figure(figsize=(12,10))
sns.heatmap(df.corr(),annot=True)
"""
**One can see that Experience attribute is highly correlated with the Age attribute so we will use this for our data cleaning purpose**

**Note: Since Age and Experience are highly correlated (0.99), so one can drop the experience column. I did the cleaning of experience column for understanding purpose.**
"""
# Let us find the unique ages which have -1, -2 and -3 entries in the Experience column
df[df['Experience'] == -1]['Age'].value_counts()
# We will find the mean of positive experience values for above ages and use it to replace all the experience entries 
# having -1 value
l1 = df[df['Experience'] == -1]['Age'].value_counts().index.tolist()
ind_1 = df[df['Experience'] == -1]['Experience'].index.tolist()
for i in ind_1:
    df.loc[i,'Experience'] = df[(df['Age'].isin(l1)) & (df.Experience > 0)].Experience.mean()
# Let us check the values are correctly replaced.
df[df['Experience'] == -1]['Age'].value_counts()
df[df['Experience'] == -2]['Age'].value_counts()
# We will find the mean of positive experience values for above ages and use it to replace all the experience entries 
# having -2 value
l2 = df[df['Experience'] == -2]['Age'].value_counts().index.tolist()
ind_2 = df[df['Experience'] == -2]['Experience'].index.tolist()
for i in ind_2:
    df.loc[i,'Experience'] = df[(df['Age'].isin(l2)) & (df.Experience > 0)].Experience.mean()
df[df['Experience'] == -3]['Age'].value_counts()
# We will find the mean of positive experience values for above ages and use it to replace all the experience entries 
# having -3 value
l3 = df[df['Experience'] == -3]['Age'].value_counts().index.tolist()
ind_3 = df[df['Experience'] == -3]['Experience'].index.tolist()
for i in ind_3:
    df.loc[i,'Experience'] = df[(df['Age'].isin(l3)) & (df.Experience > 0)].Experience.mean()
df.Experience.describe()
"""
#### Now we have a clean data to work with...!
"""
# Let us see the distribution 5000 entries in target column
df['Personal Loan'].value_counts()
"""
**480 customers out of 5000 had opted for personal loan in the previous campaign**
"""
# The column attribute "ID" doesn't provide any significant information about a customer
# buying a personal loan hence we will skip analysis of the same.
"""
**Analyzing the Age attribute**
"""
plt.figure(figsize=(12,8))
sns.distplot(df[df['Personal Loan'] == 0]['Age'],kde=False, color='b', label='Personal Loan=0')
sns.distplot(df[df['Personal Loan'] == 1]['Age'],kde=False, color='r',label='Personal Loan=1')
plt.legend()
plt.title("Age Distribution")
age_cut = pd.cut(df['Age'],bins=[20,30,40,50,60])
pd.crosstab(age_cut,df['Personal Loan']).apply(lambda r: r/r.sum()*100, axis=1)
"""
**From above table as well as distribution plot of Age attribute, one can observe that most of the customers lie in the age group of 30 to 60. Also,one can observe that 10.5% of the total customers in age group 20-30 have bought personal loan from the bank, while in age groups (30-40), (40-50) and (50-60), there is a conversion rate of around 9%.**
"""
"""
**Analyzing the Experience attribute**
"""
plt.figure(figsize=(12,8))
sns.distplot(df[df['Personal Loan'] == 0]['Experience'],kde=False, color='b', label='Personal Loan=0')
sns.distplot(df[df['Personal Loan'] == 1]['Experience'],kde=False, color='r',label='Personal Loan=1')
plt.legend()
plt.title("Experience Distribution")
exp_cut = pd.cut(df['Experience'],bins=[0,10,20,30,40,50])
pd.crosstab(exp_cut,df['Personal Loan']).apply(lambda r: r/r.sum()*100, axis=1)
"""
**One can observe that out of the total customers with experience in the range 40-50 show a good conversion rate of almost 13% for buying the personal loan, a healthy conversion rate of about 10.30% in the experience range 0 to 10, while in the ranges (10-20), (20-30) and (30-40) years of experience it is around 9%.**
"""
"""
**Analyzing the Income attribute**
"""
plt.figure(figsize=(12,8))
sns.distplot(df[df['Personal Loan'] == 0]['Income'],kde=False, color='b', label='Personal Loan=0')
sns.distplot(df[df['Personal Loan'] == 1]['Income'],kde=False, color='r',label='Personal Loan=1')
plt.legend()
plt.title("Income Distribution")
inc_cut = pd.cut(df['Income'],bins=[0,50,100,150,200,250])
pd.crosstab(inc_cut,df['Personal Loan']).apply(lambda r: r/r.sum()*100, axis=1)
"""
**No customer with income < 50,000\$ opted for the personal loan where as half of the cutomers with income within the range of 150 to 200 thousand dollars purchased personal loan...! Customers within range of (100 to 150) and (200 to 250) thousand dollars showed a conversion rate of about 28.5% and 18.75%, respectively.**
"""
"""
**Analyzing the Family attribute**
"""
# Since it is a ordinal categorical variable, we will use countplot
sns.countplot(x='Family',hue='Personal Loan',data=df)
pd.crosstab(df['Family'],df['Personal Loan']).apply(lambda r: r/r.sum()*100, axis=1)
"""
**About 13% customers with family size of 3 and 11% customers with family size of 4, purchased personal loans from the bank.**
"""
# Similar to ID attribute, we drop ZIP Code column analysis since it is not relevant to customer 
# buying the personal loan 
"""
**Analyzing the CCAvg attribute**
"""
plt.figure(figsize=(12,8))
sns.distplot(df[df['Personal Loan'] == 0]['CCAvg'],kde=False, color='b', label='Personal Loan=0')
sns.distplot(df[df['Personal Loan'] == 1]['CCAvg'],kde=False, color='r',label='Personal Loan=1')
plt.legend()
plt.title("CCAvg Distribution")
ccavg_cut = pd.cut(df['CCAvg'],bins=[0,2,4,6,8,10])
pd.crosstab(ccavg_cut,df['Personal Loan']).apply(lambda r: r/r.sum()*100, axis=1)
"""
**One can see that, CCAvg shows good correlation with target attribute. Customers with more average spending on credit cards per month also show more tendancy to buy the personal loans. Customers with avg. credit card spending in the range of 4 to 6 thousand dollars show around 47% conversion rate.**
"""
"""
**Analyzing the Education attribute**
"""
# Since Education is an ordinal categorical variable, we will use countplot
sns.countplot(df['Education'],hue=df['Personal Loan'])
pd.crosstab(df['Education'],df['Personal Loan']).apply(lambda r: r/r.sum()*100, axis=1)
"""
**Customers with "graduate" and "Advanced/Professional" education levels show a good conversion rate of about 13% and 13.65%, respectively.**
"""
"""
**Analyzing the Mortgage attribute**
"""
plt.figure(figsize=(8,8))
sns.distplot(df[df['Personal Loan'] == 0]['Mortgage'],kde=False, color='b', label='Personal Loan=0')
sns.distplot(df[df['Personal Loan'] == 1]['Mortgage'],kde=False, color='r',label='Personal Loan=1')
plt.legend()
plt.title("Mortgage Distribution")
mort_cut = pd.cut(df['Mortgage'],bins=[0,100,200,300,400,500,600])
pd.crosstab(mort_cut,df['Personal Loan']).apply(lambda r: r/r.sum()*100, axis=1)
"""
**Customers having house Mortgage value in the ranges (300 to 400), (400 to 500) and (500 to 600) show good tendency to buy the personal loans.** 
"""
"""
**Analyzing the binary variables Securities Account, CD Account, Online and CreditCard**

Since they are nominal variables we will use count plot and box plots for analysis
"""
plt.figure(figsize=(12,12))
plt.subplot(2,2,1)
sns.countplot(df['Securities Account'],hue=df['Personal Loan'])
plt.subplot(2,2,2)
sns.countplot(df['CD Account'],hue=df['Personal Loan'])
plt.subplot(2,2,3)
sns.countplot(df['Online'],hue=df['Personal Loan'])
plt.subplot(2,2,4)
sns.countplot(df['CreditCard'],hue=df['Personal Loan'])
pd.crosstab(df['Securities Account'],df['Personal Loan']).apply(lambda r: r/r.sum()*100, axis=1)
"""
**Customers with Securities account have slightly higher percentage of buying the personal loan than the customers with no Securities account**
"""
pd.crosstab(df['CD Account'],df['Personal Loan']).apply(lambda r: r/r.sum()*100, axis=1)
"""
**Customers with CD account have very high percentage (46.3%) of buying the personal loan than the customers with no CD account (7.2%)**
"""
pd.crosstab(df['Online'],df['Personal Loan']).apply(lambda r: r/r.sum()*100, axis=1)
"""
**Looks like customer using internet banking facilities has no effect on buying the personal loans..!**
"""
pd.crosstab(df['CreditCard'],df['Personal Loan']).apply(lambda r: r/r.sum()*100, axis=1)
"""
**Similar to Online attribute, customer using using a credit card issued by UniversalBank has no effect on buying the personal loans.**
"""
sns.FacetGrid(data=df,row='Education',col='Family',hue='Personal Loan').map(plt.hist,'Income').add_legend()
"""
**Irrespective of their income, undergraduate customers with 1 or 2 family members generally do not opt for personal loans.**
"""
sns.boxplot(x='Education',y='Income',hue='Personal Loan',data=df)
"""
**In each education category, one can see that customers with higher the income tend to buy personal Loans.**
"""
sns.boxplot(x='Family',y='Income',hue='Personal Loan',data=df)
"""
**For customers with 1,2,3 or 4 family members, higher income is an important factor to  buy personal Loans.**
"""
# Before moving further, let us plot the pairplot using all attributes
sns.pairplot(df_orig,hue='Personal Loan',diag_kind='hist')
"""
**Observations**
* Age has a positive linear relationship with Experience
* Income, CCAvg, Mortgage histograms are not normally distributed
"""
"""
### Analyzing the statistical significance of relevant independent attributes for target attribute prediction 
"""
# Let us see the correlation of all independent attributes with target attribute i.e., personal loan 
df.corr().loc['Personal Loan',:].sort_values(ascending=False)
"""
**From qualitative and quantitative exploratory data analysis done until now, one can see that Income and CCAvg are two most important features deciding whether a customer will buy a personal loan or not. Other attributes such as CD Account, Mortgage, Education, Family, Securities Account are also of some importance as per their mentioned order. But, let us verify the importance of above attributes using statistical approach.**
"""
"""
**Since, Age and Experience have very low correlation with target attribute, they seem to be ineffective for predicting whether customer will buy a personal loan or not. But is this statement statistically correct? Whether Age and Experience have no influence in predicting the target attribute?**
"""
"""
Note: since, Age and Experience are highly correlated, we can drop one of them in the analysis.
"""
# Significance test for numerical columns
import statsmodels.api as sm
df_num = df.loc[:,['Personal Loan', 'Income', 'CCAvg', 'CD Account', 'Mortgage', 'Education', 'Family', 'Securities Account', 'Age']]
df_num['intercept'] = 1
log_mod = sm.Logit(df_num['Personal Loan'], df_num[['intercept', 'Income', 'CCAvg', 'Mortgage', 'Age']]).fit()
log_mod.summary()
"""
**One can see that, $p$-values for Income, CCAvg are less than $\alpha = 0.05$. Hence with 95% confidence, we can say that they are significant for predicting the target attribute class. Age attribute seems to be insignificant for the given problem**
"""
# Let see the statistical significance of ordinal categorical variables Family and Education
df_ordc = df.loc[:,['Personal Loan','Family','Education']]
df_ordc['intercept'] = 1
log_mod = sm.Logit(df_ordc['Personal Loan'], df_ordc[['intercept', 'Family', 'Education']]).fit()
log_mod.summary()
"""
**By observing p-values of Family and Education, we can say that both the attributes are significant for predicting the target attribute class**
"""
"""
**In the context of independent categorical variables with binary response, one can see by observing their correlation with target attribute that only CD Account shows better correlation whereas Securities Account is slightly correlated. Let us verify them statistically for their significance**
"""
df_bc = df.loc[:,['Personal Loan','CD Account','Securities Account']]
df_bc['intercept'] = 1
log_mod = sm.Logit(df_bc['Personal Loan'], df_bc[['intercept', 'CD Account','Securities Account']]).fit()
log_mod.summary()
"""
**Since the p-values are less than 0.05, both the attributes are useful for target class prediction**
"""
df.head()
"""
## 2) Getting the target column distribution
"""
# Since target attribute is binary in nature, let us see count for each class
df['Personal Loan'].value_counts()
"""
**One can see that out of 5000 customers 480 cutomers opted for the personal loan**
"""
# Converting above target class distribution as dataframe
df_target = df['Personal Loan'].value_counts()
df_target = pd.DataFrame({'class':df_target.index, 'count':df_target.values})
df_target
# barplot for target column distribution
sns.barplot(x='class',y = 'count',data=df_target);
# Let us add the percentage column to the dataframe.
df_target['Percentage'] = df_target['count']/df_target['count'].sum()*100
df_target
# Let us plot the Pie Plot
plt.pie(df_target['Percentage'],labels=['No Personal Loan','Personal Loan'],autopct= '%1.1f%%');
"""
**One can see that, 9.6% of the total customers bought the personal loan in the previous campaign** 
"""
"""
**Observation: Clearly, there is a high amount of class imbalance in the target attribute..!**
"""
"""
## 4) Split the data into training and test set in the ratio of 70:30 respectively
"""
# Train test split
# We will drop the Age, ID columns from training as well as test dataset
X = df.iloc[:,2:-1]
y = df['Personal Loan']

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.30, random_state = 7 )
"""
## 5) Use different classification models (Logistic, K-NN and Naïve Bayes) to predict the likelihood of a customer buying personal loans
"""
"""
**About metrics used for evaluation** 

In order to minimize the cost of campaigning for the bank, a classifation algorithm in this context, should minimize both false positives (FP) as well as false negatives (FN). Hence, along with accuracy score, we will also observe f1-score. Ideally, both accuracy as well as f1-score should be 1 (or 100 in terms of percentage).
"""
"""
**We will first evaluate the model performances on unscaled data with their default parameters**
"""
"""
### Logistic Regression
"""
# create model using constructor
LogRegModel = LogisticRegression()
# fit the model to training set
LogRegModel.fit(X_train,y_train)
# Predict the test data to get y_pred
y_pred = LogRegModel.predict(X_test)
# get accuracy of model
lr_acc_score = accuracy_score(y_test,y_pred)
# get F1-score of model
lr_f1_score = f1_score(y_test,y_pred) 
# get the confusion matrix
lr_confmat = confusion_matrix(y_test,y_pred)
# get the classification report
lr_classrep = classification_report(y_test,y_pred)

print("The accuracy of the model is {} %".format(lr_acc_score*100))
print("The f1-score of the model is {} %".format(lr_f1_score*100))
print("The confusion matrix for logistic regression is: \n",lr_confmat)
print("Detailed classification report for logistic regression is: \n",lr_classrep)
"""
**Observations: One can see that although accuracy is good, f1-score is low in this case i.e., precision and recall values are not upto the mark especially recall value is low (.28) for class 1**

**Recall value of 28% means, out of all the customers who would actually buy the loan, only 28% were correctly predicted to be positive (would buy the personal loan) thus bank is missing on 72% of potential customers resulting in recampaigning cost**

**58% of precision value means out of all positively predicted customers (who would buy the personal loan) only 58% were correctly predicted.**
"""
"""
### Naive Bayes Classifier
"""
# create model using constructor
NBModel = GaussianNB()
# fit the model to training set
NBModel.fit(X_train,y_train)
# Predict the test data to get y_pred
y_pred = NBModel.predict(X_test)
# get accuracy of model
nb_acc_score = accuracy_score(y_test,y_pred)
# get F1-score of model
nb_f1_score = f1_score(y_test,y_pred) 
# get the confusion matrix
nb_confmat = confusion_matrix(y_test,y_pred)
# get the classification report
nb_classrep = classification_report(y_test,y_pred)

print("The accuracy of the model is {} %".format(nb_acc_score*100))
print("The f1-score of the model is {} %".format(nb_f1_score*100))
print("The confusion matrix for Naive Bayes classifier is: \n",nb_confmat)
print("Detailed classification report for Naive Bayes classifier is: \n",nb_classrep)
"""
**Observations: One can see that the although accuracy is decreased slightly than logistic regression case, use of Naive Bayes Classifier for this problem shows improved f1-score. Recall value is improved in case of class 1**
"""
"""
### K-NN Classifier
"""
# create model using constructor
KNNModel = KNeighborsClassifier() # Calling default constructor
# fit the model to training set
KNNModel.fit(X_train,y_train)
# Predict the test data to get y_pred
y_pred = KNNModel.predict(X_test)
# get accuracy of model
knn_acc_score = accuracy_score(y_test,y_pred)
# get F1-score of model
knn_f1_score = f1_score(y_test,y_pred) 
# get the confusion matrix
knn_confmat = confusion_matrix(y_test,y_pred)
# get the classification report
knn_classrep = classification_report(y_test,y_pred)

print("The accuracy of the model is {} %".format(knn_acc_score*100))
print("The f1-score of the model is {} %".format(knn_f1_score*100))
print("The confusion matrix for K-NN classifier is: \n",knn_confmat)
print("Detailed classification report for K-NN classifier is: \n",knn_classrep)
"""
**One can see that the although accuracy is similar to logistic regression case, f1-score is lowest among the compared algorithms. Recall value is merely .24 in case of class 1 i.e., out of all the customers who would actually buy the loan, only 24% were correctly predicted to be positive (would buy the personal loan) thus bank is missing on 76% of potential customers**
"""
"""
## Comparison of Classification Algorithms used for this problem
"""
df_comp = pd.DataFrame({'Classification Algorithm':['Logistic Regression', 'Naive Bayes', 'K-Nearest Neighbor'],'Accuracy (%)':[lr_acc_score*100,nb_acc_score*100,knn_acc_score*100],'f1-score (%)':[lr_f1_score*100,nb_f1_score*100,knn_f1_score*100]})

print("Following table shows comparison of the classification algorithms (using unscaled data and default parameters): ")
df_comp
"""
## Reasoning for the best model and why it performs better? (Assuming unscaled data and algorithms with default parameters)
"""
"""
**If we consider the algorithm comparison with their default parameters and unscaled data, Naive bayes classifier works best which has accuracy of 88% and f1-score of 47.67%. The reason it performs better because Naive Bayes classifier assumes the independence of the X variables (i.e., independent variables) and for this particular problem this is approximately true which is clear from the following heatmap**
"""
plt.figure(figsize=(12,10))
sns.heatmap(X.corr(),annot=True)
"""
**One can see that except CCAvg and income, there is no significant correlation among other column attributes of X**
"""
"""
## Trying K-NN with different neighbors on unscaled data.
In this section, we will try to find the accuracy and f1 score for KNN model with different values for neighbors.
"""
# KNN Accuracy for neighbors = 1,3,...99
knn_acc=[]
knn_f1 = []
for i in range(1,100,2):
    print("Calculating the K-NN classifier accuracy for {} neighbors.".format(i))
    # create model using constructor
    KNNModel = KNeighborsClassifier(n_neighbors=i) # Calling default constructor
    # fit the model to training set
    KNNModel.fit(X_train,y_train)
    # Predict the test data to get y_pred
    y_pred = KNNModel.predict(X_test)
    # get accuracy of model
    knn_acc_score = accuracy_score(y_test,y_pred)
    knn_acc.append(knn_acc_score*100)
    # get F1-score of model
    knn_f1_score = f1_score(y_test,y_pred) 
    knn_f1.append(knn_f1_score*100)
df_knn = pd.DataFrame({'n_neighbors':list(range(1,100,2)), 'Accuracy':knn_acc,'f1-score':knn_f1})  
df_knn
"""
**One can observe that K-NN algorithm with different values for neighbors doesn't provide much improvement both in accuracy as well as f1-score over best performing Naive Bayes Classifier algorithm on unscaled data.**
"""
"""
# Effect of scaling on the algorithm accuracy and f1-score for the compared models
Some algorithms perform well provided the data is standardized (e.g., K-NN), thus we will investigate the effect of scaling on algorithm accuracy and f1-score
"""
# Let us scale train as well as test data using MinMaxScaler
from sklearn.preprocessing import MinMaxScaler
scaler = MinMaxScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.fit_transform(X_test)
# Repeating all three algorithms again on the scaled data
"""
## Logistic Regression
"""
# create model using constructor
LogRegModel = LogisticRegression()
# fit the model to training set
LogRegModel.fit(X_train_scaled,y_train)
# Predict the test data to get y_pred
y_pred = LogRegModel.predict(X_test_scaled)
# get accuracy of model
lr_acc_score = accuracy_score(y_test,y_pred)
# get F1-score of model
lr_f1_score = f1_score(y_test,y_pred) 
# get the confusion matrix
lr_confmat = confusion_matrix(y_test,y_pred)
# get the classification report
lr_classrep = classification_report(y_test,y_pred)

print("The accuracy of the model is {} %".format(lr_acc_score*100))
print("The f1-score of the model is {} %".format(lr_f1_score*100))
print("The confusion matrix for logistic regression is: \n",lr_confmat)
print("Detailed classification report for logistic regression is: \n",lr_classrep)
"""
**In comparison with the logistic regression with unscaled data, in this case the algorithm accuracy is less than the former but the f1-score is increased. One can note that recall value is .91 for class-1 which is a class of interest.**

**Recall value of 91% means that, out of all the customers who would actually buy the loan, 91% were correctly predicted to be positive (would buy the personal loan).**
"""
"""
## Naive Bayes Classifier
"""
# create model using constructor
NBModel = GaussianNB()
# fit the model to training set
NBModel.fit(X_train_scaled,y_train)
# Predict the test data to get y_pred
y_pred = NBModel.predict(X_test_scaled)
# get accuracy of model
nb_acc_score = accuracy_score(y_test,y_pred)
# get F1-score of model
nb_f1_score = f1_score(y_test,y_pred) 
# get the confusion matrix
nb_confmat = confusion_matrix(y_test,y_pred)
# get the classification report
nb_classrep = classification_report(y_test,y_pred)

print("The accuracy of the model is {} %".format(nb_acc_score*100))
print("The f1-score of the model is {} %".format(nb_f1_score*100))
print("The confusion matrix for Naive Bayes classifier is: \n",nb_confmat)
print("Detailed classification report for Naive Bayes classifier is: \n",nb_classrep)
"""
**Scaling actually decreases the performance of Naive Bayes classifier, although no significant change in accuracy however the f1-score is merely 8% which is a below par performance.**
"""
"""
## K-NN Classifier
"""
# create model using constructor
KNNModel = KNeighborsClassifier() # Calling default constructor
# fit the model to training set
KNNModel.fit(X_train_scaled,y_train)
# Predict the test data to get y_pred
y_pred = KNNModel.predict(X_test_scaled)
# get accuracy of model
knn_acc_score = accuracy_score(y_test,y_pred)
# get F1-score of model
knn_f1_score = f1_score(y_test,y_pred) 
# get the confusion matrix
knn_confmat = confusion_matrix(y_test,y_pred)
# get the classification report
knn_classrep = classification_report(y_test,y_pred)

print("The accuracy of the model is {} %".format(knn_acc_score*100))
print("The f1-score of the model is {} %".format(knn_f1_score*100))
print("The confusion matrix for K-NN classifier is: \n",knn_confmat)
print("Detailed classification report for K-NN classifier is: \n",knn_classrep)
"""
**Evidently K-NN with default parameters (5-neighbor) on scaled data performs very well..The accuracy is 97% and f1-score is also very good.**
"""
"""
**K-NN for different values of neighbors**
"""
# KNN Accuracy for neighbors = 1,3,...99
knn_acc=[]
knn_f1 = []
for i in range(1,100,2):
    print("Calculating the K-NN classifier accuracy for {} neighbors.".format(i))
    # create model using constructor
    KNNModel = KNeighborsClassifier(n_neighbors=i) # Calling default constructor
    # fit the model to training set
    KNNModel.fit(X_train_scaled,y_train)
    # Predict the test data to get y_pred
    y_pred = KNNModel.predict(X_test_scaled)
    # get accuracy of model
    knn_acc_score = accuracy_score(y_test,y_pred)
    knn_acc.append(knn_acc_score*100)
    # get F1-score of model
    knn_f1_score = f1_score(y_test,y_pred) 
    knn_f1.append(knn_f1_score*100)
df_knn = pd.DataFrame({'n_neighbors':list(range(1,100,2)), 'Accuracy':knn_acc,'f1-score':knn_f1})
df_knn
"""
**One can see that K-NN with 5 neighbor performs the best in terms of accuracy (97%) as well as f1-score(81.63%).**
"""
"""
# So finally the winner is K-NN classifier with 5 neighbors on the scaled data...! 
"""
"""
**Reason: One can see that performance of K-NN classifier is dramatically increased when we scale the data since it is a distance-based algorithm. It's good performance can be attributed to K-NN classifier's robustness against outliers. Also, one can observe that class distributions of the important independent features such as Income, CCAvg show approximately non-overlapping nature for which K-NN is suitable.**
"""
"""
## Concluding the notebook with SVM performance on the scaled dataset..!
"""
# create model using constructor
SVMModel = SVC() # Calling default constructor
# fit the model to training set
SVMModel.fit(X_train_scaled,y_train)
# Predict the test data to get y_pred
y_pred = SVMModel.predict(X_test_scaled)
# get accuracy of model
svm_acc_score = accuracy_score(y_test,y_pred)
# get F1-score of model
svm_f1_score = f1_score(y_test,y_pred) 
# get the confusion matrix
svm_confmat = confusion_matrix(y_test,y_pred)
# get the classification report
svm_classrep = classification_report(y_test,y_pred)

print("The accuracy of the model is {} %".format(svm_acc_score*100))
print("The f1-score of the model is {} %".format(svm_f1_score*100))
print("The confusion matrix for SVM classifier is: \n",svm_confmat)
print("Detailed classification report for SVM classifier is: \n",svm_classrep)
"""
**Playing with the C value of SVC constructor**
"""
svm_acc=[]
svm_f1 = []
for i in range(1,1000,100):
    print("Calculating the SVM classifier accuracy for C = {}.".format(i))
    # create model using constructor
    SVMModel = SVC(C=i) # Calling default constructor
    # fit the model to training set
    SVMModel.fit(X_train_scaled,y_train)
    # Predict the test data to get y_pred
    y_pred = SVMModel.predict(X_test_scaled)
    # get accuracy of model
    svm_acc_score = accuracy_score(y_test,y_pred)
    svm_acc.append(svm_acc_score*100)
    # get F1-score of model
    svm_f1_score = f1_score(y_test,y_pred) 
    svm_f1.append(svm_f1_score*100)
df_svm = pd.DataFrame({'C':list(range(1,1000,100)), 'Accuracy':svm_acc,'f1-score':svm_f1})
df_svm
"""
# For Support Vector Classifier with C = 1, we get accuracy of 97.46% and f1-score around 85% which is the best result for the given problem in comparison with the Logistic Regression, Naive Bayes Classifier and K-NN Classifiers used in this notebook.
"""