"""
<a id="table-of-contents"></a>
"""
"""
<div style="background:#2b6684;color:white; font-family:'Goudy Old Style';padding:0.5em;border-radius:0.2em;font-size:30px;color:white"><u>Content in this Notebook</u></div>
<div style="background:#2b6684;color:white; font-family:'Goudy Old Style';padding:0.5em;border-radius:0.2em;font-size:30px;color:white"><u>Predict churn: Logistic Regression</u>    
<p style="font-family:cursive;font-size:17px; color:white" >Notebook is a part of FREE ML course by Glexey Grigorev.
<a style="font-family:cursive;font-size:17px; color: yellow" href="https://datatalks.club/courses/2021-winter-ml-zoomcamp.html" target="_blank"> <u>Link for the Course</u></a></p>
<li style = "line-height: 0.7"><a style="font-family:cursive;font-size:17px; color:#ecfe15" href = "#1"> 1. Introduction </a></li>      
<li style = "line-height: 0.7"><a style="font-family:cursive;font-size:17px; color:#ecfe15" href = "#2" target="_blank"> 2. Impoting Libraries </a></li> 
<li style = "line-height: 0.7"><a style="font-family:cursive;font-size:17px; color:#ecfe15;" href = "#3"> 3. Loading and Reading Data </a></li> 
<li  style = "line-height: 0.7"><a style="font-family:cursive;font-size:17px; color:#ecfe15" href = "#4"> 4. Cleaning Strings in Column and Values</a></li> 
<li style = "line-height: 0.7"><a style="font-family:cursive;font-size:17px; color:#ecfe15" href = "#5"> 5. Data Preparation </a></li>  
<li style = "line-height: 0.7"><a style="font-family:cursive;font-size:17px; color:#ecfe15" href = "#6"> 6. Setting up Validation Framework </a></li>  
<li style = "line-height: 0.7"><a style="font-family:cursive;font-size:17px; color:#ecfe15" href = "#7"> 7. Exploratory Data Analysis  </a></li>
<li style = "line-height: 0.7"><a style="font-family:cursive;font-size:17px; color:#ecfe15" href = "#8.1"> 8. Feature Importance </a></li>
    <ul> 
    <li style = "line-height: 0.7"><a style="font-family:cursive;font-size:17px; color:#ecfe15" href = "#8.1"> 8.1 Feature Importance: Churn Rate and Risk Ratio</a></li> 
    <li style = "line-height: 0.7"><a style="font-family:cursive;font-size:17px; color:#ecfe15" href = "#8.2"> 8.2 Feature Importance: Mutual Information </a></li> 
    <li style = "line-height: 0.7"><a style="font-family:cursive;font-size:17px; color:#ecfe15" href = "#8.3">  8.3 Feature Importance: Correlation</a></li>
 </ul>
<li style = "line-height: 0.7"><a style="font-family:cursive;font-size:17px; color:#ecfe15" href = "#9"> 9. One Hot Encoding using DictVectorizer </a></li>
<li style = "line-height: 0.7"><a style="font-family:cursive;font-size:17px; color:#ecfe15" href = "#10"> 10. Logistic Regression</a></li>     
<li style = "line-height: 0.7"><a style="font-family:cursive;font-size:17px; color:#ecfe15" href = "#11"> 11. Training logistic regression with Scikit-Learn</a></li> 
<li style = "line-height: 0.7"><a style="font-family:cursive;font-size:17px; color:#ecfe15" href = "#12"> 12. Model interpretation</a></li>  
<li style = "line-height: 0.7"><a style="font-family:cursive;font-size:17px; color:#ecfe15" href = "#13"> 13. Using the model </a></li>  
<li style = "line-height: 0.7"><a style="font-family:cursive;font-size:17px; color:#ecfe15" href = "#14"> 14. Summary</a></li>    
           
  
</div>

"""
"""
[back to top](#table-of-contents)
<a id="1"></a>

<div style="background:#2b6684;color:white; font-family:'Goudy Old Style';padding:0.5em;border-radius:0.2em;font-size:30px;color:yellow">1. Introduction
<p style="font-family:cursive;font-size:17px;color:white"><u>Context</u>: Predict behavior to retain customers. You can analyze all relevant customer data and develop focused customer retention programs." [IBM Sample Data Sets]</p>
<p style="font-family:cursive;font-size:17px;color:white"><u>Content</u>: Each row represents a customer, each column contains customer’s attributes described on the column Metadata.</p>   
<ul style="font-family:cursive;font-size:17px;color:white">The data set includes information about:
<li> Customers who left within the last month – the column is called Churn</li>
<li > Services that each customer has signed up for – phone, multiple lines, internet, online security, online backup, device protection, tech support, and streaming TV and movies</li>
    <li >Customer account information – how long they’ve been a customer, contract, payment method, paperless billing, monthly charges, and total charges</li>
    <li>Demographic info about customers – gender, age range, and if they have partners and dependents</li></ul>
    
    
<p style="font-family:cursive;font-size:17px;color:white"><p style="font-family:cursive;font-size:17px; color:white" >
<a style="font-family:cursive;font-size:17px; color: yellow" href="https://www.kaggle.com/blastchar/telco-customer-churn" target="_blank"> <u>Dataset in Kaggle</u></a></p>
<p style="font-family:cursive;font-size:17px; color:white" >
<a style="font-family:cursive;font-size:17px; color: yellow" href="https://raw.githubusercontent.com/alexeygrigorev/mlbookcamp-code/master/chapter-03-churn-prediction/WA_Fn-UseC_-Telco-Customer-Churn.csv" target="_blank"> <u>GitHub Repository</u></a></p>
</div>

"""
"""
[back to top](#table-of-contents)
<a id="2"></a>

<div style="background:#2b6684;color:white; font-family:'Goudy Old Style';padding:0.5em;border-radius:0.2em;font-size:25px;color:yellow">2. Importing Libraries
    
</div>
"""
import pandas as pd
import numpy as np

import matplotlib.pyplot as plt
import seaborn as sns

plt.rcParams['figure.figsize'] = (16, 8)
plt.style.use('fivethirtyeight')
"""
[back to top](#table-of-contents)
<a id="3"></a>

<div style="background:#2b6684;color:white; font-family:'Goudy Old Style';padding:0.5em;border-radius:0.2em;font-size:25px;color:yellow">3. Loading and Reading Data
    
</div>
"""
df = pd.read_csv('../input/telco-customer-churn/WA_Fn-UseC_-Telco-Customer-Churn.csv')
df.head()
"""
[back to top](#table-of-contents)
<a id="4"></a>

<div style="background:#2b6684;color:white; font-family:'Goudy Old Style';padding:0.5em;border-radius:0.2em;font-size:25px;color:yellow">4. Data Cleaning: Cleaning Strings in Column and values
    
</div>
"""
df.columns = df.columns.str.lower().str.replace(' ', '_')

categorical_columns = list(df.dtypes[df.dtypes == 'object'].index)

for c in categorical_columns:
    df[c] = df[c].str.lower().str.replace(' ', '_')
    
df.head()
"""
[back to top](#table-of-contents)
<a id="5"></a>

<div style="background:#2b6684;color:white; font-family:'Goudy Old Style';padding:0.5em;border-radius:0.2em;font-size:25px;color:yellow">5. Data Preparation
    
</div>
"""
df.info()
df['totalcharges'].value_counts()
"""
We saw above that total charges column is object type
Hence we convert that to numerical column
we also see that when we replaced " " by "_" missing values here got replaced by "_"
"""
# We convert totalcharges to numerical and to ignore the errors (in our case  "_") we use errors='coerce'
# Store the results back to the original column
df.totalcharges = pd.to_numeric(df.totalcharges, errors='coerce')

#  Finally take care of the the missing values in totalcharges columns by filling up by 0.
df.totalcharges = df.totalcharges.fillna(0)
"""Next we convert YES/NO values in churn variable to int 1/0 and store it back to original col"""
df.churn = (df.churn == 'yes').astype(int)
"""
[back to top](#table-of-contents)
<a id="6"></a>

<div style="background:#2b6684;color:white; font-family:'Goudy Old Style';padding:0.5em;border-radius:0.2em;font-size:25px;color:yellow">6. Setting up the validation framework
    
</div>
"""
"""We are using sklearn train_test_split function to split the data into full_train and test set"""
from sklearn.model_selection import train_test_split

# Making full_train and test set
df_full_train, df_test = train_test_split(df, test_size=0.2, random_state=1)

# Next making train and validation set from full_train set
df_train, df_val = train_test_split(df_full_train, test_size=0.25, random_state=1)

# checking the number of values in datasets
print("Train_set %s:" %len(df_train), "Validation_set %s:" %len(df_val), "Test_set %s:" %len(df_test))

# Dropping indexes of these datasets
df_train = df_train.reset_index(drop=True)
df_val = df_val.reset_index(drop=True)
df_test = df_test.reset_index(drop=True)

# Creating Dependent Variables
y_train = df_train.churn.values
y_val = df_val.churn.values
y_test = df_test.churn.values

# Deleting target column i.e. churn from training datasets
del df_train['churn']
del df_val['churn']
del df_test['churn']
"""
[back to top](#table-of-contents)
<a id="7"></a>

<div style="background:#2b6684;color:white; font-family:'Goudy Old Style';padding:0.5em;border-radius:0.2em;font-size:25px;color:yellow">7. Exploratory Data Analysis
    
</div>
"""
"""Removing the index column"""
df_full_train = df_full_train.reset_index(drop=True)

"""Checking for missing values"""
df_full_train.isnull().sum()
"""Checking the target variable- churn, for the ratio of 0s and 1s
also known as Global Churn Rate"""
print(df_full_train['churn'].value_counts(normalize = True))

"""Also means Global Churn Rate of 1s as when adding in mean only 1 will be adding."""
print(df_full_train['churn'].mean())


"""Creating Categorical and Numerical Columns"""
useful_features = [col for col in df_full_train.columns if col not in ['customerid', 'churn']]
categorical = [col for col in useful_features if df_full_train[col].dtype == 'object'] + ['seniorcitizen']
numerical = [col for col in useful_features if col not in categorical]
"""
[back to top](#table-of-contents)
<a id="8.1"></a>

<div style="background:#2b6684;color:white; font-family:'Goudy Old Style';padding:0.5em;border-radius:0.2em;font-size:25px;color:yellow">8.1 Feature Importance: Churn Rate and Risk Ratio
    
</div>
"""
"""
# <div style="background:#2b6684;color:white; font-family:cursive;padding:0.5em;border-radius:0.2em;font-size:15px;color:white">1. Difference = Group Churn Rate - Global Churn Rate </div>
- **If this value is less than 0 : ``More Likely  to  churn``**
- **If this value is more than 0 : ``Less Likely  to  churn``**

# <div style="background:#2b6684;color:white; font-family:cursive;padding:0.5em;border-radius:0.2em;font-size:15px;color:white">2. Risk Ratio = Group Churn Rate / Global Churn Rate </div>
- **If this value is more than 1 : ``More Likely  to  churn``**
- **If this value is less than 1 : ``Less Likely  to  churn``**

"""
# Library to use display multiple output, especially when using loops
from IPython.display import display

"""Calculating Global Churn Rate"""
global_churn = df_full_train.churn.mean()

"""Creating a for loop to display Differences and Risk Ratio for different groups"""
for c in categorical:
    print(c)
    df_group = df_full_train.groupby(c)['churn'].agg(['mean', 'count'])
    df_group['diff'] = df_group['mean'] - global_churn
    df_group['risk'] = df_group['mean'] / global_churn
    display(df_group.style.background_gradient("bone_r"))
    print()
    print()
"""
[back to top](#table-of-contents)
<a id="8.2"></a>

<div style="background:#2b6684;color:white; font-family:'Goudy Old Style';padding:0.5em;border-radius:0.2em;font-size:25px;color:yellow">8.2 Feature Importance: Mutual Information 
<ul>
<li style= "font-size:15px;font-family:cursive;color:white";>Concept from information theory, it tells us how much we can learn about one variable if we know the value of another </li></ul>
</div>
"""
"""Library for mutual Information"""
from sklearn.metrics import mutual_info_score


"""Defining a function to calculate mutaul info score and it takes only one variable"""
def mutual_info_churn_score(series):
    return mutual_info_score(series, df_full_train.churn)

"""Applying the above function to categorical variables"""
mi = df_full_train[categorical].apply(mutual_info_churn_score)
mi.sort_values(ascending = False).to_frame().reset_index().rename({'index': 'Variables', 0: 'Improtance'}, axis = 1).style.background_gradient('crest')
"""
[back to top](#table-of-contents)
<a id="8.3"></a>

<div style="background:#2b6684;color:white; font-family:'Goudy Old Style';padding:0.5em;border-radius:0.2em;font-size:25px;color:yellow">8.3 Feature Importance: Correlation
<ul style= "font-size:15px;font-family:cursive;color:white">
<li >We use Correlation to udnerstand the relation between two numeric variables.</li>
<li>We use Correlation coefficient to quatify the realtion between two numeric variables.</li></ul>
</div>
"""
"""Checking the correaltion of numerical Columns with Target variable churn"""

df_full_train[numerical].corrwith(df_full_train.churn)
"""Correlation matrix"""
df[numerical].corr()
"""Using heatmap"""
sns.heatmap(df[numerical].corr(), square = True, annot = True, lw = 0.2)
"""Checking the correlation by groups in Numeric columns"""
print("Mean Churn Rate when Tenure <= 2: ",df_full_train[df_full_train.tenure <= 2].churn.mean())
print("Mean Churn Rate when Tenure between 2 and 12: ",df_full_train[(df_full_train.tenure > 2) & (df_full_train.tenure <= 12)].churn.mean())
print("Mean Churn Rate when Tenure more than 12: ",df_full_train[df_full_train.tenure > 12].churn.mean())
print()
print("Mean Churn Rate when monthlycharges <= 20: ",df_full_train[df_full_train.monthlycharges <= 20].churn.mean())
print("Mean Churn Rate when monthlycharges between 20 and 50: ",df_full_train[(df_full_train.monthlycharges > 20) & (df_full_train.monthlycharges <= 50)].churn.mean())
print("Mean Churn Rate when monthlycharges more than 50: ",df_full_train[df_full_train.monthlycharges > 50].churn.mean())
"""
[back to top](#table-of-contents)
<a id="9"></a>

<div style="background:#2b6684; font-family:'Goudy Old Style';padding:0.5em;border-radius:0.2em;font-size:25px;color:yellow">9. One Hot Encoding using DictVectorizer
    <ul style= "font-size:15px;font-family:cursive;color:white">
<li ;>Takes in Dictionary and converts itno vector form. Categorical columns are converted using One Hot Encoding. Numerical Values are untouched and returned as Vectors as is.</li>
<li >Sparsity of the matrix can be controlled by parameter sparse = False</li></ul>
</div>
"""
from sklearn.feature_extraction import DictVectorizer

dv = DictVectorizer(sparse=False)                              

train_dict = df_train[useful_features].to_dict(orient='records') # For making dict row-wise we use orient = 'records
X_train = dv.fit_transform(train_dict)

val_dict = df_val[useful_features].to_dict(orient='records') # For making dict row-wise we use orient = 'records'
X_val = dv.transform(val_dict)
"""
[back to top](#table-of-contents)
<a id="10"></a>

<div style="background:#2b6684; font-family:'Goudy Old Style';padding:0.5em;border-radius:0.2em;font-size:25px;color:yellow">10. Logistic Regression<ul style= "font-size:15px;font-family:cursive;color:white">
<li >Similar to Linear Regression as both of them are linear models.</li>
<li >Logistic Regression uses sigmoid function over the score of dot product and hence converts the results into the range of 0,1.</li>
<li >Sigmoid Function = $\large g(x_i) = \Large \frac{\mathrm{1} }{\mathrm{1} + e^{-x} }  $</li>
<li >Linear regression is straightforward to understand and explain, and can be regularized to avoid overfitting. In addition, linear models can be updated easily with new data using stochastic gradient descent.</li></ul>
</div> 
"""
def sigmoid(z):
    return 1 / (1 + np.exp(-z))
def linear_regression(xi):
    result = w0
    
    for j in range(len(w)):
        result = result + xi[j] * w[j]
        
    return result

def logistic_regression(xi):
    score = w0
    
    for j in range(len(w)):
        score = score + xi[j] * w[j]
        
    result = sigmoid(score)
    return result
"""
[back to top](#table-of-contents)
<a id="11"></a>

<div style="background:#2b6684; font-family:'Goudy Old Style';padding:0.5em;border-radius:0.2em;font-size:25px;color:yellow">11.  Training logistic regression with Scikit-Learn<ul style= "font-size:15px;font-family:cursive;color:white">
<li >Train a model with Scikit-Learn</li>
<li >Apply it to the validation dataset</li>
<li >Calculate the accuracy</li></ul>
</div> 
"""
from sklearn.linear_model import LogisticRegression

model = LogisticRegression(solver='lbfgs')
# solver='lbfgs' is the default solver in newer version of sklearn
# for older versions, you need to specify it explicitly
model.fit(X_train, y_train)
"""model intercept: Bias Term (w0)"""
print("w0 = ", model.intercept_[0])

"""Coeffficients: Weights (wi)"""
print("wi = ",model.coef_[0].round(3))
"""Applying the model to Validation set to predict the probability"""
y_pred = model.predict_proba(X_val)[:, 1]

"""Making churn decision using probability threshold of 0.5"""
churn_decision = (y_pred >= 0.5)

"""Accuracy Rate: mean of correct predictions"""
(y_val == churn_decision).mean()
"""Observe the above result in a dataset"""
df_pred = pd.DataFrame()
df_pred['probability'] = y_pred
df_pred['prediction'] = churn_decision.astype(int)
df_pred['actual'] = y_val

df_pred['correct'] = df_pred.prediction == df_pred.actual
display(df_pred)
display("Accuracy Rate", df_pred['correct'].mean())
"""
[back to top](#table-of-contents)
<a id="12"></a>

<div style="background:#2b6684; font-family:'Goudy Old Style';padding:0.5em;border-radius:0.2em;font-size:25px;color:yellow">12.  Model interpretation
    <ul style= "font-size:15px;font-family:cursive;color:white">
<li > Look at the coefficients</li>
<li >Train a smaller model with fewer features</li></ul>
</div> 
"""
"""Understanding the use of Zip function"""
a = [1, 2, 3, 4]
b = 'abcd'

"""zip() comnines the both values by position"""
dict(zip(a, b))
"""Now using zip to combine features and their respective coefficients"""
dict(zip(dv.get_feature_names(), model.coef_[0].round(3)))
"""Now to understand Interpretation: Let's take small subset of the variables"""
small = ['contract', 'tenure', 'monthlycharges']

"""Create a model on these subset of columns"""
df_train[small].iloc[:10].to_dict(orient='records')

# Creating training and Validation set
dicts_train_small = df_train[small].to_dict(orient='records')
dicts_val_small = df_val[small].to_dict(orient='records')

# Using dictVectorizer for OHE on categorical colums
dv_small = DictVectorizer(sparse=False)
dv_small.fit(dicts_train_small)

# Checking the feature names to see how many new created by DictVectorizer
dv_small.get_feature_names()
"""Creating Training set"""
X_train_small = dv_small.transform(dicts_train_small)

"""making model on Training set"""
model_small = LogisticRegression(solver='lbfgs')
model_small.fit(X_train_small, y_train)

"""Assigning variables w0 and wi's to bias term and weights"""
w0 = model_small.intercept_[0]
wi = model_small.coef_[0].round(3)

"""Finally zipping w0 and wi understand interpret the model prediction"""
dict(zip(dv_small.get_feature_names(), wi.round(3)))
"""
We are given following info about the customer:
- Contract : Two years
- monthly_charges : 30
- Tenure : 24

We also know the coeff/w of:
- Contract : two years = -0.949
- monthlycharges = 0.027
- tenure' = -0.036

Also bias term(w0) = -2.47

Basic Interpretation: When we dont know anything about the customer, customer is less likely to churn

Using our Linear Formula: 
w0 + w1.x1 + w2.x2 + . . .
we get:
"""

print(-2.47 + (-0.949) + 30 * 0.027 + 24 * (-0.036))

# Finally using sigmoid function on this value will give us the Probability of customer will churn or not
sigmoid(-3.473)
"""
[back to top](#table-of-contents)
<a id="13"></a>

<div style="background:#2b6684; font-family:'Goudy Old Style';padding:0.5em;border-radius:0.2em;font-size:25px;color:yellow">13.  Using the model
</div> 
"""
"""Now we train the model on the full_train set and predict on test set"""

dicts_full_train = df_full_train[categorical + numerical].to_dict(orient='records')

dv = DictVectorizer(sparse=False)
X_full_train = dv.fit_transform(dicts_full_train)

y_full_train = df_full_train.churn.values

model = LogisticRegression(solver='lbfgs')
model.fit(X_full_train, y_full_train)

dicts_test = df_test[categorical + numerical].to_dict(orient='records')

X_test = dv.transform(dicts_test)

y_pred = model.predict_proba(X_test)[:, 1]

churn_decision = (y_pred >= 0.5)

print("Accuracy on Test Set: ", (churn_decision == y_test).mean())
"""Predicting on an individual customer to check how the model behaves"""
customer = dicts_test[-1]
customer
X_small = dv.transform([customer])
print("Predicted Probability: ",model.predict_proba(X_small)[0, 1])
print("Original Value: ",y_test[-1])
"""
[back to top](#table-of-contents)
<a id="14"></a>

<div style="background:#2b6684; font-family:'Goudy Old Style';padding:0.5em;border-radius:0.2em;font-size:25px;color:yellow">14.  Summary
<ul style= "font-size:15px;font-family:cursive;color:white">
<li >Feature importance - risk, mutual information, correlation</li>
<li >One-hot encoding can be implemented with DictVectorizer</li>
<li >Logistic regression - linear model like linear regression</li>
<li >Output of log reg - probability</li>
<li >Interpretation of weights is similar to linear regression</li></ul>
</div>
"""