"""
1. For data analysis, follow [this (Exploratory Data Analysis)](https://www.kaggle.com/sabbiu/exploratory-data-analysis)
2. *[Current]* Logistic Regression
3. [Playing with pipeline](https://www.kaggle.com/sabbiu/playing-with-sklearn-pipeline)
"""
import numpy as np
import pandas as pd
train = pd.read_csv('../input/train.csv')
NUM_TRAIN = train.shape[0]
NUM_TRAIN
test = pd.read_csv('../input/test.csv')
combined = train.drop('Survived', axis=1).append(test)
combined.reset_index(inplace=True)
combined.drop('index', axis=1, inplace=True)
combined.isnull().sum()
"""
# Data Cleaning & Feature Engineering
"""
"""
Let's drop Cabin (as it has so many null data)
Also, dropping PassengerId (as it is not a useful feature)
"""
combined.drop(['Cabin', 'PassengerId', 'Ticket'], axis=1, inplace=True)
"""
**Fare**  
It has one null data.
"""
combined[combined['Fare'].isnull()]
combined[
    (combined['Pclass'] == 3) &
    (combined['Sex'] == 'male') &
    (combined['Embarked'] == 'S') &
    (combined['Age'] > 50) &
    (combined['Fare'].notnull())
]['Fare'].mean()
 combined.loc[1043,'Fare'] = 7.45
"""
**Embarked**  
It has two missing data
"""
combined[combined['Embarked'].isnull()]
train.loc[[61,829], 'Survived']
combined[
    (combined['Sex'] == 'female') &
    (combined['Pclass'] == 1) &
    (combined['Embarked'].notnull())
]['Embarked'].value_counts()
import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline
sns.boxplot(data = combined[
    (combined['Sex'] == 'female') &
    (combined['Pclass'] == 1) &
    (combined['Embarked'].notnull())
], x='Embarked', y='Age')
"""
Both port C and S seems to have approximately equal number of females from Pclass 1. From the boxplot, putting age 38 to pclass 'C' and age 62 to pclass 'S'.
"""
combined.loc[61, 'Embarked'] = 'C'
combined.loc[829, 'Embarked'] = 'S'
combined.isnull().sum()
"""
**Name**  
Let's first extract out title from name. Then, we can deal with age.
"""
import re
combined['Title'] = combined['Name'].apply(lambda x: re.findall('([A-Za-z]+)\.', x)[0])
combined['Title'].value_counts()
combined['Title'].replace(['Mr', 'Mrs', 'Miss', 'Master', 'Don', 'Rev', 'Dr', 'Mme', 'Ms',
       'Major', 'Lady', 'Sir', 'Mlle', 'Col', 'Capt', 'Countess',
       'Jonkheer', 'Dona'], ['Mr', 'Mrs', 'Miss', 'Master', 'Other', 'Other', 'Other', 'Miss', 'Miss',
       'Other', 'Other', 'Other', 'Miss', 'Other', 'Other', 'Other',
       'Other', 'Other'], inplace=True)
combined['Title'].value_counts()
combined.drop('Name', axis=1, inplace=True)
"""
**Age**
"""
combined['Age'].isnull().sum()
grouped_for_age = combined.dropna().groupby(['Pclass','Embarked', 'Title']).median()['Age'].reset_index()
grouped_for_age
grouped_for_age[
    (grouped_for_age['Pclass'] == 1) &
    (grouped_for_age['Embarked'] == 'C') &
    (grouped_for_age['Title'] == 'Master') 
]['Age'].values[0]
combined['Age'] = combined[['Pclass', 'Embarked', 'Title']].apply(lambda x: grouped_for_age[
    (grouped_for_age['Pclass'] == x[0]) &
    (grouped_for_age['Embarked'] == x[1]) &
    (grouped_for_age['Title'] == x[2]) 
]['Age'].values[0], axis=1)
combined.isnull().sum()
"""
**Familysize**
"""
combined['Familysize'] = combined['Parch'] + combined['SibSp'] + 1
combined.drop(['Parch', 'SibSp'], axis=1, inplace=True)
combined.info()
"""
**One hot encoding**
"""
combined = pd.get_dummies(combined, columns=['Sex', 'Embarked', 'Title'], drop_first=True)
combined.head()
"""
This is kind of final data. But, I want to check this data on three conditions:
1. Without Fare (because, Pclass might have redundant information)
2. As it is Age and Fare
3. Bucketing Age and Fare
"""
combined_1 = combined.drop('Fare', axis=1)
combined_2 = combined.copy()
pd.qcut(combined['Age'], 5).unique()
combined_3 = combined.copy()
def bucket_age(row):
    age = row['Age']
    if age > 39: return 5
    elif age > 30: return 4
    elif age > 26: return 3
    elif age > 23.5: return 2
    else: return 1

combined_3['Age'] = combined_3.apply(bucket_age, axis=1)
pd.qcut(combined['Fare'], 5).unique()
def bucket_fare(row):
    fare = row['Fare']
    if fare > 41.579: return 5
    elif fare > 21.558: return 4
    elif fare > 10.5: return 3
    elif fare > 7.854: return 2
    else: return 1

combined_3['Fare'] = combined_3.apply(bucket_fare, axis=1)
combined_1.head()
combined_2.head()
combined_3.head()
from sklearn.preprocessing import StandardScaler
scaled_combined_1 = pd.DataFrame(StandardScaler().fit_transform(combined_1),
                                 index=combined_1.index,
                                 columns=combined_1.columns)
scaled_combined_1.head()
scaled_combined_2 = pd.DataFrame(StandardScaler().fit_transform(combined_2),
                                 index=combined_2.index,
                                 columns=combined_2.columns)
scaled_combined_2.head()
scaled_combined_3 = pd.DataFrame(StandardScaler().fit_transform(combined_3),
                                 index=combined_3.index,
                                 columns=combined_3.columns)
scaled_combined_3.head()
"""
# Feature Correlation
"""
sns.heatmap(scaled_combined_1.corr(), annot=True, cmap='coolwarm')
sns.heatmap(scaled_combined_2.corr(), annot=True, cmap='coolwarm')
sns.heatmap(scaled_combined_3.corr(), annot=True, cmap='coolwarm')
"""
# Logistic Regression Model
"""
def train_test_target(df):
    train_ = df[:NUM_TRAIN]
    test_ = df[NUM_TRAIN:]
    target_ = train['Survived']
    return train_, test_, target_
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import confusion_matrix, classification_report, roc_curve, auc
X_1, test_1, y_1 = train_test_target(scaled_combined_1)
X_train_1, X_test_1, y_train_1, y_test_1 = train_test_split(X_1, y_1, test_size=0.3, random_state=42)
logmodel_1 = LogisticRegression(solver='liblinear')
logmodel_1.fit(X_train_1, y_train_1)
predictions_1 = logmodel_1.predict(X_test_1)
print(classification_report(y_test_1, predictions_1))
sns.heatmap(confusion_matrix(y_test_1, predictions_1), annot=True, cmap='coolwarm')
y_score_1 = logmodel_1.decision_function(X_test_1)
FPR_1, TPR_1, _ = roc_curve(y_test_1, y_score_1)
ROC_AUC_1 = auc(FPR_1, TPR_1)
plt.plot(FPR_1, TPR_1, label=f'ROC curve {ROC_AUC_1}')
plt.plot([0,1], [0,1], 'k--')
plt.xlim([0,1])
plt.ylim([0,1.05])
plt.legend()
logmodel_1.fit(X_1, y_1)

submission = pd.DataFrame({
    'PassengerId': test['PassengerId'],
    'Survived': logmodel_1.predict(test_1)
})
submission.to_csv('submission_1.csv', index=False)
"""
**Fitting model to other data**
"""
X_2, test_2, y_2 = train_test_target(scaled_combined_2)
X_train_2, X_test_2, y_train_2, y_test_2 = train_test_split(X_2, y_2, test_size=0.3, random_state=42)
logmodel_2 = LogisticRegression(solver='liblinear')
logmodel_2.fit(X_train_2, y_train_2)
predictions_2 = logmodel_2.predict(X_test_2)
X_3, test_3, y_3 = train_test_target(scaled_combined_3)
X_train_3, X_test_3, y_train_3, y_test_3 = train_test_split(X_3, y_3, test_size=0.3, random_state=42)
logmodel_3 = LogisticRegression(solver='liblinear')
logmodel_3.fit(X_train_3, y_train_3)
predictions_3 = logmodel_3.predict(X_test_3)
print(classification_report(y_test_2, predictions_2))
print(classification_report(y_test_3, predictions_3))
fig, ax = plt.subplots(1, 3, figsize=(15, 4))
sns.heatmap(confusion_matrix(y_test_1, predictions_1), annot=True, cmap='coolwarm', ax=ax[0])
sns.heatmap(confusion_matrix(y_test_2, predictions_2), annot=True, cmap='coolwarm', ax=ax[1])
sns.heatmap(confusion_matrix(y_test_3, predictions_3), annot=True, cmap='coolwarm', ax=ax[2])
y_score_2 = logmodel_2.decision_function(X_test_2)
FPR_2, TPR_2, _ = roc_curve(y_test_2, y_score_2)
ROC_AUC_2 = auc(FPR_2, TPR_2)

y_score_3 = logmodel_3.decision_function(X_test_3)
FPR_3, TPR_3, _ = roc_curve(y_test_3, y_score_3)
ROC_AUC_3 = auc(FPR_3, TPR_3)
plt.plot(FPR_1, TPR_1, label=f'ROC curve 1 {ROC_AUC_1}', color='red')
plt.plot(FPR_2, TPR_2, label=f'ROC curve 2 {ROC_AUC_2}', color='blue')
plt.plot(FPR_3, TPR_3, label=f'ROC curve 3 {ROC_AUC_3}', color='green')
plt.plot([0,1], [0,1], 'k--')
plt.xlim([0,1])
plt.ylim([0,1.05])
plt.legend()
logmodel_2.fit(X_2, y_2)
submission_2 = pd.DataFrame({
    'PassengerId': test['PassengerId'],
    'Survived': logmodel_2.predict(test_2)
})
submission_2.to_csv('submission_2.csv', index=False)

logmodel_3.fit(X_3, y_3)
submission_3 = pd.DataFrame({
    'PassengerId': test['PassengerId'],
    'Survived': logmodel_3.predict(test_3)
})
submission_3.to_csv('submission_3.csv', index=False)
"""
It seems, performance of model is in the order: 2 > 1 > 3.  
Let's perform all the works from now onwards on model 2.
"""
"""
# Cross Validation Score
"""
from sklearn.model_selection import cross_val_score
cv_score = cross_val_score(LogisticRegression(solver='liblinear'), X_2, y_2, cv=5, scoring='accuracy')
print(cv_score, np.mean(cv_score))
"""
# Hyperparameter Tuning and Optimization using GridSearchCV
"""
from sklearn.model_selection import GridSearchCV, StratifiedKFold
%%time
if False:
    C_values = [0.1, 0.2, 0.3, 0.4, 0.5, 0.8, 1.0, 2.0, 5.0, 10.0, 20.0]
    # New (Second) param grid
    param_grid = [{
        'penalty': ['l2'],
        'C': C_values,
        'solver': ['lbfgs', 'newton-cg', 'sag']
    }, {
        'penalty': ['l1', 'l2'],
        'C': C_values,
        'solver': ['liblinear', 'saga']
    }, {
        'penalty': ['none'],
        'solver': ['lbfgs', 'newton-cg', 'saga', 'sag']
    }, {
        'penalty': ['elasticnet'],
        'C': C_values,
        'l1_ratio': [0, 0.2, 0.5, 0.8, 1.0],
        'solver': ['saga']
    }]
    #     # Previous (First) param grid
    #     param_grid = [{
    #         'penalty': ['l2'],
    #         'C': C_values,
    #         'solver': ['lbfgs', 'newton-cg']
    #     }, {
    #         'penalty': ['l1', 'l2'],
    #         'C': C_values,
    #         'solver': ['liblinear']
    #     }, {
    #         'penalty': ['none'],
    #         'solver': ['lbfgs', 'newton-cg']
    #     }]
    grid = GridSearchCV(estimator=LogisticRegression(max_iter=5000),
                       param_grid=param_grid,
                       scoring='accuracy',
                       cv=StratifiedKFold(n_splits=5))
    grid.fit(X_2, y_2)
    print(f'Best Score: {grid.best_score_}')
    print(f'Best Parameters: {grid.best_params_}')
    print(f'Best Estimator: {grid.best_estimator_}')
    
"""
**With Previous param_grid**  
With scoring `accuracy`: `{'C': 0.2, 'penalty': 'l1', 'solver': 'liblinear'}`  
With scoring `roc_auc`: `{'C': 0.4, 'penalty': 'l2', 'solver': 'liblinear'}`
"""
roc_auc_params = {'C': 0.4, 'penalty': 'l2', 'solver': 'liblinear', 'max_iter':5000}
accuracy_params = {'C': 0.2, 'penalty': 'l1', 'solver': 'liblinear', 'max_iter':5000}

logmodel_roc_auc = LogisticRegression(**roc_auc_params)
logmodel_roc_auc.fit(X_2, y_2)
submission_2_roc_auc = pd.DataFrame({
    'PassengerId': test['PassengerId'],
    'Survived': logmodel_roc_auc.predict(test_2)
})
submission_2_roc_auc.to_csv('submission_2_roc_auc.csv', index=False)

logmodel_accuracy = LogisticRegression(**accuracy_params)
logmodel_accuracy.fit(X_2, y_2)
submission_2_accuracy = pd.DataFrame({
    'PassengerId': test['PassengerId'],
    'Survived': logmodel_accuracy.predict(test_2)
})
submission_2_accuracy.to_csv('submission_2_accuracy.csv', index=False)
"""
**With New param_grid (Comprising all solvers)**  
roc_auc: `{'C': 0.3, 'l1_ratio': 0.8, 'penalty': 'elasticnet', 'solver': 'saga'}`  
accuracy: `{'C': 0.8, 'penalty': 'l1', 'solver': 'saga'}`

"""
roc_auc_params_new = {'C': 0.3, 'l1_ratio': 0.8, 'penalty': 'elasticnet', 'solver': 'saga', 'max_iter': 5000}
accuracy_params_new = {'C': 0.8, 'penalty': 'l1', 'solver': 'saga', 'max_iter': 5000}

logmodel_roc_auc_new = LogisticRegression(**roc_auc_params_new)
logmodel_roc_auc_new.fit(X_2, y_2)
submission_2_roc_auc_new = pd.DataFrame({
    'PassengerId': test['PassengerId'],
    'Survived': logmodel_roc_auc_new.predict(test_2)
})
submission_2_roc_auc_new.to_csv('submission_2_roc_auc_new.csv', index=False)

logmodel_accuracy_new = LogisticRegression(**accuracy_params_new)
logmodel_accuracy_new.fit(X_2, y_2)
submission_2_accuracy_new = pd.DataFrame({
    'PassengerId': test['PassengerId'],
    'Survived': logmodel_accuracy_new.predict(test_2)
})
submission_2_accuracy_new.to_csv('submission_2_accuracy_new.csv', index=False)

"""
# Interpretation and Conclusion
"""
"""
These are the model accuracies I got.

| | Submission Score | CV Accuracy | CV ROC AUC |
|-|-|-|-|
| *Before Hyperparameter Tuning* ||||
|1 (removing fare feature)| 0.78947 | 0.82159 | 0.86572 |
|2 (all features)| **0.79425** | **0.82385** | 0.86866 |
|3 (bucketing age and fare) | 0.78468 | 0.81151 | **0.86908** |
| *After Hyperparameter Tuning (Data 2)* ||||
|First Param Grid (scoring=accuracy)| **0.80382** | 0.82496 | 0.86880 |
|First Param Grid (scoring=roc_auc)| 0.79904 | 0.82272 | 0.86883 |
|Second Param Grid (scoring=accuracy)| 0.79904 | **0.82609** | 0.86815 |
|Second Param Grid (scoring=roc_auc)| **0.80382** | 0.82383 | **0.86888** |

*Note: The CV Accuracy and CV ROC AUC are obtained by providing necessary parameters in Cross Validation Section.*

After performing hyperparameter tuning, models have comparatively higher accuracy (~0.80).

Know about different types of solver from here, https://stackoverflow.com/a/52388406/6892277
"""
"""
**logmodel_2**
"""
logmodel_2
intr_1 = pd.DataFrame(index=X_2.columns, data=logmodel_2.coef_[0], columns=['Slope'])
intr_1.iloc[intr_1['Slope'].abs().argsort()[::-1]]
"""
**logmodel_accuracy**
"""
logmodel_accuracy
intr_2 = pd.DataFrame(index=X_2.columns, data=logmodel_accuracy.coef_[0], columns=['Slope'])
intr_2.iloc[intr_2['Slope'].abs().argsort()[::-1]]
"""
**logmodel_roc_auc_new**
"""
logmodel_roc_auc_new
intr_3 = pd.DataFrame(index=X_2.columns, data=logmodel_roc_auc_new.coef_[0], columns=['Slope'])
intr_3.iloc[intr_3['Slope'].abs().argsort()[::-1]]
"""
Here, we can notice the following things:

From [EDA](https://www.kaggle.com/sabbiu/exploratory-data-analysis), we concluded the following:  
* **Pclass:** Passenger Class 1 have higher rate of survival.
  * Pclass has higher negative slope, which means, higher the value of Pclass, lower is the chance of survival.
* **Age:** Children (age lower than ~9) have higher rate of survival.
  * Age also has negative slope.
* **Sex:** Females have higher rate of survival.
  * Title_Mr and Sex_male, both have negative slope. While, Title_Mrs has positive slope.
  * But, from logmodel_2, Title_Miss has negative slope. This means, Title_Miss has lower chance of survival. From our analysis in EDA, this is not the case. It might be because, Title_Mr, and Sex_male might have captured most of the Survivor of Title_Miss.
* **Sex and Pclass:** Females of upper passenger class (1 and 2) have higher rate of survival.
* **Embarked:** People who embarked from *C* have comparatively higher rate of survival.
  * Embarked_C has been dropped.
* **Embarked and Pclass:** People who embarked from port *S* into class 3 have comparatively low survival rate. Very few poeple embarked from port *Q* went to class 1 and 2.
* **Family Size (Parch + SibSp + 1):** Highest chance of survival are for those families who have family size of 4.
  * Negative slope indicates that, larger families have lower chance of survival.
* **Family Size and Sex:** Females having family size less than 5 have higher rate of survival.

**Regularization**

When regularization is l2, also known as ridge regression, there is a very small slope associated with less useful features. And, for l1 regularization, also known as lasso regression, less useful features have 0 slope value, i.e, they are discarded.
![Lasso_Ridge](https://i.stack.imgur.com/A8qcK.png)
"""