"""
Hi, guys.

This is my first join the Kaggle competition and write code in python (I used R before, and my best score in this competition also is made by R). So, I hope you all can help me to correct my error and misunderstanding of any concept and process (especially the stacking ensemble part).

In this note book, I try to roughly implement the whole workflow. here is the composition of my code:

[1]. import and load data

[2]. Data Cleansing 

[3]. EDA

[4]. Feature Engineering

[5]. Modeling selection 

[6]. Ensemble Generation


Here is a wired question I find: in the grid search part, I use 'random_state' or 'seed' to control the Randomness, but I find the best parameter is lightly different each time.
"""
# import libraries
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.ensemble import RandomForestClassifier
from sklearn.ensemble import RandomForestRegressor
from sklearn.ensemble import ExtraTreesClassifier
from sklearn.model_selection import GridSearchCV
from sklearn.model_selection import train_test_split
import xgboost as xgb

# load data set
df_train = pd.read_csv('../input/titanic/train.csv')
df_train.head()
df_test = pd.read_csv('../input/titanic/test.csv')
df_test.head()
# remove the "Survived" column in "df_train" and join bind two sets
df = pd.concat([df_train.drop("Survived", axis=1), df_test])
df.head()
# remove all string columns: "Name", "Ticket", "Cabin"
df = df.drop(["Name", "Ticket", "Cabin"], axis = 1)
df.head()
"""
# Data Cleansing
"""
# check missing value
df.isna().sum()
# deal with NA in Age
df[df.Age.isna()]
# there are many rows contain NA in Age columns and I use the median Age to replace the NA
df.Age = df.Age.fillna(df.Age.median())

# deal with NA in Fare
df[df.Fare.isna()]
# it is a 3th class passenger and there is strongly relationship between Fare and Class,so I use the mean(Fare) in 3th class to replace the missing value
df.Fare = df.Fare.fillna(df.Fare[df.Pclass ==3].median())

# deal with NA in Embarked
df[df.Embarked.isna()]
# the Embarked is independent, so I use mode to replace the NA
df.Embarked = df.Embarked.fillna(df.Embarked.mode()[0])
# check missing value again
df.isna().sum()
# update the df_train
df_train.Age = df.Age[0:len(df_train)]
df_train.Fare = df.Fare[0:len(df_train)]
df_train.Embarked = df.Embarked[0:len(df_train)]
"""
# EDA
In the EDA part, I have two tasks:

First, check all variables' distribution.

Second, check the relationship between survived and other variables in df_train
"""
# Pclass
ind = list(set(df.Pclass))
count = df.groupby("Pclass").count().Sex
count0 = df_train[df_train.Survived == 0].groupby("Pclass").count().Sex
count1 = df_train[df_train.Survived == 1].groupby("Pclass").count().Sex

# distrubition
plt.bar(ind, count)
plt.ylim(0,750)
plt.show()

# relationship
p0 = plt.bar(ind, count0)
p1 = plt.bar(ind, count1, bottom = count0)
plt.ylim(0,750)
plt.legend((p0[0], p1[0]), ('Not Survived','Survived'))
plt.show()
# Sex
ind = sorted(list(set(df.Sex)))
count = df.groupby("Sex").count().Pclass
count0 = df_train[df_train.Survived == 0].groupby("Sex").count().Survived
count1 = df_train[df_train.Survived == 1].groupby("Sex").count().Survived

# distrubition
plt.bar(ind, count)
plt.ylim(0,900)
plt.show()

# relationship
p0 = plt.bar(ind, count0)
p1 = plt.bar(ind, count1, bottom = count0)
plt.ylim(0,900)
plt.legend((p0[0], p1[0]), ('Not Survived','Survived'))
plt.show()
# Age
#distribution
plt.boxplot(df.Age)
plt.show()

#relationship
p0 = plt.hist(df_train.Age[df_train.Survived == 0], width = 5, alpha = 0.5)
p1 = plt.hist(df_train.Age[df_train.Survived == 1], width = 5, alpha = 0.5)
plt.legend(('Not Survived','Survived'))
plt.show()
# SibSp
ind = list(set(df.SibSp))
count = df.groupby("SibSp").count().Pclass

# distrubition
plt.bar(ind, count)
plt.ylim(0,900)
plt.show()

# relationship
sib = df_train.groupby(['SibSp','Survived']).agg(total=('Pclass', 'size'))
sib["Percentage"] = sib / sib.groupby(level=0).sum() *100
sib
# Parch
ind = list(set(df.Parch))
count = df.groupby("Parch").count().Pclass

# distrubition
plt.bar(ind, count)
plt.ylim(0,900)
plt.show()

# relationship
par = df_train.groupby(['Parch','Survived']).agg(total=('Pclass', 'size'))
par["Percentage"] = par / par.groupby(level=0).sum() *100
par
# Fare
#distribution
plt.boxplot(df.Fare)
plt.show()

#relationship
p1 = plt.hist(df_train.Fare[df_train.Survived == 0], bins = 20, alpha = 0.5)
p2 = plt.hist(df_train.Fare[df_train.Survived == 1], bins = 20, alpha = 0.5)
plt.legend(('Not Survived','Survived'))
plt.show()
# Embarked
ind = sorted(list(set(df.Embarked)))
count = df.groupby("Embarked").count().Pclass
count0 = df_train[df_train.Survived == 0].groupby("Embarked").count().Survived
count1 = df_train[df_train.Survived == 1].groupby("Embarked").count().Survived

# distrubition
plt.bar(ind, count)
plt.ylim(0,900)
plt.show()

# relationship
p0 = plt.bar(ind, count0)
p1 = plt.bar(ind, count1, bottom = count0)
plt.ylim(0,900)
plt.legend((p0[0], p1[0]), ('Not Survived','Survived'))
plt.show()
"""
# Feature Engineering
"""
#Encode Features
#Sex 0: female, 1: male
df.Sex = df.Sex.replace("female", 0)
df.Sex = df.Sex.replace("male", 1)

#Embarked 0:C, 1:Q, 2:S
df.Embarked = df.Embarked.replace("C",0)
df.Embarked = df.Embarked.replace("Q",1)
df.Embarked = df.Embarked.replace("S",2)
#genearate dummy variables for Pclass, SibSp, Parch, and Embarked
df_dummy = pd.get_dummies(df, columns = ['Pclass', 'SibSp', 'Parch', 'Embarked'])
# create train and test sets
train_x = df_dummy[0:len(df_train)]
train_y = df_train.Survived
train_x
# check the feature inportance by randomforest
model_rf = RandomForestRegressor()
model_rf.fit(train_x, train_y)

ind =  model_rf.feature_importances_.argsort()
plt.barh(train_x.columns[ind], model_rf.feature_importances_[ind])
# some Prach and SibSp dummy variables are low importance and only exist in the train set or test set. So, I plan to encode them and reduce the number of feature
# in SibSp, 0: 0 ,1: 1, 2: 2+
# in Parch, 0: 0, 1: 1, 2: 2+
df.loc[df.SibSp >= 2, 'SibSp'] = 2
df.loc[df.Parch >= 2, 'Parch'] = 2

df.head()
#repeat. geneate dummy variables and create train and test sets
df_dummy = pd.get_dummies(df, columns = ['Pclass', 'SibSp', 'Parch', 'Embarked'])

# create train and test sets
train_x = df_dummy[0:len(df_train)]
train_y = df_train.Survived
test_x = df_dummy[len(df_train):]
"""
# Model Selection
"""
# model list used in this case
models = ["Random Forest", "Xgboost", "Extra Randomized Trees"]
scores = []
# Grid Search
# Random Forset
grid_rf = {
    "n_estimators": np.linspace(100,1000,5, dtype = int),
    "max_depth": [3,5,7],
    "max_features": [2,3,4,5,6,7,8,9],
    "min_samples_leaf": [3,5,7],
    "min_samples_split":[3,5,7],
    "random_state": [2020,2021]
}

model_rf = RandomForestClassifier()
search_rf = GridSearchCV(estimator = model_rf, 
                         param_grid = grid_rf, 
                         cv = 5,
                         n_jobs = -1,
                         verbose = 2)
# do not run this code in your local PC, it is time consuming
search_rf.fit(train_x, train_y)
# find the best parameters
best_rf = search_rf.best_params_
best_rf
# it seems RandomForestRegressor cannot accept the dict type parameters, so I have to rewrite the best parameters
final_rf =  RandomForestClassifier(**best_rf)
final_rf.fit(train_x, train_y)
pred_rf = final_rf.predict(test_x)
pred_rf
# Xgboost
grid_xgb = {
    'booster': ['gbtree', 'gblinear'],
    'objective': ['binary:logistic'],
    'subsample': [0.6,0.7,0.8,0.9],
    'colsample_bytree': [0.6,0.7,0.8,0.9],
    'eta': [0.05,0.1,0.2,0.3],
    'max_depth': [3,5,7],
    'seed': [2021,2022],
    'eval_metric': ['logloss']
}


model_xgb = xgb.XGBClassifier()
search_xgb = GridSearchCV(estimator = model_xgb, 
                         param_grid = grid_xgb, 
                         cv = 5,
                         n_jobs = -1,
                         verbose = 2)
# do not run this code in your local PC, it is time consuming
search_xgb.fit(train_x, train_y)
# find the best parameters
best_xgb = search_xgb.best_params_
best_xgb
# final xgboost model
final_xgb = xgb.XGBClassifier(**best_xgb)
final_xgb.fit(train_x, train_y)
pred_xgb = final_xgb.predict(test_x)
pred_xgb
# Extra Randomized Trees. this part is very simular with random forest, so I copy my code to here
grid_ert = {
    "n_estimators": np.linspace(100,1000,5, dtype = int),
    "max_depth": [3,5,7],
    "max_features": [2,3,4,5,6,7,8,9],
    "min_samples_leaf": [3,5,7],
    "min_samples_split":[3,5,7],
    "random_state": [2020,2021]
}

model_ert = ExtraTreesClassifier()
search_ert = GridSearchCV(estimator = model_ert, 
                         param_grid = grid_ert, 
                         cv = 5,
                         n_jobs = -1,
                         verbose = 2)
# do not run this code in your local PC, it is time consuming
search_ert.fit(train_x, train_y)
# find the best parameters
best_ert = search_ert.best_params_
best_ert
# final Extra Randomized Trees model
final_ert =  ExtraTreesClassifier(**best_ert)
final_ert.fit(train_x, train_y)
pred_ert = final_ert.predict(test_x)
pred_ert
"""
# Ensemble Generation
"""
# a majority vote ensemble
def ensumble_majority(trainx, trainy, testx):
    final_rf =  RandomForestClassifier(**best_rf)
    final_rf.fit(trainx, trainy)
    pred_rf = final_rf.predict(testx)
    
    final_ert =  ExtraTreesClassifier(**best_ert)
    final_ert.fit(trainx, trainy)
    pred_ert = final_rf.predict(testx)
    
    final_xgb = xgb.XGBClassifier(**best_xgb)
    final_xgb.fit(trainx, trainy)
    pred_xgb = final_xgb.predict(testx)
    
    collection = pd.DataFrame({'pred_rf':pred_rf,'pred_ert':pred_ert,'pred_xgb':pred_xgb})
    collection['sum'] = collection.sum(axis = 1)
    collection.loc[collection['sum'] > 1.5, 'pred'] = 1
    collection.loc[collection['sum'] < 1.5, 'pred'] = 0
    
    return(collection.pred)
    
pred = ensumble_majority(train_x, train_y, test_x)

res = pd.DataFrame({'PassengerId':test_x.PassengerId, 'Survived': pred})
res.to_csv('result.csv', index=False)
# a simple ensemble can use the majority vote rule, but here I try to build a simple stacking ensemble.
# I use random forest and Extra Randomized Trees as basic models, and choose xgboost as the second layer models

def ensemble_stacking(trainx, trainy, testx):
    '''
    this function is to ensemble three models' predictions: random forest, extra randomized trees, and xgboost
    '''
    # first, split the train set into two equal parts.
    px1,px2, py1, py2 = train_test_split(trainx, trainy, random_state= 2021, test_size=0.5)
    
    # second, train basic models by the two sets.
    # random forest
    final_rf =  RandomForestClassifier(**best_rf)
    final_rf.fit(px1,py1)
    pred_rf = final_rf.predict(px2)
    
    
    # extra randomize trees
    final_ert =  ExtraTreesClassifier(**best_ert)
    final_ert.fit(px2,py2)
    pred_ert = final_ert.predict(px1)
    
    # third, combind the two basic models predictions and as a new feature add into the original train set.
    # it means to create a new train setand the new set = original train set + tow basic models' predictions
    
    new_x = pd.concat([px1, px2])
    new_y = pd.concat([py1, py2])
    feature = np.concatenate([pred_ert,pred_rf])
    new_x['feature'] = feature
    
    # finally, train xgboost model by the new set and make the final prediction.
    
    final_xgb = xgb.XGBClassifier(**best_xgb)
    final_xgb.fit(train_x, train_y)
    pred_xgb = final_xgb.predict(test_x)
    pred_xgb
    
    return(pred_xgb)
# call ensemble function
pred = ensemble_stacking(train_x, train_y, test_x)
res = pd.DataFrame({'PassengerId':test_x.PassengerId, 'Survived': pred})
res.to_csv('result_stacking.csv', index=False)
"""
Thank you for viewing my notebook!
I want to share some of my experiences here.

[1] more try, more findings. I check every model I used in this competition. I find some models can get better scores and parts of models' predictions are better than the ensemble result. (In R, I try more than 10 models, but in Python, I just choose 3 models.)

[2] string columns are very important. In my code, I delete all string columns because I don't know how to deal with them. But I notice that many high score notebooks extract information from the string columns and generate features finally. So, I guess that is why I cannot reach the top 10% :(

[3] Generating FEATURES! I think the most important step in the whole process is to determine features. The quality of features determines your score's mean value, and the rest parts, like grid search and ensemble, are the sd of your score. Therefore, if you want to get a higher score, pay more attention to your features.
"""