"""
<img src="https://upload.wikimedia.org/wikipedia/commons/thumb/8/81/Titanic_in_color.png/320px-Titanic_in_color.png"/>
<center>Image Source: Fidodog14 and SandyShores03, Public domain, via Wikimedia Commons</center>
"""
"""
# About Competition
Titanic ML competition is the best first challenge for you to dive into ML competitions and familiarize yourself with how the Kaggle platform works.
 Machine learning can predict which passengers survived the Titanic shipwreck. The competition is simple: use machine learning to create a model that predicts which survivors survived the wreck.Competition file is available [here](https://www.kaggle.com/c/titanic).
"""
"""
# Load Competition Dataset
"""
"""
Competition dataset located in "/kaggle/input"; This path defined by Kaggle to access the competition file. We will list two files from this path as input files.
"""
import numpy as np # linear algebra
import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)
import os
from zipfile import ZipFile
for dirname, _, filenames in os.walk('/kaggle/input'):
    for filename in filenames:
        path=os.path.join(dirname, filename)
        if 'train' in path:
            __training_path=path
        elif 'test' in path:
            __test_path=path
"""
## Input Dataset
"""
#loaded files
print(f'Training path:{__training_path}\nTest path:{__test_path}')
# Kaggle Environment Prepration
#update kaggle env
import sys
#you may update the environment that allow you to run the whole code
!{sys.executable} -m pip install --upgrade scikit-learn=="0.24.2"
#record this information if you need to run the Kernel internally
import sklearn; sklearn.show_versions()
def proc_multifiles_zip(path, condition, delimiter=','):
    zip_file = ZipFile(path)
    for text_file in zip_file.infolist():
        if condition in text_file.filename:
            return pd.read_csv(zip_file.open(text_file.filename), delimiter=delimiter)
"""
# Exploratory Data Analysis (EDA)
## General Structure
Titanic Dataset includes <b>12</b> columns and <b>891</b> rows.
There are <b>3</b> different data types as follows: *int64, object, float64*.
"""
"""
Let's review the dataset description:
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>count</th>
      <th>mean</th>
      <th>std</th>
      <th>min</th>
      <th>25%</th>
      <th>50%</th>
      <th>75%</th>
      <th>max</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>PassengerId</th>
      <td>891.0</td>
      <td>446.000000</td>
      <td>257.353842</td>
      <td>1.00</td>
      <td>223.5000</td>
      <td>446.0000</td>
      <td>668.5</td>
      <td>891.0000</td>
    </tr>
    <tr>
      <th>Survived</th>
      <td>891.0</td>
      <td>0.383838</td>
      <td>0.486592</td>
      <td>0.00</td>
      <td>0.0000</td>
      <td>0.0000</td>
      <td>1.0</td>
      <td>1.0000</td>
    </tr>
    <tr>
      <th>Pclass</th>
      <td>891.0</td>
      <td>2.308642</td>
      <td>0.836071</td>
      <td>1.00</td>
      <td>2.0000</td>
      <td>3.0000</td>
      <td>3.0</td>
      <td>3.0000</td>
    </tr>
    <tr>
      <th>Age</th>
      <td>714.0</td>
      <td>29.699118</td>
      <td>14.526497</td>
      <td>0.42</td>
      <td>20.1250</td>
      <td>28.0000</td>
      <td>38.0</td>
      <td>80.0000</td>
    </tr>
    <tr>
      <th>SibSp</th>
      <td>891.0</td>
      <td>0.523008</td>
      <td>1.102743</td>
      <td>0.00</td>
      <td>0.0000</td>
      <td>0.0000</td>
      <td>1.0</td>
      <td>8.0000</td>
    </tr>
    <tr>
      <th>Parch</th>
      <td>891.0</td>
      <td>0.381594</td>
      <td>0.806057</td>
      <td>0.00</td>
      <td>0.0000</td>
      <td>0.0000</td>
      <td>0.0</td>
      <td>6.0000</td>
    </tr>
    <tr>
      <th>Fare</th>
      <td>891.0</td>
      <td>32.204208</td>
      <td>49.693429</td>
      <td>0.00</td>
      <td>7.9104</td>
      <td>14.4542</td>
      <td>31.0</td>
      <td>512.3292</td>
    </tr>
  </tbody>
</table>
"""
"""
<b> Is there any null value?</b> 
The answer is <b>Yes</b>; let's review top 3 of those columns with the number of Null values.

<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>Column</th>
      <th>#Null</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>Cabin</td>
      <td>687</td>
    </tr>
    <tr>
      <th>1</th>
      <td>Age</td>
      <td>177</td>
    </tr>
    <tr>
      <th>2</th>
      <td>Embarked</td>
      <td>2</td>
    </tr>
  </tbody>
</table>

As partial of the results shown above, there are total <b>3</b> columns with Null values.
"""
"""
There is <b>2</b> unique value in <b>Survived</b> column which is a target column.
let's see frequency of values for the target column of {col}:
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>Survived</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>549</td>
    </tr>
    <tr>
      <th>1</th>
      <td>342</td>
    </tr>
  </tbody>
</table>
"""
"""
## Skewness
In probability theory and statistics, skewness is a measure of the asymmetry of the probability distribution of a real-valued random variable about its mean. The skewness value can be positive, zero, negative, or undefined.

More detail can be found [here](https://en.wikipedia.org/wiki/Skewness) and the [Probability and Statistics Tables and Formulae](http://tomlr.free.fr/Math%E9matiques/Math%20Complete/Probability%20and%20statistics/CRC%20-%20standard%20probability%20and%20Statistics%20tables%20and%20formulae%20-%20DANIEL%20ZWILLINGER.pdf) by Zwillinger and Kokoska.

Here are two samples Skewness data for positive and negative skew data.<img src="https://upload.wikimedia.org/wikipedia/commons/f/f8/Negative_and_positive_skew_diagrams_%28English%29.svg" alt="source:Wikimedia">

First, we we will calculate the skewness for each columns in Titanic Dataset if each selected column has positive float/integer values.

Second, we will review the following conditions.

- if $Skewness>1$ or $Skewness<-1$, we will consider it as highly skewed;

- if $0.5<Skewness<=1$ or $-0.5<Skewness<=-1$, we will consider it as moderate skewed;

<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>Column</th>
      <th>Skewness</th>
      <th>Skewness_Type</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>Pclass</td>
      <td>-0.630548</td>
      <td>moderately skewed</td>
    </tr>
    <tr>
      <th>1</th>
      <td>SibSp</td>
      <td>3.695352</td>
      <td>highly skewed</td>
    </tr>
    <tr>
      <th>2</th>
      <td>Parch</td>
      <td>2.749117</td>
      <td>highly skewed</td>
    </tr>
    <tr>
      <th>3</th>
      <td>Fare</td>
      <td>4.787317</td>
      <td>highly skewed</td>
    </tr>
  </tbody>
</table>
"""
"""
# Finding Intresting Datapoints
Let's process each field by their histogram frequency and check if there is any intresting data point.

There are <b>4</b> number of intresting values in the following columns.
The below table shows each <b>Value</b> of each <b>Field</b>(column) with their total frequencies, <b>Lower</b> shows the lower frequency of normal distribution, <b>Upper</b> shows the upper bound frequency of normal distribution, and <b>Criteria</b> shows if the frequnecy passed <b>Upper bound</b> or <b>Lower bound</b>.
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>Field</th>
      <th>Value</th>
      <th>Frequency</th>
      <th>Lower</th>
      <th>Upper</th>
      <th>Criteria</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>Age</td>
      <td>24.00</td>
      <td>30</td>
      <td>0.0088</td>
      <td>29.9736</td>
      <td>Upper</td>
    </tr>
    <tr>
      <th>1</th>
      <td>SibSp</td>
      <td>0.00</td>
      <td>608</td>
      <td>5.0012</td>
      <td>607.7606</td>
      <td>Upper</td>
    </tr>
    <tr>
      <th>2</th>
      <td>Parch</td>
      <td>0.00</td>
      <td>678</td>
      <td>1.0018</td>
      <td>677.6640</td>
      <td>Upper</td>
    </tr>
    <tr>
      <th>3</th>
      <td>Fare</td>
      <td>8.05</td>
      <td>43</td>
      <td>1.0000</td>
      <td>42.9753</td>
      <td>Upper</td>
    </tr>
  </tbody>
</table>



For example, in the <b>Age</b> column the value of <b>24.0</b> has <b>30</b> repeatation but this number is not between Lower bound(0.0088) and Upper bound(29.97360000000002).


Let     $C_0=24.0$   and   $Freq(C_0)=30$     ,   $Upper(C_0)=29.97360000000002$     ,   $Lower(C_0)=0.0088$

$Freq(C_0) > Upper(C_0)$.
"""
"""
# Input Dataset
"""
def __load__data(__training_path, __test_path, concat=False):
	"""load data as input dataset
	params: __training_path: the training path of input dataset
	params: __test_path: the path of test dataset
	params: if it is True, then it will concatinate the training and test dataset as output
	returns: generate final loaded dataset as dataset, input and test
	"""
	# LOAD DATA
	import pandas as pd
	__train_dataset = pd.read_csv(__training_path, delimiter=',')
	__test_dataset = pd.read_csv(__test_path, delimiter=',')
	return __train_dataset, __test_dataset
__train_dataset, __test_dataset = __load__data(__training_path, __test_path, concat=True)
__train_dataset.head()
"""
### Visualization for data distribution of columns
"""
import matplotlib.pyplot as plt
import seaborn as sns
import numpy as np


fig, axes = plt.subplots(1, 2, figsize=(16, 4))
axes = axes.ravel()
for index,col in enumerate(['Age', 'Fare']):
	sns.histplot(__train_dataset[col],ax=axes[index])
	axes[index].tick_params(axis="x", rotation=90)
	axes[index].set_title("Distribution of %s" %col )
fig.tight_layout();


fig, axes = plt.subplots(2, 4, figsize=(16, 8))
axes = axes.ravel()
for index,col in enumerate(['Pclass', 'Sex', 'SibSp', 'Parch', 'Embarked']):
	sns.countplot(x = __train_dataset[col],ax=axes[index])
	axes[index].tick_params(axis="x", rotation=90)
	axes[index].set_title("Distribution of %s" %col )
for index in range(-3,0):
	axes[index].set_visible(False)
fig.tight_layout();


plt.figure(figsize=(5,5))
sns.countplot(x = __train_dataset["Survived"])
plt.xticks(rotation=90)
plt.title("Distribution Plots for Target Column")
fig.tight_layout();
# STORE SUBMISSION RELEVANT COLUMNS
__test_dataset_submission_columns = __test_dataset['PassengerId']
"""
### Discard Irrelevant Columns
In the given input dataset there are <b>1</b> column that can be removed as follows:* PassengerId *.
"""
# DISCARD IRRELEVANT COLUMNS
__train_dataset.drop(['PassengerId'], axis=1, inplace=True)
__test_dataset.drop(['PassengerId'], axis=1, inplace=True)
"""
## Remove Missing Values in Categorical Columns
"""
"""
In the given input dataset there are <b>5 columns</b> with  missing data as follows:<b>SibSp, Age, Fare, Pclass, Parch</b>
The following code removes the missing values from those columns. We use an average value (median) of each column to replace the Null values.
"""
"""
## Imputation Transformer
We will use which is an imputation transformer for completing missing values.
We can use out-of-the-box imputation transformer from Scikit-Learn packages. The detail and the list of complete parameters can be found [here](https://scikit-learn.org/stable/modules/generated/sklearn.impute.SimpleImputer.html)
"""
# PREPROCESSING-1
from sklearn.impute import SimpleImputer
import numpy as np
_NUMERIC_COLS_WITH_MISSING_VALUES = ['SibSp', 'Age', 'Fare', 'Pclass', 'Parch']
for _col in _NUMERIC_COLS_WITH_MISSING_VALUES:
    __simple_imputer = SimpleImputer(missing_values=np.nan, strategy='mean')
    __train_dataset[_col] = __simple_imputer.fit_transform(__train_dataset[_col].values.reshape(-1,1))[:,0]
    if _col in __test_dataset:
        __test_dataset[_col] = __simple_imputer.transform(__test_dataset[_col].astype(__train_dataset[_col].dtypes).values.reshape(-1,1))[:,0]
"""
## Remove Missing Values in Numerical Columns

In the given input dataset there are <b>5 columns </b> with  missing data as follows:<b>Cabin, Embarked, Ticket, Name, Sex</b>

The following code removes the missing values from those columns. We use average value (median) of each column to replace the null values.
"""
# PREPROCESSING-2
from sklearn.impute import SimpleImputer
import numpy as np
_STRING_COLS_WITH_MISSING_VALUES = ['Cabin', 'Embarked', 'Ticket', 'Name', 'Sex']
for _col in _STRING_COLS_WITH_MISSING_VALUES:
    __simple_imputer = SimpleImputer(missing_values=np.nan, strategy='most_frequent')
    __train_dataset[_col] = __simple_imputer.fit_transform(__train_dataset[_col].values.reshape(-1,1))[:,0]
    if _col in __test_dataset:
        __test_dataset[_col] = __simple_imputer.transform(__test_dataset[_col].astype(__train_dataset[_col].dtypes).values.reshape(-1,1))[:,0]
"""
## Encoding Ordinal Categorical Features
Let's transfer categorical features as an integer array.
We will use Ordinal Encoder as explained [here](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OrdinalEncoder.html).

In the given input dataset there are <b>2</b> columns that can be transfered to integer and it includes:* Sex,Embarked *.
"""
# PREPROCESSING-3
from sklearn.preprocessing import OrdinalEncoder
_CATEGORICAL_COLS = ['Sex', 'Embarked']
_ohe = OrdinalEncoder(handle_unknown="use_encoded_value", unknown_value=-1)
__train_dataset[_CATEGORICAL_COLS] = pd.DataFrame(_ohe.fit_transform(__train_dataset[_CATEGORICAL_COLS]), columns=_CATEGORICAL_COLS)
__test_dataset[_CATEGORICAL_COLS] = pd.DataFrame(_ohe.transform(__test_dataset[_CATEGORICAL_COLS]), columns=_CATEGORICAL_COLS)
"""
# Text Processing
The dataset has <b>3</b> text values as follows: <b>Name,Ticket,Cabin</b>.
Now, let's covert the text as follows.

- First, convert text to lowercase;

- Second, strip all punctuations;

- Finally, convert all numbers in text to 'num'; therefore, in the next step our model will use a single token instead of valriety of tokens of numbers.
"""
# PREPROCESSING-4
import nltk
import re
import string
_TEXT_COLUMNS = ['Name', 'Ticket', 'Cabin']
def process_text(__dataset):
    for _col in _TEXT_COLUMNS:
        process_text = [t.lower() for t in __dataset[_col]]
        # strip all punctuation
        table = str.maketrans('', '', string.punctuation)
        process_text = [t.translate(table) for t in process_text]
        # convert all numbers in text to 'num'
        process_text = [re.sub(r'\d+', 'num', t) for t in process_text]
        __dataset[_col] = process_text
    return __dataset
__train_dataset = process_text(__train_dataset)
__test_dataset = process_text(__test_dataset)
"""
### Visualization for feature heatmap
"""
_COLS_FOR_HEATMAP = ['Age', 'Fare', 'Pclass', 'Sex', 'SibSp', 'Parch', 'Embarked', 'Survived']
plt.figure(figsize=(20,20))
plt.title("Pearson Correlation HeatMap of Features")
sns.heatmap(__train_dataset[_COLS_FOR_HEATMAP].corr(),annot=True)
"""
### Target Column
We need to predict the target column.
Therefore, we need to detach the target column in prediction.
Note that if we don't drop the field, it will generate a model with high accuracy on training and worst accuracy on test (because the value in test dataset is Null).
Here is the list of *target column*: <b>Survived</b>
"""
# DETACH TARGET
__feature_train = __train_dataset.drop(['Survived'], axis=1)
__target_train =__train_dataset['Survived']
__feature_test = __test_dataset
"""
# Text Vectorizer
In the next step, we will transfer pre-processed text columns to a vector representation. The vector representations allows us to train a model based on numerical representations.
We will use TfidfVectorizer and more detail can be found [here](https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html).
"""
# PREPROCESSING-5
from sklearn.feature_extraction.text import TfidfVectorizer
import scipy.sparse as sparse
from scipy.sparse import hstack, csr_matrix
_TEXT_COLUMNS = ['Ticket', 'Name', 'Cabin']
__temp_train_data = __feature_train[_TEXT_COLUMNS]
__feature_train.drop(_TEXT_COLUMNS, axis=1, inplace=True)
__feature_train_object_array = []
__temp_test_data = __feature_test[_TEXT_COLUMNS]
__feature_test.drop(_TEXT_COLUMNS, axis=1, inplace=True)
__feature_test_object_array = []
for _col in _TEXT_COLUMNS:
    __tfidfvectorizer = TfidfVectorizer(max_features=3000)
    vector_train = __tfidfvectorizer.fit_transform(__temp_train_data[_col])
    __feature_train_object_array.append(vector_train)
    vector_test = __tfidfvectorizer.transform(__temp_test_data[_col])
    __feature_test_object_array.append(vector_test)
__feature_train = sparse.hstack([__feature_train] + __feature_train_object_array).tocsr()
__feature_test = sparse.hstack([__feature_test] + __feature_test_object_array).tocsr()
"""
# Training and Prediction
First, we will train a model based on pre-processed the training dataset.
Second, let's predict test values based on the trained model.
"""
"""
## Random Forest Classifier
We will use *RandomForestClassifier* which is a meta estimator that fits a number of decision tree classifiers on various sub-samples of the dataset and uses averaging to improve the predictive accuracy and control over-fitting.
The sub-sample size is controlled with the max_samples parameter if bootstrap=True (default), otherwise the whole dataset is used to build each tree.
More details about *RandomForestClassifier* can be found [here](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html).
"""
# MODEL
import numpy as np
from sklearn.ensemble import RandomForestClassifier
__model = RandomForestClassifier()
__model.fit(__feature_train, __target_train) 
__y_pred = __model.predict(__feature_test)
"""
# Submission File
We have to maintain the target columns in "submission.csv" which will be submitted as our prediction results.
"""
# SUBMISSION
submission = pd.DataFrame(columns=['PassengerId'], data=__test_dataset_submission_columns)
submission = pd.concat([submission, pd.DataFrame(__y_pred, columns=['Survived'])], axis=1)
submission.head()
# save submission file
submission.to_csv("kaggle_submission.csv", index=False)