# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: Python 3
#     name: python3
# ---

# + [markdown] id="view-in-github" colab_type="text"
# <a href="https://colab.research.google.com/github/Ciiku-Kihara/KNN-TITANIC-PROJECT-SPAM-MAIL-PREDICTION/blob/main/KNN_Titanic_Project.ipynb" target="_parent"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/></a>

# + [markdown] id="ROdDdygdOWJG"
# ## 1. Defining the question

# + [markdown] id="112GSu7CPl5-"
# ### a) Specifying the analysis question
#
# Build a model that determines whether a person survived the titanic accident or not.

# + [markdown] id="TBT5LvwBPmkP"
# ### b) Defining the metric for success
#
# Be able to effectively use K Nearest Neighbours(KNN) to build an efficient model that can predict whether a person survived the titanic accident or not with at least 90% accuracy 

# + [markdown] id="xdPS32qUPnPK"
# ### c) Understanding the context
#
# The RMS Titanic sank in the early morning hours of 15 April 1912 in the North Atlantic Ocean, four days into her maiden voyage from Southampton to New York City. The largest ocean liner in service at the time, Titanic had an estimated 2,224 people on board when she struck an iceberg.
#
# Her sinking two hours and forty minutes later at 02:20 on Monday, 15 April, resulted in the deaths of more than 1,500 people, making it one of the deadliest peacetime maritime disasters in history.
#
# Titanic sank with over a thousand passengers and crew still on board. Almost all of those who jumped or fell into the water drowned or died within minutes due to the effects of cold shock and incapacitation.
# The disaster shocked the world and caused widespread outrage over the lack of lifeboats, lax regulations, and the unequal treatment of the three passenger classes during the evacuation.

# + [markdown] id="h0uUC0OpPnxc"
# ### d) Recording the experimental design
#
# - Read and explore the given dataset
# - Define the appropriateness of the available data to answer the given question
# - Find and deal with outliers, anomalies, and missing data within the dataset
# - Perform Exploratory Data Analysis.
# - Build a model using KNN algorithm 
# - Challenge the solution.
# - Optimize the model
# - Make conclusions
#
#

# + [markdown] id="LznBGpVFPoOT"
# ### e) Data Relevance
#
# The dataset contains adequate information to build my models

# + [markdown] id="V0rTKmBbPor5"
# ## 2. Importing relevant libraries

# + id="lJHfv_3FbnN3"
# Loading necessary libraries

import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, confusion_matrix, classification_report
from sklearn.model_selection import GridSearchCV
from sklearn.preprocessing import MinMaxScaler, LabelEncoder
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis as LDA
from sklearn.neighbors import KNeighborsClassifier

# + [markdown] id="Zp8drdSRQjg0"
# ## 3. Loading and checking the data

# + id="3xFo-4FwcSpQ"
# Loading our dataset 

titanic =  pd.read_csv('titanic.csv')

# + colab={"base_uri": "https://localhost:8080/", "height": 229} id="tx1TuYKLce6R" outputId="d9039d55-90f4-4152-8ad3-0782d5a63b7e"
# Getting a preview of the first 5 rows

titanic.head()

# + colab={"base_uri": "https://localhost:8080/"} id="SApzDmXYemul" outputId="c0e624f9-1d6b-43a0-ae55-944e3c8eec88"
# Determining the number of rows and columns in the dataset

titanic.shape

# This dataset has 891 observations and 12 columns

# + colab={"base_uri": "https://localhost:8080/"} id="BndCOUEFfM9a" outputId="3af8050c-bd13-4405-f274-28077f8cf6fc"
# Determining the names of the columns present in the dataset

titanic.columns

# + colab={"base_uri": "https://localhost:8080/", "height": 284} id="YXt7KcR4llU1" outputId="c4c675af-7f8a-4d8a-cf77-b0c8104a05cc"
# Obtaining descriptive statistics

titanic.describe()

# + colab={"base_uri": "https://localhost:8080/"} id="g4WLnrwGfZFN" outputId="5e55c616-c4f3-441a-e797-970e77b08651"
# Checking if each column is of the appropriate data type

titanic.dtypes

# All the columns are of the appropriate data type.

# + [markdown] id="PSOXSQciSFn0"
# ## 4. External data source validation

# + [markdown] id="_6hx-9sDM66o"
# The dataset has been validated against the Titanic dataset on Kaggle. The link is provided below
#
# https://www.kaggle.com/hesh97/titanicdataset-traincsv 
#
# The data dictionary can be accessed here:
#
# https://www.kaggle.com/c/titanic/data

# + [markdown] id="iT1z5dFagd6g"
# ## 5. Data cleaning
#

# + id="d10waXUBgSnH" colab={"base_uri": "https://localhost:8080/", "height": 106} outputId="3ab3f489-82fa-407e-f55f-ea59c959f16d"
# Stripping the columns of any probable white spaces

titanic.columns = titanic.columns.str.strip().str.lower()

# Preview of the dataset

titanic.head(2)

# + colab={"base_uri": "https://localhost:8080/"} id="WJNqiDb0gSvi" outputId="7711a352-8c8b-465b-e84c-bb5938c806af"
# Checking for presence of null values

titanic.isnull().sum()

# There are 3 columns with missing values. 

# + colab={"base_uri": "https://localhost:8080/"} id="k3qhOCl_dIFW" outputId="401aa0ee-3a70-4c46-a16d-362ffea6c5d4"
# Previewing the columns again

titanic.columns

# + colab={"base_uri": "https://localhost:8080/"} id="7qQuV8F7B2Qg" outputId="03967d4c-829f-43b7-cedd-f9b78d0d286d"
# Dealing with the missing values
# Dropping the cabin column

titanic.drop('cabin', axis = 1, inplace=True)

# Imputing the median for age

titanic['age'] = titanic['age'].fillna(titanic['age'].median())

# Foward filling the null values in the embarked columns

titanic['embarked'] = titanic['embarked'].fillna(method = 'ffill')

# Confirming that there are no null values

titanic.isna().sum()

# + colab={"base_uri": "https://localhost:8080/", "height": 195} id="1ZifcAM8g4FE" outputId="5fa19d90-5f23-46d7-f77e-4f49b3165c67"
# Getting a preview of the dataset

titanic.head()

# + colab={"base_uri": "https://localhost:8080/"} id="RJii5OZdhoIz" outputId="a9f90093-02c4-428a-eb93-50da6a1f2116"
# Checking if there are any duplicated rows

titanic.duplicated().sum()

# There are no doplicates in the data

# + colab={"base_uri": "https://localhost:8080/"} id="tHFyTqExxFvP" outputId="eebe3805-7aaf-4e1c-ee69-4d12edce34b6"
# Previewing the columns again

titanic.columns

# + colab={"base_uri": "https://localhost:8080/"} id="gnCGxac1TiV4" outputId="9ead4789-284e-46aa-bb73-5f47e6200665"
# Checking for any anomalies in the categorical variables

qcol =['survived', 'pclass', 'sex', 'sibsp', 'parch', 'embarked']

for col in qcol:
  print(col, ':', titanic[col].unique(), '\n')

# The categorical columns have no errors in their entries

# + colab={"base_uri": "https://localhost:8080/", "height": 993} id="6nJRzbosrVw0" outputId="7af97d22-52d5-41df-b969-62e4e9113f69"
# Checking for Outliers

numna = ['fare', 'age']

for column in numna:
    plt.figure(figsize=(8,8))
    titanic.boxplot([column], fontsize= 14)
    plt.ylabel('count', fontsize = 14)
    plt.title('Boxplot - {}'.format(column), fontsize = 16)

# Both age and fare have outliers

# + colab={"base_uri": "https://localhost:8080/"} id="LxRfQNleU1pv" outputId="a225d3f1-afa2-4d50-dbf1-437f59a70d48"
# Determining how many rows would be lost if outliers were removed

# Calculating our first, third quantiles and then later our IQR
# ---
Q1 = titanic.quantile(0.25)
Q3 = titanic.quantile(0.75)
IQR = Q3 - Q1

# Removing outliers based on the IQR range
# ---
# 
titanic_new = titanic[~((titanic < (Q1 - 1.5 * IQR)) | (titanic > (Q3 + 1.5 * IQR))).any(axis=1)]

# Printing the shape of our new dataset
# ---
# 
print(titanic_new.shape)

# Printing the shape of our old dataset
# ---
#
print(titanic.shape)

# Number of rows removed

rows_removed = titanic.shape[0] - titanic_new.shape[0]
print('Number of rows removed:', rows_removed)

# Percentage of rows removed of the percentage
row_percent = (rows_removed/titanic.shape[0]) * 100
print('Percentage of rows removed:', row_percent)

# There will be no need to remove the outliers since it 
# doesn't seem like there are obvious errors in the data.
# Furthermore, removing outliers will reduce the diversity of the dataset 

# + id="YbGca6TYn1wv"
# Dropping the unnecessary columns

titanic.drop(['ticket', 'passengerid'], axis=1, inplace=True)

# + colab={"base_uri": "https://localhost:8080/", "height": 195} id="GLDq9sduodDr" outputId="779390d3-f38f-4dfe-e380-635077568937"
# Feature extraction
# Splitting the name column to get only the title

title = [i.split(",")[1].split(".")[0].strip() for i in titanic['name']]

titanic['title'] = pd.Series(title)

titanic.head()

# + colab={"base_uri": "https://localhost:8080/", "height": 195} id="ucL27q26slmt" outputId="cc81ef8d-a873-4e2d-a793-b1afdc460745"
# Dropping the name column

titanic.drop('name', axis=1, inplace=True)

# Checking on the columns again
titanic.head()

# + [markdown] id="hKEgp4tisMLz"
# ## 6. Exploratory Data Analysis

# + [markdown] id="a7vEdTzIsRoB"
# ### a) Univariate Analysis

# + colab={"base_uri": "https://localhost:8080/"} id="xXwwxR1qr0xK" outputId="0686d270-a2f3-4a30-c860-ada21dc52d6e"
# Previewing the columns

titanic.columns

# + colab={"base_uri": "https://localhost:8080/", "height": 1000} id="qAT5Lq44oq2h" outputId="cf059abd-791f-488d-a5ed-f042f4ff0921"
# Countplots for the categorical columns

qcol = ['survived', 'pclass', 'sex', 'sibsp', 'parch',
       'embarked']

for i in qcol:
  plt.figure(figsize= (8,8))
  sns.countplot(x = titanic[i])
  plt.title('Countplot - {}'.format(i), fontsize = 16)
  plt.xlabel(i , fontsize = 14)
  plt.ylabel('Count', fontsize = 14)
  plt.xticks(fontsize = 14)
  plt.yticks(fontsize = 14)
  plt.show()

# + [markdown] id="RTX6pMg2wNuR"
# From these countplots, we can make the following observations:
# Of the total number recorded
#
# * Those who did not survive were more than those who did. Approximately 600 passengers did not survive while about 300 passengers survived
# * Most of the passengers were in ticket class(pclass) 3 followed by passengers in ticket class 1. Ticket class 2 had the least number of passengers.
# * Of the total number of passengers, about 560 passengers were male and 300 were female.
# * Approximately 600 passengers had no siblings or spouses(sibsp) on board. About 200 passengers had 1 sibling or spouse on board.
# * More than 650 passengers had no parent or child(parch) on board
# *Approximately 650 passengers embarked from Southampton port(S), about 180 passengers from the Cherbourg(C) port while less than 100 passengers from the Queenstown port(Q).

# + colab={"base_uri": "https://localhost:8080/"} id="eQ68UEoJhx5I" outputId="9923a032-df98-4622-b3bd-ab06a84f119c"
# Frequency of the titles

titanic['title'].value_counts()

# All the titles from Master to Don I will title as 'Other'

# + colab={"base_uri": "https://localhost:8080/"} id="cFHlQyXTjMUU" outputId="b08ba7a0-05d0-40d5-e3c2-ccf6bf833aaf"
titanic['title'].unique()

# + colab={"base_uri": "https://localhost:8080/"} id="0Zd8wV68ig6Y" outputId="59c02c15-55ed-47fc-e2ab-5ca3dfd9c555"
# Replacing the less popular titles with other

other = ['Master', 'Don', 'Rev', 'Dr', 'Mme', 'Ms',
       'Major', 'Lady', 'Sir', 'Mlle', 'Col', 'Capt', 'the Countess',
       'Jonkheer']

titanic['title'] = titanic['title'].replace(other, 'Other')

titanic['title'].value_counts()

# Most of the passengers went by the title Mr while 67 of the 
# went by the less common titles like Mademoiselle, Jonkheer and so forth

# + colab={"base_uri": "https://localhost:8080/", "height": 523} id="rqgAxohLBjAQ" outputId="40b29833-8dd4-4dd9-cc90-a33da9d6ce9e"
# Histogram on age to see the shape of the distribution

def histogram(var1, bins):
  plt.figure(figsize= (10,8)),
  sns.set_style('darkgrid'),
  sns.set_palette('colorblind'),
  sns.histplot(x = var1, data=titanic,  bins = bins , shrink= 0.9, kde = True)

histogram('age', 60)
plt.title('Histogram of age of passengers', fontsize = 16)
plt.xlabel('Age', fontsize = 14)
plt.ylabel('Count', fontsize = 14)
plt.xticks(fontsize = 14)
plt.yticks(fontsize = 14)
plt.show()



# + colab={"base_uri": "https://localhost:8080/"} id="ynTcE2DXDjPo" outputId="bcad4823-57f9-4a73-cc51-3ff8e9743e1c"
# Measures of central tendency

titanic['age'].describe()

# The mean age is 29 
# The maximum age is 80
# The minimum age is 0.42
# 25% of passengers are aged below 22 while 50% of passengers are aged below
# 28 and 75% of passengers are aged below 35

# + colab={"base_uri": "https://localhost:8080/"} id="iXNSNm-lKGUM" outputId="1d5699e6-2e8d-4352-8082-fa096307a75b"
# Defining a variable to calculate variation

def variation(var):
  print('The skewness is:', titanic[var].skew())
  print('The kurtosis is:', titanic[var].kurt())
  print('The coefficient of variation is:', titanic[var].std()/titanic[var].mean())

# Checking on coefficent of variance, skewness and kurtosis

variation('age')

# The distribution is fairly symmetrical and is slightly leptokurtic.
# There is low variance in the age of the passengers

# + colab={"base_uri": "https://localhost:8080/", "height": 523} id="qCRS-q6dFrBQ" outputId="e4b6c1f4-8c58-4ecb-bd13-aab3f2699037"
# Histogram of passenger fare

histogram('fare', 60)
plt.title('Histogram of passenger fare', fontsize = 16)
plt.xlabel('Fare', fontsize = 14)
plt.ylabel('Count', fontsize = 14)
plt.xticks(fontsize = 14)
plt.yticks(fontsize = 14)
plt.show()


# + colab={"base_uri": "https://localhost:8080/"} id="C-sEQ6uNHUgq" outputId="e341b673-ba1e-4da0-cdb7-8135cfe89063"
# Measures of central tendency

titanic['fare'].describe()

# The mean fare is 32.2 
# The maximum fare  is 512
# The minimum fare is 0
# 25% of passenger fares are below 7.9 while 50% of pasenger fares are below
# 14 and 75% of passenger fares are below 31.

# The fare price of 512 represents the prce for the luxury tickets. It is not an error

# + colab={"base_uri": "https://localhost:8080/"} id="4SiI7TkY5Omt" outputId="342ed868-86d7-4c05-f601-42501d443fef"
# Checking on coefficent of variance, skewness and kurtosis

variation('fare')

# The distribution is skewed to the right and is leptokurtic.
# There is high variance in fare values

# + [markdown] id="MTfYjlQV9_GU"
# ### b) Bivariate Analysis

# + colab={"base_uri": "https://localhost:8080/"} id="5ud6_d-axjUr" outputId="ab04dfde-f62a-40be-89de-54269ac6fc0d"
# Preview of the coluns

titanic.columns

# + colab={"base_uri": "https://localhost:8080/", "height": 1000} id="WRQuQiI7OaL-" outputId="d4678210-7729-400e-d6d0-d1c9fa97087e"
# Bar charts to compare the whether a passenger survived vs the different categorical variables 
qcol1 =['pclass', 'sex', 'sibsp', 'parch',
       'embarked', 'title']

for i in qcol1:
  table=pd.crosstab(titanic[i],titanic['survived'])
  table.div(table.sum(1).astype(float), axis=0).plot(kind='bar', figsize = (10,8), stacked=False)
  plt.title('Bar Chart of Survived vs {}'.format(i), fontsize = 16)
  plt.xlabel(i, fontsize = 14)
  plt.ylabel('Proportion of Respondents', fontsize = 14)
  plt.xticks(rotation = 360, fontsize = 14)
  plt.yticks(fontsize = 14)
  plt.show()
  print(' ')
  print(' ')


# + [markdown] id="TUkYFbytjXrI"
# The observations made from the graphs above are:
#
# - 62% of those in ticket class 1 survived, 48% of those in ticket class 2 survived and only 25% of those in ticket class 3 survived.
# - More than 70% of females survived while less than 20% of males survived.
# - All of those who had 8 or 5 siblings or spouses on board died. Almost 70% of those who had no siblings or spouses on board did not survive.
# - More than 65% of those who had no parents or children on board died. However, the number of passengers who survived was higher for those who had 1 or 3 parents or siblings.
# - Of those who embarked from Cherbourg port 55% survived. Of those who embarked from Queenstown port, 40% survived and of those who embarked from Southampton Port, about 34% survived.  
# - The number of passengers who survived was higher for those who took the title 'Mrs', 'Miss' or had other less common titles. Less than 20% of individuals with the title 'Mr' survived

# + colab={"base_uri": "https://localhost:8080/", "height": 166} id="O3CrJRRByZx4" outputId="1754ee23-dc1a-43a7-a462-f8ac45b5ee43"
# Looking at the relationship between fare and the port that
# the passengers embarked from

titanic.groupby('embarked')['fare'].describe()

# Those who embarked from Cherbourg paid more on average than those who embarked 
# from the other 2 ports.

# + colab={"base_uri": "https://localhost:8080/", "height": 136} id="4wcHJHYb08Za" outputId="2cca8fe4-cde5-45e3-c17e-cebeee3db1fa"
# Looking at the relationship between fare and sex

titanic.groupby('sex')['fare'].describe()

# Females paid more on average than men

# + colab={"base_uri": "https://localhost:8080/", "height": 523} id="9q-NF2sVoYzD" outputId="b8ecf73c-07da-44b8-ad94-199237c33ce5"
# Comparing age and survival status

plt.figure(figsize=(10,8))
sns.boxplot(x = titanic['survived'], y= titanic['age'])
plt.title('Age vs survival status', fontsize = 16)
plt.xlabel('Survival status', fontsize = 14)
plt.ylabel('Age', fontsize = 14)
plt.xticks(fontsize = 14)
plt.yticks(fontsize = 14)
plt.show()

# There is not much of a difference in the ages for those who survived and those that
# did not survive.

# + colab={"base_uri": "https://localhost:8080/", "height": 523} id="xli8Lz1a6GDb" outputId="8cefe2d5-17a3-4c16-9627-73048fd42355"
# Comparing fare and survival status

plt.figure(figsize=(10,8))
sns.boxplot(x = titanic['survived'], y= titanic['fare'])
plt.title('Passenger fare vs survival status', fontsize = 16)
plt.xlabel('Survival status', fontsize = 14)
plt.ylabel('Passenger fare', fontsize = 14)
plt.xticks(fontsize = 14)
plt.yticks(fontsize = 14)
plt.show()

# Those who survived had a higher median fare than those who did not

# + [markdown] id="dmHL48kwIIb7"
# ### c) Multivariate Analysis 

# + colab={"base_uri": "https://localhost:8080/"} id="26Tj9yNAAFEn" outputId="430fcb4d-caa6-463a-f491-a242d6f0dbb2"
# Getting a preview of the columns again

titanic.columns

# + id="zvpcz_P2I8aw" colab={"base_uri": "https://localhost:8080/", "height": 505} outputId="04ad196f-c1a3-47b1-d060-0e3d5440a55c"
# Correlation Heatmap for numerical features

plt.figure(figsize=(10,8)) 
sns.heatmap(titanic.corr(),annot=True)
plt.title('Correlation matrix', fontsize = 16)
plt.xticks(fontsize = 14)
plt.yticks(fontsize = 14) 
plt.show()

# There is a moderately high correlation between sibsp and parch.
# The correlation coefficient between the two is 0.41

# + [markdown] id="G29Pa1eMPl8R"
# ## 7. Feature Selection by checking for multicollinearity
# > Multicollinearity is the occurrence of high intercorrelations among 2 or more independent variables in a model with multiple features.
#
# > Testing for multicollinearity is extremely important because it can lead to:
#
# * Inaccurate estimates of the regression coefficients
# * Inflation of the standard errors of the regression coefficients
# * False and non-significant p-values
# * Degredation of the predictability of the model
#
# Checking for correlations in independent variables helps to do features selection. Any variables with a VIF value above 4 will be dropped and will not be used in building the model

# + colab={"base_uri": "https://localhost:8080/"} id="tCGUS-p7CTVp" outputId="0dc7e8d1-aef3-450f-b653-cf6160c1c9fa"
# Encoding the categorical columns

for i in qcol1:
  titanic[i] = titanic[i].astype('category')
  titanic[i] = titanic[i].cat.codes

# Confirming the changes
for i in qcol1:
  print(i, ':', titanic[i].unique(), '\n')
 
print(titanic.dtypes)

# + id="bjb2OKj7EUVN"
# Specifying X and y

X = titanic.drop('survived', 1)
y = titanic['survived']

# + colab={"base_uri": "https://localhost:8080/", "height": 284} id="dwxiYWOUEubr" outputId="fa732f8b-6d30-4e4f-d170-880808bd3680"
# Checking for multicollinearity

correlations = X.corr()
correlations

# + colab={"base_uri": "https://localhost:8080/", "height": 284} id="h9S4YG8gE0x2" outputId="8c7b8891-90f7-4bca-9aff-1fee280824eb"
# Creating the VIF dataframe

VIF = pd.DataFrame(np.linalg.inv(correlations.values), index = correlations.index, columns = correlations.columns)
VIF

# + colab={"base_uri": "https://localhost:8080/"} id="7FDfLwUlFE3n" outputId="a27be8be-f035-4a72-dc8e-cfd715ad890e"
# Extracting the VIF values

pd.Series(np.diag(VIF), index=[VIF.columns])

# There is no multicollinearity

# + [markdown] id="h8yrx7-28pK8"
# There is no multicollinearity so no features will be dropped.

# + [markdown] id="k4tcH-U_F3np"
# ## 8. Implementing the solution

# + [markdown] id="9yysvwtaGFWQ"
# ### a) Building my model using K Nearest Neighbors algorithm.
#
# KNN is a non parametric algorithm that can be used with both classification and regression problems. However, it is more widely used to solve classification problems.
#
# KNN works by finding the distances between a new point and all the examples in the data, selecting the specified number examples (K) closest to the new point, then votes for the most frequent label (in the case of classification) or averages the labels (in the case of regression).
#
# Some of the advantages of using KNN are:
# * It is easy to use 
# * Has quick calculation time
# * Does not make any assumptions about the data (non-parametric)
#
#
#

# + colab={"base_uri": "https://localhost:8080/"} id="OySYU6dYJz-l" outputId="98a2931d-9a5f-48aa-9262-115fdc8e81d0"
# Splitting into train and test using 80-20 ratio

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, random_state = 99)

# fitting and making predictions

classifier = KNeighborsClassifier()

classifier.fit(X_train, y_train)

y_pred = classifier.predict(X_test)

# Evaluating the model

print('Accuracy:', accuracy_score(y_test, y_pred), '\n')
print(confusion_matrix(y_test, y_pred), '\n')
print( classification_report(y_test, y_pred))

# At 80-20 split ratio, the accuracy is 70.9%

# + [markdown] id="tcUkJnXH201K"
# * With an 80-20 split the accuracy is 70.9%
#
# From the first row of the confusion matrix we see that:
# * 87 passengers were correctly classified as having not survived  
# * 26 were wrongly classified as having not survived.
#
# From the second row of the confusion matrix we see that 
# * 26 passengers where wrongly classified as having survived 
# * 93 were correctly classified as having survived
#
# The recall is 77% 
#

# + colab={"base_uri": "https://localhost:8080/"} id="112Q7CnHLz5k" outputId="5ffe9dfc-58e5-4c8d-f040-6541fadc7f48"
# Splitting into train and test using 70-30 ratio

X_train1, X_test1, y_train1, y_test1 = train_test_split(X, y, test_size = 0.3, random_state = 99)

# fitting and making predictions

classifier = KNeighborsClassifier()

classifier.fit(X_train1, y_train1)

y_pred1 = classifier.predict(X_test1)

# Evaluating the model

print('Accuracy:', accuracy_score(y_test1, y_pred1), '\n')
print(confusion_matrix(y_test1, y_pred1), '\n')
print( classification_report(y_test1, y_pred1))

# At 70-30 split ratio, the accuracy has risen to 73.1%

# + [markdown] id="gmM9DBUb5V-R"
# * With an 70-30 split the accuracy is 73.1%
#
# From the first row of the confusion matrix we see that:
# * 140 passengers were correctly classified as having not survived  
# * 34 were wrongly classified as having not survived.
#
# From the second row of the confusion matrix we see that 
# * 38 passengers where wrongly classified as having survived 
# * 56 were correctly classified as having survived
#
# The recall is 80%

# + colab={"base_uri": "https://localhost:8080/"} id="kvS1N3uvMepH" outputId="6ed62c31-4925-451f-9c90-cf20d1fbac5d"
# Splitting into train and test using 60-40 ratio

X_train2, X_test2, y_train2, y_test2 = train_test_split(X, y, test_size = 0.4, random_state = 99)

# fitting and making predictions

classifier = KNeighborsClassifier()

classifier.fit(X_train2, y_train2)

y_pred2 = classifier.predict(X_test2)

# Evaluating the model

print('Accuracy:', accuracy_score(y_test2, y_pred2), '\n')
print(confusion_matrix(y_test2, y_pred2), '\n')
print( classification_report(y_test2, y_pred2))

# At 60-40 split ratio, the accuracy of the model dropped to 69.7%
# which was lower than the 80-20 split

# I will choose to optimize the 70-30 option

# + [markdown] id="Gb-xHuXB5rP9"
# * With an 60-40 split the accuracy is 69.7%
#
# From the first row of the confusion matrix we see that:
# * 169 passengers were correctly classified as having not survived  
# * 61 were wrongly classified as having not survived.
#
# From the second row of the confusion matrix we see that 
# * 47 passengers where wrongly classified as having survived 
# * 80 were correctly classified as having survived
#
# The recall is 73%
#

# + [markdown] id="psgdp3q8TLv0"
# The 70-30 split recoreded the highest accuracy and recall before any optimiation technique is put in place. Therefore I will optimize the model built using the 70-30 split

# + [markdown] id="aukLrlGkN-fn"
# **Optimizing the model**
#
# KNN can be optimized using the following techniques:
# * Dimensionality Reduction with Linear Discriminant Analysis
# * Rescaling our data which makes the distance metric more meaningful. 
# * Changing the distance metric for different applications.
# * Implementing weighted voting
# * Applying  additional appropriate nearest-neighbor techniques

# + colab={"base_uri": "https://localhost:8080/"} id="ikaiPba-SxEQ" outputId="e1c89cef-4066-40d8-b3f9-da89708b21f9"
# Performing feature scaling

scale = MinMaxScaler()
X_train1 = scale.fit_transform(X_train1)
X_test1 = scale.transform(X_test1)

# fitting and making predictions

classifier = KNeighborsClassifier()

classifier.fit(X_train1, y_train1)

y_pred1 = classifier.predict(X_test1)

# Evaluating the model

print('Accuracy:', accuracy_score(y_test1, y_pred1), '\n')
print(confusion_matrix(y_test1, y_pred1), '\n')
print( classification_report(y_test1, y_pred1))

# After feature scaling, the accuracy has risen from 73.1% to 80.2%
# The recall also increased from 80% to 86%

# + id="MI8601nSOKsf"
# Hyper parameter tuning

param_grid = {'n_neighbors' : np.arange(3,21,2),
              'algorithm' : ['auto', 'kd_tree'],
              'metric' : ['minkowski', 'euclidean', 'manhattan'],
              'weights' : ['uniform', 'distance']}

search = GridSearchCV(classifier, param_grid = param_grid, cv = 10, scoring= 'accuracy' )

# + colab={"base_uri": "https://localhost:8080/"} id="HHrl-bQ7POM6" outputId="20d62687-f24f-4a3f-b5ce-68d3c92764cf"
# Getting the best parameters

search.fit(X_train1, y_train1)
print(search.best_params_)
print(search.best_score_)

# + colab={"base_uri": "https://localhost:8080/"} id="_Ticm0jlPa0a" outputId="cb1dc5e9-75f2-4306-cb7f-b750fad002c5"
# Fitting and making predictions using the suggested parameters

classifier = KNeighborsClassifier(n_neighbors=11,  metric='manhattan')

classifier.fit(X_train1, y_train1)

y_pred1 = classifier.predict(X_test1)

# Evaluating the model

print('Accuracy:', accuracy_score(y_test1, y_pred1), '\n')
print(confusion_matrix(y_test1, y_pred1), '\n')
print( classification_report(y_test1, y_pred1))

# The recall after optimizing the model increased from 86% to 88%. However,
# the accuracy if the model did not increase by much

# + colab={"base_uri": "https://localhost:8080/"} id="O4dSzENkQ855" outputId="11fc76d1-a905-44e3-c2df-9053235fb077"
# Attempting to use dimensionality reduction

lda = LDA(n_components=1)
X_train1 = lda.fit_transform(X_train1, y_train1)
X_test1 = lda.transform(X_test1)

classifier.fit(X_train1, y_train1)

y_pred1 = classifier.predict(X_test1)

# Evaluating the model

print('Accuracy:', accuracy_score(y_test1, y_pred1), '\n')
print(confusion_matrix(y_test1, y_pred1), '\n')
print( classification_report(y_test1, y_pred1))

# With dimensionality reduction there is a reduction in the accuracy of the model


# + [markdown] id="CsIUlQ617pCr"
# ## 9. Challenging the solution
#
# - I will compare the performance of KNN with Catboost
#

# + colab={"base_uri": "https://localhost:8080/"} id="yas0kTZWirR9" outputId="996bdf24-5a66-4bb9-802a-02c4db86b335"
# Instaling catboost

# !pip install catboost

# + colab={"base_uri": "https://localhost:8080/"} id="bNZCL3n5i9af" outputId="ae4ea0f5-e145-4817-b3f5-f8842d932e18"
# Converting age and fare to integers

X['age'] = X['age'].astype('int64')
X['fare'] = X['fare'].astype('int64')

# Changing the data to categorical columns

for i in qcol1:
  X[i] = X[i].astype('category')

titanic.dtypes

# Confirming the changes

X.dtypes

# + colab={"base_uri": "https://localhost:8080/"} id="JRxfEhbXhuVc" outputId="4779bafa-423f-4230-abbd-9b7d8768472a"
# Building the model

from catboost import CatBoostClassifier

# Splitting X and y

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.3, random_state = 99)

# Building the model

cat = CatBoostClassifier(learning_rate=0.2, max_depth=5, iterations=1010)

#Fitting the model

cat.fit(X_train,y_train, cat_features= qcol1)

# Making predictions

y_pred_cat = cat.predict(X_test)

# Checking for accuracy

print('Accuracy:', accuracy_score(y_test, y_pred_cat), '\n')
print(confusion_matrix(y_test, y_pred_cat), '\n')
print( classification_report(y_test, y_pred_cat))

# With a learning rate of 0.2, max_depth of 5 and 1010 iterations
# the accuracy came up to 83.9%

# + [markdown] id="QYS02OfFIJDG"
# ## 10. Conclusion
#
# Some of the conclusions that can be made from this project:
#
# - For this specific dataset, the KNN optimization procedure that worked best was feature scaling.
# - Changing the distance metric to Manhattan increased the recall by 2%
# - When compared to the model created using catboost, Catboost did better. It recorded a higher recall by 1% and higher accuracy by about 3%

# + [markdown] id="v8eH73txK2sl"
# # Recommendations
#
# Some of the recommendations I would make are:
#
# * When building a model using KNN, perform feature scaling to improve the accuracy of your model. If your data is skewed, perform normalization. If it is normal, perform standardization.
# * Try to play around with other algorithms such as Random Forest and Gradient Boost and compare model performance.
# * Acquire a dataset with high dimensionality and perform dimensionality reduction on it. Compare its performance before and after the dimensionality reduction.

StarcoderdataJupyterScriptsDedupFiltered