"""
## "What are the characteristics of a terror attack which could predict which terror organisation may have been responsible for an incident ? "

### 1. Motive : Motive may be political
### 2. Description of the attack
### 3. Area or Region of attack
### 4. Weapons used
### 5. Pattern of attacks (duration of attacks)

These insights will guide the feature engineering process.


#### Problem Formulation: Lots of missing values is a spefic charateristic of Global Terrorist Database. Thus even lots of attacks have not been attributed to a particular terrorist group. This also leads to the problem formulation of this challenge solution.

1.The attacks have not been attributed to a particular terrorist group are assigned to the Test Dataset.

2.The models that have been attributed to a particular terrorist group acts as a Train Dataset.

#### Challenge Goal: To accurately train a model to predict the terrorist organisations behind the attacks that have been not been attributed to a terrorist group. 


"""
"""
## 1.
### Import packages
"""
#Importing libraries

import os
import pandas as pd 
import numpy as np

from IPython.display import display
import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline
sns.set(style='white', context='notebook', palette='deep')

colors = ["#66c2ff", "#5cd6d6", "#00cc99", "#85e085", "#ffd966", "#ffb366", "#ffb3b3", "#dab3ff", "#c2c2d6"]
sns.set_palette(palette = colors, n_colors = 4)

print('Data Manipulation, Mathematical Computation and Visualisation packages imported!')

import warnings
def ignore_warn(*args, **kwargs):
    pass
warnings.warn = ignore_warn #ignore annoying warning (from sklearn and seaborn)
warnings.filterwarnings("ignore", category=DeprecationWarning)
print('Deprecation warning will be ignored!')
"""
### 2.
## Load data

The Pandas package helps us work with our datasets.

"""
#Uploading the 5 excel sheets consisting the GTD data

xl_1=pd.ExcelFile(r"../input/gtd1993_0718dist.xlsx") #Uploading Excel sheet1
xl_2=pd.ExcelFile(r"../input/gtd_14to17_0718dist.xlsx") #Uploading Excel sheet2
xl_3=pd.ExcelFile(r"../input/gtd_70to95_0718dist.xlsx") #Uploading Excel sheet3
xl_4=pd.ExcelFile(r"../input/gtd_96to13_0718dist.xlsx") #Uploading Excel sheet4
#xl_5=pd.ExcelFile(r"C:\Users\aniru\OneDrive\Desktop\Studies\Data Science\Fellowship.AI\data5.xlsx") #Uploading Excel sheet5

print('Excel Files of GTD Uploaded!')
#Parsing the excel sheets to be suitable for pandas dataframe. 
#The sheets are named as "Data" in all the five GTD excel files.

df_1= xl_1.parse("Data") #Excel sheet1 parsed into dataframe1 (df_1)
df_2= xl_2.parse("Data") #Excel sheet2 parsed into dataframe2 (df_2)
df_3= xl_3.parse("Data") #Excel sheet3 parsed into dataframe3 (df_3)
df_4= xl_4.parse("Data") #Excel sheet4 parsed into dataframe4 (df_4)
#df_5= xl_5.parse("Data") #Excel sheet5 parsed into dataframe5 (df_5)

print('Excel sheets converted to DataFrames!')
#Appending all the 5 dataframes into a single dataframe

#df_2 has been appended into df_1
df_1=df_1.append(df_2, ignore_index=True)
#df_3 has been appended into df_1
df_1=df_1.append(df_3, ignore_index=True)
#df_4 has been appended into df_1
df_1=df_1.append(df_4, ignore_index=True)


# The number of rows and columns of terror_df, found out by pd.Dataframe.shape

#df_1 acts as a backup dataframe 
terror_df=df_1.copy()

terror_df.head(10)

#terror_df is the copy of df1 where all other dataframes have been appended
terror_df.shape
"""
## 2.

### Dataset Preparation
"""
# To avoid confusion,the dataset is restricted to only attacks that were of terrorist nature.

terror_df = terror_df[(terror_df.crit1 == 1) & (terror_df.crit2 == 1) & (terror_df.crit3 == 1) & (terror_df.doubtterr == 0)]
"""
The GTD database is known for its lots of missing values. We check percentage of missing values in each columns and remove columns having more than 20% of missing data.
"""
#missing value counts in each of these columns

miss = terror_df.isnull().sum()/len(terror_df)
miss = miss[miss > 0]
miss.sort_values(inplace=True)
miss

# Removing columns having more than 20% of missing values

frac = len(terror_df) * 0.8
terror_df=terror_df.dropna(thresh=frac, axis=1)
terror_df= terror_df.replace(-9,0)
terror_df= terror_df.drop(["city"], axis=1)
print('Removing columns having 20% of missing values')
#missing value counts in each of these columns after removing columns having more than 20% of missing values

miss = terror_df.isnull().sum()/len(terror_df)
miss = miss[miss > 0]
miss.sort_values(inplace=True)
miss
#replacing few strings with generalised strings to be common for all rows

terror_df.weaptype1_txt.replace(
    'Vehicle (not to include vehicle-borne explosives, i.e., car or truck bombs)',
    'Vehicle', inplace = True)

terror_df.gname = terror_df.gname.str.lower()
# replacing 'nkill', 'nwound', 'ncasualties' with has_casualities boolean

terror_df['ncasualties'] = terror_df['nkill'] + terror_df['nwound']
terror_df['has_casualties'] = terror_df['ncasualties'].apply(lambda x: 0 if x == 0 else 1)
terror_df= terror_df.drop(['nkill', 'nwound', 'ncasualties'], axis=1)
"""
### A missing value is an entry in a column that has no assigned value. This can mean multiple things:

1. A missing value may be the result of an error during the production of the dataset. 
This could be a human error, or machinery error depending on where the data comes from.
2. A missing value in some cases, may just mean a that a 'zero' should be present. In which case, it can be replaced by a 0. The data description provided helps to address situations like these.
3. However, missing values represent no information. Therefore, does the fact that you don't know what value to assign an entry, mean that filling it with a 'zero' is always a good fit?
4. Some algorithms do not like missing values. Some are capable of handling them, but others are not. Therefore since we are using a variety of algorithms, it's best to treat them in an appropriate way.

If we have missing values, we have two options:

1. Delete the entire row
2. Fill the missing entry with an imputed value

Lets see, what I have done with this dataset.
"""
#visualising missing values

miss = terror_df.isnull().sum()/len(terror_df)
miss = miss[miss > 0]
miss = miss.to_frame()
miss.columns = ['count']
miss.index.names = ['Name']
miss['Name'] = miss.index

#plot the missing value count
sns.set(style="whitegrid", color_codes=True)
sns.barplot(x = 'Name', y = 'count', data=miss)
plt.xticks(rotation = 90)
plt.show()
"""
### In order to impute missing numeric values we do the following

#### 1. Replace missing 'specificity' and 'natlty1' with the mode, i.e replacing with the values which occur the most.

#### 2. Replace other missing numeric values with 0 as replacing it with any other imputing methods (for eg: mean, median , mode) will result in identifying wrong characteristic/ behaviour of a terror organisation.
"""
# imputing with mode
terror_df['specificity'] = terror_df['specificity'].fillna(terror_df['specificity'].mode()[0])
terror_df['natlty1'] = terror_df['natlty1'].fillna(terror_df['natlty1'].mode()[0])

#imputing with value '0'
terror_df['guncertain1'] = terror_df['guncertain1'].fillna(terror_df['guncertain1'].fillna(0))
terror_df['targsubtype1'] = terror_df['targsubtype1'].fillna(terror_df['targsubtype1'].fillna(0))
terror_df['weapsubtype1'] = terror_df['weapsubtype1'].fillna(terror_df['weapsubtype1'].fillna(0))
terror_df['ishostkid'] = terror_df['ishostkid'].fillna(terror_df['ishostkid'].fillna(0))
terror_df['latitude'] = terror_df['latitude'].fillna(terror_df['latitude'].fillna(0))
terror_df['longitude'] = terror_df['longitude'].fillna(terror_df['longitude'].fillna(0))

print("Numerical variables Imputed!")
"""
### Imputing missing categorical values with 'None'
"""
for col in ("target1","natlty1_txt", "provstate", "targsubtype1_txt", "weapsubtype1_txt"):
    terror_df[col] = terror_df[col].fillna("None")
print("Categorical variables Imputed!")

"""
All the attacks have not been attributed to a particular terrorist group are collected to a unknown_df dataset which will later act as test dataset.
"""
unknown_df= terror_df[terror_df['gname'] == "unknown"]
# removing all unknown_df values from terror_df

terror_df=terror_df[~terror_df.isin(unknown_df)]
terror_df= terror_df.dropna()
# dropping "iyear", "eventid", "imonth", "iday", "extended" as they are redundant.

terror_df=terror_df.drop(["iyear", "eventid", "imonth", "iday", "extended"], axis=1)
terror_df= terror_df.dropna()

unknown_df=unknown_df.drop(["iyear", "eventid", "imonth", "iday", "extended"], axis=1)
unknown_df= unknown_df.dropna()
"""
### This step acts as a crucial step in this Label pre-processing step. 

### In this problem the column "gname" acts as a Label in the training steps.

#### There are a high number of terrorist organisations (3058) as shown in below step named in GTD which are also attributed to multiple terrorist attacks. This will also be difficult to formulate as a classification problem as it will need lots of computation power to predict name of each individual terrorist organisation behind a attack. Thus in order to fit this problem to computation of 4GB RAM local machine following steps have been undertaken.

All the organisations are classified as Dangerous Groups, Medium Organisations and Small Organisations based on their attack frequency and count by using "BINNING".

1. Organisations having 75% of attacks attributed are classified as Dangerous Groups
2. Organisations having 75% of attacks attributed are classified as Medium Org
3. Organisations having 75% of attacks attributed are classified as Small Org
"""
# All the gname stored into an another dataset Y

Y = pd.DataFrame()
Y = terror_df["gname"]
Y_count= Y.value_counts()
print(Y_count)
# Binning whole gname values based on their frequency as explained in above steps.

Y_bins=pd.qcut(Y_count, [0, .5, .75, 1], labels=["Small Org", "Medium Org" ,"Dangerous Groups"])
Y_bins= Y_bins.to_dict()

Y=Y.reset_index()
Y=Y.drop(["index"], axis=1)

Label = []

for x in Y["gname"]:
    for key, value in Y_bins.items():
           if x==key:
            Label.append(value)

Y["Label"]= pd.Series(Label)
# Example of dataset after Binning 

Y['Label'].head(5)
# Verifying the Binning process with barchart

plt.subplot(1, 3, 3)
sns.barplot(x="Label", y=Y.index, data=Y);
# Label encoding the "Small Org", "Medium Org", and "Dangerous Groups" into integers as they are ordinal variables and as it will be easier for training the model

Y['Label'] = Y['Label'].map({"Small Org":1, "Medium Org":2, "Dangerous Groups":3})
Y['Label'].shape
#Storing the binned and encoded terror organisation names back to terror_df dataset.

terror_df["Label"]= Y["Label"].values
# As gname has been binned and encoded into "Label" column, "gname" is dropped. "Label" now acts as a label dataset while training.

terror_df= terror_df.drop(["gname"], axis=1)
"""
### 3.
### Exploratory Data Analysis

### Correlation matrix

Now that missing values and outliers have been treated, I will analyse each feature in more detail. This will give guidance on how to prepare this feature for modelling. I will analyse the features based on the different aspects of the terror organisation available in the dataset.
"""
#preparing the correlation matrix

corr = terror_df.corr()
plt.subplots(figsize=(30, 30))
cmap = sns.diverging_palette(150, 250, as_cmap=True)
sns.heatmap(corr, cmap="RdYlBu", vmax=1, vmin=-0.6, center=0.2, square=True, linewidths=0, cbar_kws={"shrink": .5}, annot = True);
print (corr['Label'].sort_values(ascending=False)[:15], '\n') #top 15 values
print ('----------------------')
print (corr['Label'].sort_values(ascending=False)[-5:]) #last 5 values`
"""
The most common relationship we may think of between two variables, would be a straight line or a linear relationship. 

What this means is that if we increase the predictor by 1 unit, the response always increases by X units. 

However, not all data has a linear relationship and therefore it may be necessary for our model to fit the more complex relationships in the data.

There are a variety of curve-fitting methods we can choose from to help us with this.

The most common way to fit curves to the data is to include polynomial terms, such as squared or cubed predictors.


Typically, we choose the model order by the number of bends you need in your line. Each increase in the exponent produces one more bend in the curved fitted line. It’s very rare to use more than a cubic term.

As all the variables in most of the top columns (from correlation matrix) have values 0 and 1, its not necessary to follow this step but in order to explain my thinking process, I have described this step.

Also, If the response data follows a pattern that descends down to a lower bound, or ascends up to an upper bound, we can fit this type of relationship by including the reciprocal (1/x) of one or more predictor variables in the model, (For eg: INT_MISC in this dataset)

"""
#Dropping the gname column from the unknown_df dataset. Its filled with "unknown" values thus its no use for us.

unknown_df= unknown_df.drop(["gname"], axis= 1)
# Storing Label column from terror_df as our Label Dataset for training process

y_train = terror_df.Label.values
# Dropping Label from terror_df as we will later convert this dataset into train dataset.

terror_df= terror_df.drop(["Label"], axis= 1)
# Concating terror_df and unknown_df into one column for encoding categorical variables and name it as all_data

# Storing shape of train dataset and test dataset for later use
ntrain = terror_df.shape[0] # terror_df is train dataset
ntest = unknown_df.shape[0] # unknown_df is test dataset

all_data= pd.concat((terror_df, unknown_df)).reset_index(drop=True)

print("all_data shape: {}".format(all_data.shape))
cols = terror_df.columns

# List all columns having categorical variables

num_cols = terror_df._get_numeric_data().columns
list(set(cols) - set(num_cols))
# One-hot encoding all nominal variables

all_data["targtype1_txt"] = pd.get_dummies(all_data["targtype1_txt"])
all_data["region_txt"] = pd.get_dummies(all_data["region_txt"])
all_data["country_txt"] = pd.get_dummies(all_data["country_txt"])
all_data["dbsource"] = pd.get_dummies(all_data["dbsource"])
all_data["provstate"] = pd.get_dummies(all_data["provstate"])
#Label encoding all categorical variables

from sklearn import preprocessing

le = preprocessing.LabelEncoder()
le.fit(all_data["attacktype1_txt"])
list(le.classes_)
all_data["attacktype1_txt"]= pd.Series(list(le.transform(all_data["attacktype1_txt"]))) 

le_2 = preprocessing.LabelEncoder()
le_2.fit(all_data["weapsubtype1_txt"])
list(le_2.classes_)
all_data["weapsubtype1_txt"]= pd.Series(list(le_2.transform(all_data["weapsubtype1_txt"]))) 

le_3 = preprocessing.LabelEncoder()
le_3.fit(all_data["target1"])
list(le_3.classes_)
all_data["target1"]= pd.Series(list(le_3.transform(all_data["target1"]))) 

le_4 = preprocessing.LabelEncoder()
le_4.fit(all_data["weaptype1_txt"])
list(le_4.classes_)
all_data["weaptype1_txt"]= pd.Series(list(le_4.transform(all_data["weaptype1_txt"]))) 

le_5 = preprocessing.LabelEncoder()
le_5.fit(all_data["targsubtype1_txt"])
list(le_5.classes_)
all_data["targsubtype1_txt"]= pd.Series(list(le_5.transform(all_data["targsubtype1_txt"]))) 

le_6 = preprocessing.LabelEncoder()
le_6.fit(all_data["natlty1_txt"])
list(le_6.classes_)
all_data["natlty1_txt"]= pd.Series(list(le_6.transform(all_data["natlty1_txt"]))) 
# Scaling all the values so that they are in the same scale

from sklearn.preprocessing import StandardScaler
sc=StandardScaler()
all_data=sc.fit_transform(all_data)

# As scaling turns all_data to numpy, we convert it back to pandas for later use.

all_data = pd.DataFrame(all_data)
#creating train and test datasets

train = all_data[:ntrain]
test = all_data[ntrain:]

print(train.shape)
print(test.shape)
"""
## 4. 
## Modeling

### Preparation of data
Now that our dataset is ready for modeling, we must prepare it from training, testing and prediction. One of the vital steps here is to reduce the number of features. I will do this using XGBoost's inbuilt feature importance functionality.
"""
from xgboost import XGBClassifier
from xgboost import plot_importance
model = XGBClassifier()
model.fit(train, y_train)
# Sort feature importances from GBC model trained earlier
indices = np.argsort(model.feature_importances_)[::-1]
indices = indices[:75]

# Visualise these with a barplot
plt.subplots(figsize=(20, 15))
g = sns.barplot(y=cols[indices], x = model.feature_importances_[indices], orient='h', palette = colors)
g.set_xlabel("Relative importance",fontsize=12)
g.set_ylabel("Features",fontsize=12)
g.tick_params(labelsize=9)
g.set_title("XGB feature importance");
# Storing and copying train and test values for safety and maybe later use

train_2 = train.copy()
test_2 = test.copy()
from sklearn.feature_selection import SelectFromModel

# Allow the feature importances attribute to select the most important features
xgb_feat_red = SelectFromModel(model, prefit = True)

# Reduce estimation, validation and test datasets
train = xgb_feat_red.transform(train)
test = xgb_feat_red.transform(test)


print("Results of 'feature_importances_':")
print('X_train: ', train.shape, '\nX_test: ', test.shape)
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

# Dividing train dataset and y_train dataset to X_train, X_test, Y_train, Y_test respectively for training the model.

X_train, X_test, Y_train, Y_test = train_test_split(train, y_train, test_size=0.3, random_state=42, stratify=y_train)
print('X_train: ', X_train.shape, '\nX_test: ', X_test.shape, '\nY_train: ', Y_train.shape, '\nY_test: ', Y_test.shape)
"""
### Training

#### We are finally ready to train our models. For this analysis I am using 5 different algorithms:

1. "K-Nearest Neighbors"
2. "Linear SVM"
3. "Naive Bayes"
4. "Decision Tree"
5. "Neural Net"

#### The method of measuring accuracy was chosen to be "Accuracy"
"""
# Importing Statistical packages!
# Importing Metrics packages!

from sklearn.model_selection import KFold
from sklearn.model_selection import cross_val_score, train_test_split
from sklearn.model_selection import GridSearchCV
from sklearn.model_selection import ShuffleSplit
from sklearn.metrics import confusion_matrix
from sklearn.metrics import make_scorer
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import RobustScaler

print('Validation and metric libraries imported')
# Importing Algorithm packages!

from sklearn.neural_network import MLPClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.svm import SVC
from sklearn.tree import DecisionTreeClassifier
from sklearn.naive_bayes import GaussianNB

print('Classification Models imported')
names = ["K-Nearest Neighbors", "Linear SVM",  "Naive Bayes", "Decision Tree", "Neural Net"]

models= [ KNeighborsClassifier(3), SVC(kernel="linear", C=0.025), GaussianNB(), DecisionTreeClassifier(max_depth=5), MLPClassifier(alpha=1)]
# First I will use ShuffleSplit as a way of randomising the cross validation samples.
shuff = ShuffleSplit(n_splits=5, test_size=0.2, random_state=42)

#create table to compare MLA metrics
columns = ['Name', 'Parameters', 'Train Accuracy Mean', 'Test Accuracy']
before_model_compare = pd.DataFrame(columns = columns)

#index through models and save performance to table
row_index = 0
for alg in models:

    #set name and parameters
    model_name = alg.__class__.__name__
    before_model_compare.loc[row_index, 'Name'] = model_name
    before_model_compare.loc[row_index, 'Parameters'] = str(alg.get_params())
    
    alg.fit(X_train, Y_train)
    
    #score model with cross validation: http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.cross_validate.html#sklearn.model_selection.cross_validate
    training_results = cross_val_score(alg, X_train, Y_train, cv = shuff, scoring= 'accuracy').mean()
    test_results = accuracy_score(Y_test, alg.predict(X_test)).mean()
    
    before_model_compare.loc[row_index, 'Train Accuracy Mean'] = (training_results)*100
    before_model_compare.loc[row_index, 'Test Accuracy'] = (test_results)*100
    
    row_index+=1
    print(row_index, alg.__class__.__name__, 'trained...')

decimals = 3
before_model_compare['Train Accuracy Mean'] = before_model_compare['Train Accuracy Mean'].apply(lambda x: round(x, decimals))
before_model_compare['Test Accuracy'] = before_model_compare['Test Accuracy'].apply(lambda x: round(x, decimals))
before_model_compare
"""
We can see that each of the models performs with varying ability, with DecisionTreeClassifier having the best accuracy score on the training dataset and accuracy on the validation dataset.

### Optimisation

As you can see from the above table, the accuracy for these models is not quite as good as it could be.
This is because we use the default configuration of parameters for each of the algorithms.
So now, we will use GridSearchCV to find the best combinations of parameters to produce the highest scoring models.

"""
models= [ KNeighborsClassifier(), SVC(), GaussianNB(), DecisionTreeClassifier(), MLPClassifier()]

KNN_param_grid = {'n_neighbors': [1,2,3]}
SVC_param_grid = {'C': [0.025, 0.030, 0.020]}
G_param_grid = {'priors': [None, None]}
DTC_param_grid = {'max_depth': range(5, 30, 2)}
MLP_param_grid = {'alpha': [1, 2, 3]}

params_grid = [KNN_param_grid, SVC_param_grid, G_param_grid, DTC_param_grid, MLP_param_grid]
#create table to compare MLA metrics
columns = ['Name', 'Parameters', 'Train Accuracy Mean', 'Test Accuracy']
after_model_compare = pd.DataFrame(columns = columns)

row_index = 0
for alg in models:
    
    gs_alg = GridSearchCV(alg, param_grid = params_grid[0], cv = shuff, scoring= 'accuracy', n_jobs=-1)
    
    params_grid.pop(0)
    #set name and parameters
    model_name = alg.__class__.__name__
    after_model_compare.loc[row_index, 'Name'] = model_name
    
    gs_alg.fit(X_train, Y_train)
    gs_best = gs_alg.best_estimator_
    after_model_compare.loc[row_index, 'Parameters'] = str(gs_alg.best_params_)
    
    #score model with cross validation: http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.cross_validate.html#sklearn.model_selection.cross_validate
    
    after_training_results = gs_alg.best_score_
    after_test_results = accuracy_score(Y_test, gs_alg.predict(X_test)).mean()
    
    after_model_compare.loc[row_index, 'Train Accuracy Mean'] = (after_training_results)*100
    after_model_compare.loc[row_index, 'Test Accuracy'] = (after_test_results)*100
    
    row_index+=1
    print(row_index, alg.__class__.__name__, 'trained...')

decimals = 3
after_model_compare['Train Accuracy Mean'] = after_model_compare['Train Accuracy Mean'].apply(lambda x: round(x, decimals))
after_model_compare['Test Accuracy'] = after_model_compare['Test Accuracy'].apply(lambda x: round(x, decimals))
after_model_compare


"""
Overall we can see that the training and test scores for each of the models have increased, which is what we want.

#### Now we have a set of highly tuned algorithms to use for "STACKING".

### Stacking
Now that we have a set of highly tuned algorithms, a rather famous and successful technique to further improve the accuracy of these models, is to use Stacking.


#### The steps for this technique are shown below:

1. Create a set of algorithms ready for stacking - We've done this...
2. Split the original training data into a training and validation sample - We've done this too...
3. Train the algorithms on the training sample - Also done this...
4. For each algorithm, apply the trained models to the validation dataset and create a set of predictions, 1 column for each model, as a new table. Call this the new training dataset.
5. Also apply the trained algorithm to the test dataset and create a final set of predictions, 1 column for each model, as a new table. Call this new test dataset.
6. For the new training dataset, we have labeled outputs, in the form of Y_test. Now we must train another model on these two feature sets: new training dataset and Y_test.
7. Use this newly trained model to predict values for new test dataset.

"""
"""
Before I start with the stacking, I need to decide which algorithms to use as my base estimators, and which to use as the meta-model.

Since KNeighborsClassifier performed the best after optimisation, I chose this to be the meta-model. All other models will be used as base estimators.

So now, I will cycle through each optimised estimator, train them on the training dataset, apply to them the validation and test datasets, then finally outputting the predictions for validation and test into two new datasets: stacked_validation_train and stacked_test_train.
"""
models= [ KNeighborsClassifier(), SVC(), GaussianNB(), DecisionTreeClassifier(), MLPClassifier()]

KNN_param_grid = {'n_neighbors': [1,2,3]}
SVC_param_grid = {'C': [0.025, 0.030, 0.020]}
G_param_grid = {'priors': [None, None]}
DTC_param_grid = {'max_depth': range(5, 30, 2)}
MLP_param_grid = {'alpha': [1, 2, 3]}

params_grid = [KNN_param_grid, SVC_param_grid, G_param_grid, DTC_param_grid, MLP_param_grid]

stacked_validation_train = pd.DataFrame()
stacked_test_train = pd.DataFrame()

row_index=0

for alg in models:
    
    gs_alg = GridSearchCV(alg, param_grid = params_grid[0], cv = shuff, scoring = 'accuracy', n_jobs=-1)
    params_grid.pop(0)
    
    gs_alg.fit(X_train, Y_train)
    gs_best = gs_alg.best_estimator_
    stacked_validation_train.insert(loc = row_index, column = names[0], value = gs_best.predict(X_test))
    print(row_index+1, alg.__class__.__name__, 'predictions added to stacking validation dataset...')
    
    stacked_test_train.insert(loc = row_index, column = names[0], value = gs_best.predict(test))
    print(row_index+1, alg.__class__.__name__, 'predictions added to stacking test dataset...')
    print("-"*50)
    names.pop(0)
    
    row_index+=1
    
print('Done')
stacked_validation_train.head(5)
stacked_test_train.head(5)
"""
### I will use these two datasets to train and produce predictions for the meta-model,K-Nearest Neighbors
"""
# First drop the Lasso results from the table, as we will be using Lasso as the meta-model
drop = ['K-Nearest Neighbors']
stacked_validation_train.drop(drop, axis=1, inplace=True)
stacked_test_train.drop(drop, axis=1, inplace=True)

# Now fit the meta model and generate predictions
meta_model = make_pipeline(RobustScaler(),KNeighborsClassifier(1))
meta_model.fit(stacked_validation_train, Y_test)

meta_model_pred = meta_model.predict(stacked_test_train)
print("Meta-model trained and applied!...")
"""
### Ensemble

As our models having varying degrees of accuracy we also use Ensemble methods.

They work on the idea that many weak learners, can produce a strong learner.


Therefore, using the meta-model that I will create, I will also combine this with the results of the individual optimised models to create an ensemble.


In order to create this ensemble, I must collect the final predictions of each of the optimised models. I will do this now.
"""
models= [ KNeighborsClassifier(), SVC(), GaussianNB(), DecisionTreeClassifier(), MLPClassifier()]
names = ["K-Nearest Neighbors", "Linear SVM",  "Naive Bayes", "Decision Tree", "Neural Net"]
KNN_param_grid = {'n_neighbors': [1,2,3]}
SVC_param_grid = {'C': [0.025, 0.030, 0.020]}
G_param_grid = {'priors': [None, None]}
DTC_param_grid = {'max_depth': range(5, 30, 2)}
MLP_param_grid = {'alpha': [1, 2, 3]}

params_grid = [KNN_param_grid, SVC_param_grid, G_param_grid, DTC_param_grid, MLP_param_grid]


final_predictions = pd.DataFrame()

row_index=0

for alg in models:
    
    gs_alg = GridSearchCV(alg, param_grid = params_grid[0], cv = shuff, scoring = 'accuracy', n_jobs=-1)
    params_grid.pop(0)
    
    gs_alg.fit(stacked_validation_train, Y_test)
    gs_best = gs_alg.best_estimator_
    final_predictions.insert(loc = row_index, column = names[0], value = gs_best.predict(stacked_test_train))
    print(row_index+1, alg.__class__.__name__, 'final results predicted added to table...')
    names.pop(0)
    
    row_index+=1

print("-"*50)
print("Done")
    
final_predictions.head()
"""
1. As you can see, each of the models produces results that vary quite widely. This is the beauty of using a combination of many different models.

2. Some models will be much better at catching certain signals in the data, whereas others may perform better in other situations.

3. By creating an ensemble of all of these results, it helps to create a more generalised model that is resistant to noise.

## 5.

### Finish

### Now, I will finish by creating an ensemble of the meta-model and optimised models.
"""
ensemble = meta_model_pred*(2.5/10)+ final_predictions["K-Nearest Neighbors"]*(1.5/10)+final_predictions["Linear SVM"]*(1.5/10)+ final_predictions["Naive Bayes"]*(1.5/10)+ final_predictions["Decision Tree"]*(1.5/10)+ final_predictions["Neural Net"]*(1.5/10)

#Final predicted values for the attacks that have been not attrbuted any attacks.

unknown_df["gname"]= ensemble.values

unknown_df.head(5)