"""
# Decision Tree example
"""
# This Python 3 environment comes with many helpful analytics libraries installed
# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python
# For example, here's several helpful packages to load

import numpy as np # linear algebra
import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)
import seaborn as sns
import matplotlib.pyplot as plt

# Input data files are available in the read-only "../input/" directory
# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory

import os
for dirname, _, filenames in os.walk('/kaggle/input'):
    for filename in filenames:
        print(os.path.join(dirname, filename))

# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using "Save & Run All" 
# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session

"""
# Data Preprocessing
"""
"""

*Assigning column names*
"""
ecoli_df =  pd.read_csv("../input/ecoli-data-set/ecoli.csv",header=None,sep="\s+")
col_names = ["squence_name","mcg","gvh","lip","chg","aac","alm1","alm2","site"]
ecoli_df.columns = col_names
ecoli_df.head()
"""
Inspect the data.
Check whether there is any obvious missing values.
"""
ecoli_df.isnull().sum()
"""
# Clean the data
"""
"""
Show the two non-numerical columns.
"""
ecoli_df.loc[:,ecoli_df.dtypes == "object"].columns.tolist()
"""
Number of unique value in squence name is equal to the number of rows of the data frame. 

Each of the squence_name appear only once, it is similar to a unique identifier of each row. 

Therefore, we *can drop* this column without affecting our further analysis and model building.


"""
print(len(ecoli_df["squence_name"].unique()) == ecoli_df.shape[0])
print(ecoli_df['lip'].value_counts())
print(ecoli_df['chg'].value_counts())
"""
The results above show that almost all values in chg are 335.

And there are only 10 rows with 1. 

Dropping classes having less than 10 instances, there is only 3 instances in lip is 1, so I drop lip and chg.
"""
"""
A function to clean the data, the first thing is to drop the squence_name column. The second is to drop the classes that have less than 10 instances.
"""
def cleaning_object(ecoli_df,cols_to_drop,class_col):
    #ob1 suppose to be squence_name
    ecoli_df = ecoli_df.drop(cols_to_drop,axis=1)
        
    #drop classes with less than 10 instances
    uni_class = ecoli_df[class_col].unique().tolist()
    for class_label in uni_class:
        num_rows = sum(ecoli_df[class_col] == class_label)
        if num_rows < 10:
            class_todrop = ecoli_df[ecoli_df[class_col] == class_label].index
            ecoli_df.drop(class_todrop,inplace = True)
    return ecoli_df
cleaned_ecoli_df = cleaning_object(ecoli_df,["squence_name",'lip','chg'],"site")
cleaned_ecoli_df.head()
"""
We can see the squence_name column is dropped.
And below table shows that there is no more classes having less than 10 instances.
"""
cleaned_ecoli_df["site"].value_counts()
"""
# Data Visualization
"""
"""
lip and chg are the binary columns, we will drop them from the visualization. Meanwhile, lip has only either 0.48 or 1.
"""
sns.pairplot(cleaned_ecoli_df[['mcg','gvh','aac','alm1','alm2','site']],hue='site')
"""
A linear trend between alm1 and alm2. As the graph shown, for alm1 and alm2 with values lowers than 0.5, the class is mainly cp, for the values higher than 0.5, the class is mainly imU (green). And there are some class pp below the straight line. And the other classes are scatter around. 

For mch with others variabels, we can see there are some clusters. 
"""
"""
# Model building
"""
from sklearn.tree import DecisionTreeClassifier
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import GridSearchCV
from sklearn.metrics import f1_score, make_scorer
"""
Grid search for the best combination of parameters.
"""
param_grid = {'max_depth': np.arange(4, 7),"criterion":['gini','entropy'],
              "max_features":np.arange(2,5),"min_samples_split": np.arange(3,6)}
#f1 = make_scorer(f1_score , average='samples')
clf = GridSearchCV(DecisionTreeClassifier(random_state=22), param_grid, cv=3,scoring ="f1_weighted",
                  return_train_score=True)
clf_scaled = GridSearchCV(DecisionTreeClassifier(random_state=22), param_grid, cv=3,scoring ="f1_weighted",
                  return_train_score=True)
"""
Scale the data by standarize them. (PS Scaling doesn't affect the performace of decision tree)
To show you scaling doesn't affect the structure of decision tree, I built two decision trees, one with scaled data, another withou.
"""
scaler = StandardScaler()
#Scale the cleaned data
scaled_ecoli_df = scaler.fit_transform(cleaned_ecoli_df[['mcg','gvh','aac','alm1','alm2']])
#put the scaled data back into data frame with column names
scaled_ecoli_df = pd.DataFrame(scaled_ecoli_df,columns = ['mcg','gvh','aac','alm1','alm2'])
#Merge that with the class column.
scaled_ecoli_df = scaled_ecoli_df.merge(cleaned_ecoli_df["site"].to_frame(),left_index=True, right_index=True)
"""
Build the model with raw features:
"""
clf.fit(cleaned_ecoli_df[['mcg','gvh','aac','alm1','alm2']],cleaned_ecoli_df["site"])
clf_scaled.fit(scaled_ecoli_df[['mcg','gvh','aac','alm1','alm2']],scaled_ecoli_df["site"])
"""
# Models comparison
"""
"""
Although model without scaling perform better in term of average cross-valdiation f1 scores, it is not necessary always the case. The training results depend on the way the train data is splitted in cross validation. Each time the code run, the data is splitted randomly. 

In fact, decision tree do not require feature scaling to be performed as they are not sensitive to the the variance in the data.

To sum up, the difference between the raw model and scaled model is due to the split of the folds in the cross validartion.
"""
print('The best score for raw data decision tree: ', clf.best_score_)
print('The best score for scaled data decision tree: ',clf_scaled.best_score_)
"""
The best parameters are differnt as shown below.Max features is the only parameter that has the same value across this tow model.
"""
print('The best combination of raw data decision tree: ',clf.best_params_)
print('The best combination of scaled data decision tree: ',clf_scaled.best_params_)
"""
Below will display the confustion matrices
"""
from sklearn.metrics import plot_confusion_matrix
#Confusion matrix for the raw features model
plot_confusion_matrix(clf,cleaned_ecoli_df[['mcg','gvh','aac','alm1','alm2']],
                      cleaned_ecoli_df["site"])
"""
Above confusion matrix shows that most of the predictions are correct. However, the model does not perform well when predicting imU. There are 13 instances of imU were predicted as im in the model without scale the data.
"""
#Confusion matrix for the scaled features model
plot_confusion_matrix(clf_scaled,scaled_ecoli_df[['mcg','gvh','aac','alm1','alm2']],
                      scaled_ecoli_df["site"])
"""
Above confusion matrix shows that most of the predictions are correct. However, the model does not perform well when predicting imU. There are 13 instances of imU were predicted as im in the model without scale the data. And 2 instances and 1 instance are predicted as om and cp respectively while they are imU in fact. 
"""
"""
# Train-test split affects how the decision tree constructed

Both models did not perform well when predicting imU. In fact, their predictions or model itself are quite similar, since scaling would not affect the performance of decision tree. The split of the tree are based on gini index or information gain(entropy), scaling does not affect the those values. As we perfore cross-validation, the data used to build the models are different, therefore, the information gain or gini index from the data are different each time. 

The 'random' split of the data is affecting how the tree split, since a subset of the data might have differnet distribution, information gain or gini index then other subset. 

If the sclaed data and the raw data use the same data to build the model, they should obtain the same split.
"""
"""
The above plots show the actual values of compressive Strength against the predicted values. The closer the data points to the straight line, the more accuracy the prediction is. 

Support vector regressor performs better than the other estimators. 

The predictions of decision tree regression are given by the average of the value of the dependent variables(features) in that leaf node. So we can see the predictions of the regression tree are lying horizontally, which means they made the same predicted values. The instances in the same leaf node will make the same prediction. In other words, the data points with the same predicted value are in the same left node.

SVM Regression tries to fit as many instances as possible on the *street* while limiting margin violations. It can be fitted to non-linear data. As a result, the data points of SVR are much more closer to the actual values. The hyperplane is more likely to wiggle aroudn the data points.
"""