"""
# Problem Statement
Crime is increasing considerably day by day. Crime is among the main issues which is growing continuously in intensity and complexity. Crime patterns are changing constantly because of which it is difficult to explain behaviours in crime patterns.

So it becomes a difficult challenge for crime analysts to analyse such voluminous crime data without any computational support. A powerful system for predicting crimes is required in place of traditional crime analysis because traditional methods cannot be applied when crime data is high dimensional and complex queries are to be processed. Therefore a crime prediction and analysis tool were needed for identifying crime patterns effectively.
"""
"""
# Data Exploration
In this section, We will load data and see it's content.

## 1. Loading the data files
"""
# import pandas
import pandas as pd

# load train and test as dataframes
train_df = pd.read_csv("../input/crime-prediction/train.csv",parse_dates=['Dates'],error_bad_lines=False)
test_df = pd.read_csv("../input/crime-prediction/test.csv",parse_dates=['Dates'],error_bad_lines=False)
train_df.isnull().any()

"""
## 2. Showing data
"""
# import display function from ipython
from IPython.display import display, HTML

# display the first rows of each dataset
display(train_df.head())
print("train shape: {}".format(train_df.shape))
display(test_df.head())
print("test shape: {}".format(test_df.shape))
"""
* As shown above, The test dataset doesn't contain 3 columns {Category, Descript, Resolution} as the category column is the target column to found in the test set.
* The test set will be used for testing only at the end of project
* Training data will be divided into training and testing -Validation- sets to train the models later.
"""
"""
----
# Data visualizing and preprocessing
In this section, We will visualize data and remove unnecessary data.
"""
"""
## 1. Removing redundant features
As shown above, there exist 2 columns -features- that are considered as redundancy. `Descript` and `Resolution` are these 2 columns as they don't exist in the testing values and also not a label required from the models, so they should be removed.
"""
train_df = train_df.drop(columns=['Descript', 'Resolution'])
train_df.head()
"""
## 2. Removing redundant rows
Now, lets visualize location information
"""
# lets see the statistics summary of locations
lons = train_df['X'] # longitudes 
lats = train_df['Y'] # latitudes

print("Longitudes summary:")
print(lons.describe())
print("\nLatitudes summary:")
print(lats.describe())
"""
-----------------------
**Observation:**
* longitudes are between [-122.52, -120.5], each value different slightly from others
* latitudes are between [37.708, 90]
* here as shown there exist some bad values -i.e. close to 90-, the reasons that this is bad that 
    * first, san fransisco latitudes are between [37.707, 37.83269] , reference: [google maps](https://www.google.com.eg/maps/place/San+Francisco,+CA,+USA/@37.7407396,-122.4303937,12z/data=!4m5!3m4!1s0x80859a6d00690021:0x4a501367f076adff!8m2!3d37.7749295!4d-122.4194155)
    * second as shown in the statistics that the most values are close to 37.7
    * Also in longitudes, san fransisco longitudes are between [-122.517652, -122.3275], from google maps

Now, to demonstrate the locations, let's plot them using scatter plot
"""
# eliminate rows with latitudes out of San Francisco range
train_df = train_df.drop(train_df[(train_df['Y'] > 37.84) | (train_df['Y'] < 37.7)].index)
# eliminate rows with longitudes out of San Francisco range
train_df = train_df.drop(train_df[((train_df['X'] > -122.32) | (train_df['X'] < -122.52))].index)
train_df.describe()

#from shapely.geometry import  Point
#import geopandas as gpd
import matplotlib.pyplot as plt
import numpy as np
#from sklearn.impute import SimpleImputer
#from sklearn.preprocessing import LabelEncoder
#from sklearn.model_selection import train_test_split
import seaborn as sns
from matplotlib import cm
#import urllib.request
#import shutil
#import zipfile
#import os
#import re
#import contextily as ctx
#import geoplot as gplt
#import lightgbm as lgb
#import eli5
#from eli5.sklearn import PermutationImportance
#from lightgbm import LGBMClassifier
from matplotlib import pyplot as plt
#from pdpbox import pdp, get_dataset, info_plots
#import shap
"""
## 3. Visualize according to locations
"""
new_lons = train_df['X'] # longitudes 
new_lats = train_df['Y'] # latitudes

# scatter plot for lons vs lats
plt.scatter(new_lons, new_lats)
plt.xlabel('lons')
plt.ylabel('lats')
plt.show()

# histogram plot for lons and lats
plt.hist(new_lons)
plt.xlabel('lons')
plt.ylabel('ocurrance number')
plt.show()
plt.hist(new_lats)
plt.xlabel('lats')
plt.ylabel('ocurrance number')
plt.show()
"""
**Observation:**  the most crimes are in the location of longitude = [-122.44, -122.40] and latitude = [37.76, 37.80]
"""
"""
### Dates & Day of the week  
These variables are distributed uniformly between 1/1/2003 to 5/13/2015 (and Monday to Sunday) and split between the training and the testing dataset as mentioned before. We did not notice any anomalies on these variables.  
The median frequency of incidents is 389 per day with a standard deviation of 48.51.
"""
#import seaborn as sns
col = sns.color_palette()

train_df['Date'] = train_df.Dates.dt.date
train_df['Hour'] = train_df.Dates.dt.hour

plt.figure(figsize=(10, 6))
data = train_df.groupby('Date').count().iloc[:, 0]
sns.kdeplot(data=data, shade=True)
plt.axvline(x=data.median(), ymax=0.95, linestyle='--', color=col[1])
plt.annotate(
    'Median: ' + str(data.median()),
    xy=(data.median(), 0.004),
    xytext=(200, 0.005),
    arrowprops=dict(arrowstyle='->', color=col[1], shrinkB=10))
plt.title(
    'Distribution of number of incidents per day', fontdict={'fontsize': 16})
plt.xlabel('Incidents')
plt.ylabel('Density')
plt.legend().remove()
plt.show()
"""
Also, there is no significant deviation of incidents frequency throughout the week. Thus we do not expect this variable to play a significant role in the prediction.
"""
data = train_df.groupby('DayOfWeek').count().iloc[:, 0]
data = data.reindex([
    'Monday', 'Tuesday', 'Wednesday', 'Thursday', 'Friday', 'Saturday',
    'Sunday'
])

plt.figure(figsize=(10, 5))
with sns.axes_style("whitegrid"):
    ax = sns.barplot(
        data.index, (data.values / data.values.sum()) * 100,
        orient='v',
        palette=cm.ScalarMappable(cmap='Reds').to_rgba(data.values))

plt.title('Incidents per Weekday', fontdict={'fontsize': 16})
plt.xlabel('Weekday')
plt.ylabel('Incidents (%)')

plt.show()
"""
### Category  
There are 39 discrete categories that the police department file the incidents with the most common being Larceny/Theft (19.91%), Non/Criminal (10.50%), and Assault(8.77%).
"""
data = train_df.groupby('Category').count().iloc[:, 0].sort_values(
    ascending=False)
data = data.reindex(np.append(np.delete(data.index, 1), 'OTHER OFFENSES'))

plt.figure(figsize=(10, 10))
with sns.axes_style("whitegrid"):
    ax = sns.barplot(
        (data.values / data.values.sum()) * 100,
        data.index,
        orient='h',
        palette="Reds_r")

plt.title('Incidents per Crime Category', fontdict={'fontsize': 16})
plt.xlabel('Incidents (%)')

plt.show()
"""
---
Finding number of occurances of for each category in data
"""
from collections import Counter

def printCategoriesOccurrence():
    
    categories = train_df['Category']
    # count the number of occurances for each category
    occurances = Counter(categories)
    sorted_occ = sorted(occurances.items(), key=lambda pair: pair[1], reverse=True)
    for key, value in sorted_occ:
        print(key, value)
    return sorted_occ
        
sorted_occ = printCategoriesOccurrence()
"""
**Observation:**
The most committed crime in San Francisco is the LARCENY/THEFT. TREA is the least.
"""
"""
## 5. Data preprocessing
"""
"""
**Enhanceing data imbalancing:**
Now as shown above, The data is imbalanced and some features has very low number of contributions in data existance. And to enhance imbalanced data we can replicate the data with low contributions and give them more weights in training.

Here we will replicate the data that has less than 1000 occurances to be 1000. Later we will give them more weights while training.
"""
import math
# if category size < 1000, duplicate it to be = 1000
for key,value in sorted_occ:
    if value<1000:
        
        temp = train_df[train_df['Category'] == key]
        train_df = train_df.append([temp]*int(math.ceil((1000-value)/float(value))), ignore_index=True)

sorted_occ = printCategoriesOccurrence()
"""
### Data Encoding
"""
# spliting train data into target and other features
target = train_df['Category']
data = train_df.drop(columns=['Category'])
"""
4 features from the data given are not numbers, so we need to convert them into numbers in order to be able to train the models on them. These features are:
* Dates
* DayOfWeek
* PdDistrict
* Address

We can use label Encoding and 1-hot encoding. 1-hot encoding may produce very high numbers of dimensions due to the many data labeles in each feature, but it is better due to the problem with label encoding is that it assumes higher the categorical value, better the category which produce more errors.

So let's use one-hot encoding with the features with little unique values and use the label encoding with the features with very high unique values

"""
features = ['Dates', 'DayOfWeek', 'PdDistrict', 'Address']
for feature in features:
    print("feature: {}    unique_size: {}".format(feature ,len(data[feature].unique())))
"""
As shown above, `DayOfWeek` and `PdDistrict` can one-hot encoded. But `Address`, `Dates` should be encoded using label encoding.

For `Dates`, lets neglect the miniutes and seconds of each date as this will not affect greatly on our predictions but on the contrary it may produce good environment for overfitting, so let's convert `Dates` by our hand to integers then convert `Address` using `cat.codes` tool.

### 5.1 Label Encoding
"""
data.head()
print(str(data['Dates'][0])[:-6])
# convert given list of dates it will trim seconds, minutes and return result
def trimMinAndSecFromDates(dates):
    result = []
    for date in dates:
        result.append(str(date)[:-6])
    return result

# trim minutes and seconds from dates
data['Dates'] = trimMinAndSecFromDates(data['Dates'])

# encode Dates using label encoding
data['Dates'] = data['Dates'].astype('category')
data['Dates_int'] = data['Dates'].cat.codes

# encode Address using label encoding
data['Address'] = data['Address'].astype('category')
data['Address_int'] = data['Address'].cat.codes

data.head()
"""
Now let's drop the `Dates` and `Address` cols as they are now useless.
"""
data.drop(columns=['Dates', 'Address'], inplace=True)
data.head()
"""
### 5.2 One-Hot Encoding
Now let's one hot encode the `DayOfWeek` and `PdDistrict` using pandas.get_dummies()
"""
# get dummies for each feature
DayOfWeek_dummies = pd.get_dummies(data['DayOfWeek'])
PdDistrict_dummies = pd.get_dummies(data['PdDistrict'])

# join dummies to the original dataframe
data = data.join(DayOfWeek_dummies)
data = data.join(PdDistrict_dummies)

data.head()
"""
Now let's drop the `DayOfWeek` and `PdDistrict` cols as they are now useless.
"""

data.drop(columns=['DayOfWeek', 'PdDistrict'], inplace=True)
print("data size =",len(data))
data.head()

data.drop(columns=['Date', 'Hour'], inplace=True)

import numpy as np
data[~data.isin([np.nan, np.inf, -np.inf]).any(1)].astype(np.float64)
data.replace([np.inf, -np.inf], np.nan)
data.isnull().any()
data.dropna()
data.columns
"""
**Observation:** Now the training data are ready to be used for our models with 21 dimensions and 885669 samples to be traind on
"""
data.shape
"""
## 5. Data splitting
split training data into train set with size 80% and test set with size 20% - i.e. validation set.
"""
from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test =  train_test_split(data, target, test_size=0.2, random_state=0, stratify=target)
print("train size: {}, test size: {}".format(X_train.shape[0], X_test.shape[0]))

target
"""

--------------
# Models Implementation
"""
"""
We are going to implement various models and train them on our data and comapre their performance later using using mult-class logarithmec loss.
"""
"""
## 1. Implementing Models
In this section, we will train various model on data.
"""
data.isnull().any()

"""
### Training pipline
Now, we will train various learners, so we will make a pipline -i.e function- to call it in training the models instead of repeating it with each model. 

The function returns a result dict which contains the train time, fbeta score and log loss of all samples in train data.

**Hint:** We shouldn't use accuracy here as the data is imbalanced.
"""
from sklearn.metrics import log_loss, fbeta_score
from time import time

# train function takes learner, the train data and target
def train_test_pipeline(learner, X_train, y_train, X_test, y_test):
    
    results = {}
    
    # training learner
    start = time()
    learner.fit(X_train, y_train)
    end = time()
    results['train_time'] = end - start
    
     # remove missed classes after fitting for logloss
    for category in list(set(target) - set(learner.classes_)):
        X_train = X_train.drop(y_train[y_train == category].index)
        y_train = y_train[y_train != category]
        X_test = X_test.drop(y_test[y_test == category].index)
        y_test = y_test[y_test != category]
    # predict samples in training set
    predictions = learner.predict(X_train)
    predictions_proba = learner.predict_proba(X_train)
    
    # calculate fbeta and log loss
    results['fscore'] = fbeta_score(y_train, predictions, beta=.5, average='micro')
    results['logloss'] = log_loss(y_train, predictions_proba)
    
    # predict testing samples and time of prediction
    start = time()
    predictions = learner.predict(X_test)
    predictions_proba = learner.predict_proba(X_test)
    end = time()
    results['test_time'] = end - start
    
    # calculate fbeta and log loss for testing set
    results['fscore_test'] = fbeta_score(y_test, predictions, beta=.5, average='micro')
    results['logloss_test'] = log_loss(y_test, predictions_proba)
    
    
    print ("{} trained".format(learner.__class__.__name__))
    
    return results

# do train_test_pipeline then visualize resutls for given models on first n samples and test on first m
# returns predictions proba for last model in given list - this will be used for 1 model only -
def train_test_models(models, names=None, n=len(y_train), m=len(y_test)):
    results = {}
    i = 0
    for model in models:
        if not names:
            model_name = model.__class__.__name__
        else:
            model_name = names[i]
            i += 1
        results[model_name] = train_test_pipeline(model, X_train[:n], y_train[:n], X_test[:m], y_test[:m])

    # print results
    for model in results:
        model_res = results[model]
        print ("model: {}".format(model))
        print ("fscore:\t\t{}\nlogloss:\t{}\ntrain time:\t{}".format(model_res['fscore'], model_res['logloss'], model_res['train_time']))
        print ("fscore_test:\t\t{}\nlogloss_test:\t{}\ntest time:\t{}".format(model_res['fscore_test'], model_res['logloss_test'], model_res['test_time']))

    # visualize the results    
    visualize(results, random_results)
"""
### Visualization
visualize training and testing results
"""
import matplotlib.pyplot as plt
def visualize(results, random_results):
    bar_width = 0.3
    fig, ax = plt.subplots(6,1,figsize = (12,32))
    for j, metric in enumerate(['train_time', 'fscore', 'logloss', 'test_time', 'fscore_test', 'logloss_test']):
        ax[j].set_xlabel("Learners")
        ax[j].set_ylabel(metric)
        ax[j].set_title(metric)
        for k, learner in enumerate(results.keys()):
            ax[j].bar(learner, results[learner][metric], width=bar_width)
    
    # add horizontal line for random model results
    ax[0].axhline(y=random_results['train_time'], linestyle='dashed')
    ax[1].axhline(y=random_results['fscore'], linestyle='dashed')
    ax[2].axhline(y=random_results['logloss'], linestyle='dashed')  
    ax[3].axhline(y=random_results['test_time'], linestyle='dashed')
    ax[4].axhline(y=random_results['fscore_test'], linestyle='dashed')
    ax[5].axhline(y=random_results['logloss_test'], linestyle='dashed') 
"""
### 1.1 Naive Random Predictor
Random predictor which always predict the category randomly.
"""
import random
class random_model:
 
    def __init__(self, categories):
        self.categories = categories
        self.classes_ = categories

    # always return a random value from categories
    def __getRandomValue(self): return random.choice(self.categories) 
    
    # no need for fit here
    def fit(self, X_train, y_train): pass
    
    def predict(self, X):
        result = [[] for i in range(len(X))]
        for j in range(len(X)):
            result[j] = self.__getRandomValue()
        return result
        
    def predict_proba(self, X):
        result = [[] for i in range(len(X))]
        for j in range(len(X)):
            row = [0.0]*len(self.categories)
            prediction = self.__getRandomValue()
            for i in range(len(self.categories)): 
                if(self.categories[i] == prediction):
                    row[i] = 1.0
                    break
            result[j] = row
        return result
"""
### 1.2 other models
In this section, We will train other models using training pipline that we implemented previously. The models to use are:
* KNNeighbors
* DecisionTree
* ExtraTrees
* Neural network MLP
* Support vector machine
* xgboost
"""
"""
#### 1.2.1 Initializing models
Here, we enhance imbalanced data more by putting the `class_weight` parameter = `balanced` which mean that the weights will be automatically adjusted inversely proportional to class frequencies in the input data as `n_samples / (n_classes * np.bincount(y))`
"""
from sklearn.neighbors import KNeighborsClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import ExtraTreesClassifier
from sklearn.neural_network import MLPClassifier
from sklearn.svm import SVC 
from xgboost import XGBClassifier

# initializing models
model_KNN = KNeighborsClassifier(n_jobs=-1, weights='distance')
model_tree = DecisionTreeClassifier(class_weight='balanced')
model_extraTrees = ExtraTreesClassifier(n_jobs=-1, class_weight='balanced')
model_NN = MLPClassifier(learning_rate='invscaling', shuffle=True)
model_SVC = SVC(probability=True, class_weight='balanced') # One-to-One
model_XGB = XGBClassifier(one_drop=1)
"""
#### 1.2.1 Initializing models
Here, we enhance imbalanced data more by putting the `class_weight` parameter = `balanced` which mean that the weights will be automatically adjusted inversely proportional to class frequencies in the input data as `n_samples / (n_classes * np.bincount(y))`
"""
model_random = random_model(categories=target.unique())
random_results = train_test_pipeline(model_random, X_train, y_train, X_test, y_test)
"""
#### 1.2.3 train and visualize other models with small number of samples
"""
models = [model_KNN, model_tree, model_extraTrees, model_NN, model_SVC, model_XGB]
train_test_models(models, n=10000, m=2000)
"""
**Observation:**

For training:
* The slowest model is SVC then XGBClassifier
* All models do better than random predictor

For test data: 
* The best model in f1 score and logloss is XGBClassifier then SVC
* The slowest model in testing time is SVC the XGBClassifier
* All models do better thean random predictor
    
As shown, It seems that the best model to use is XGBClassifier due to it's scores and it has a suitable time in training and testing. SVC also did well but it needs huge amount of time in processing data in training and testing.
"""
"""
The XGBClassifier gives bad results than expected, This is due to that at first time we trained on very small numbers of training data, Now, let's try other classifiers on all data except SVC and ExtraTree due to that they need huge amount of available resources.
"""
# training other models on all data
models = [model_KNN, model_tree, model_NN]
train_test_models(models,n=20000,m=2000)
model_NN_tuned = MLPClassifier(learning_rate='adaptive', shuffle=True, epsilon=1e-8, activation='relu',
                               hidden_layer_sizes=100, solver='adam', verbose=True)

models = [model_NN_tuned]
train_test_models(models,n=20000,m=4000)
X_train.isnull().any()
y_train.isnull().any()

model_KNN = KNeighborsClassifier(n_jobs=-1, weights='distance')
X=X_train.iloc[0:30000]
y=y_train.iloc[0:30000]
model_KNN.fit(X_train,y_train)
print("knn training accuracy",model_KNN.score(X,y))
print(model_KNN.predict(X))


model_tree.fit(X,y)
print("training accuracy",model_tree.score(X,y))

model_extraTrees.fit(X,y)
print("training accuracy EXTRATREES",model_extraTrees.score(X,y))

'''
model_SVC.fit(X,y)
print("training accuracy",model_SVC.score(X,y))

model_XGB.fit(X,y)
print("training accuracy XGB",model_XGB.score(X,y))
'''
'''
model_NN_tuned.fit(X,y)
print("knn training accuracy",model_NN_tuned.score(X,y))
'''

