"""
## You're here! 
Welcome to your first competition in the [ITI's AI Pro training program](https://ai.iti.gov.eg/epita/ai-engineer/)! We hope you enjoy and learn as much as we did prepairing this competition.


## Introduction

In the competition, it's required to predict the `Severity` of a car crash given info about the crash, e.g., location.

This is the getting started notebook. Things are kept simple so that it's easier to understand the steps and modify it.

Feel free to `Fork` this notebook and share it with your modifications **OR** use it to create your submissions.

### Prerequisites
You should know how to use python and a little bit of Machine Learning. You can apply the techniques you learned in the training program and submit the new solutions! 

### Checklist
You can participate in this competition the way you perefer. However, I recommend following these steps if this is your first time joining a competition on Kaggle.

* Fork this notebook and run the cells in order.
* Submit this solution.
* Make changes to the data processing step as you see fit.
* Submit the new solutions.

*You can submit up to 5 submissions per day. You can select only one of the submission you make to be considered in the final ranking.*


Don't hesitate to leave a comment or contact me if you have any question!
"""
"""
## Import the libraries

We'll use `pandas` to load and manipulate the data. Other libraries will be imported in the relevant sections.
"""
import pandas as pd
import os
##########################################
import itertools
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from matplotlib.ticker import NullFormatter
import pandas as pd
import numpy as np
import matplotlib.ticker as ticker
from sklearn import preprocessing
%matplotlib inline
from sklearn.model_selection import train_test_split  #>>>>
from sklearn.linear_model import LinearRegression     #>>>>
##########################################
from pandas.plotting import scatter_matrix
import matplotlib.pyplot as plt
import xml.etree.ElementTree as Xet
"""
## Exploratory Data Analysis
In this step, one should load the data and analyze it. However, I'll load the data and do minimal analysis. You are encouraged to do thorough analysis!

Let's load the data using `pandas` and have a look at the generated `DataFrame`.
"""
"""
### 1 - Data understanding
"""
print('---------------File: train.csv-------------------')
dataset_path = '/kaggle/input/car-crashes-severity-prediction/'
df = pd.read_csv(os.path.join(dataset_path, 'train.csv'))
print("#################################################")
print("The shape of the dataset is {}.".format(df.shape))
print("#################################################")
print("Sample of the dataset:\n")
print(df.head())
print("#################################################")
print("Sample of the dataset:\n")
print(df.describe())
print("#################################################")
print("check the null values")
print(df.isnull().sum())
print("#################################################")
print("check the data types and duplicates")
print(df.info())
print("#################################################")
print("check the unique of each coloum")
print("ID=",df.ID.unique())
print("Lat=",df.Lat.unique())
print("Lng=",df.Lng.unique())
print("Bump=",df.Bump.unique())
#print("Distance(mi)=",df[Distance(mi)].unique())
print("Crossing=",df.Crossing.unique())
print("Give_Way=",df.Give_Way.unique())
print("Junction=",df.Junction.unique())
print("No_Exit=",df.No_Exit.unique())
print("Railway=",df.Railway.unique())
print("Roundabout=",df.Roundabout.unique())
print("Stop=",df.Stop.unique())
print("Amenity=",df.Amenity.unique())
print("Side=",df.Side.unique())
print("Severity=",df.Severity.unique())
print("timestamp=",df.timestamp.unique())
print("#################################################")
print('---------------File: weather-sfcsv.csv-------------------')
df_weather = pd.read_csv(os.path.join(dataset_path, 'weather-sfcsv.csv'))
print("#################################################")
print("The shape of the dataset is {}.".format(df_weather.shape))
print("#################################################")
print("Sample of the dataset:\n")
print(df_weather.head())
print("#################################################")
print("Sample of the dataset:\n")
print(df_weather.describe())
print("#################################################")
print("check the null values")
print(df_weather.isnull().sum())
print("#################################################")
print("check the data types and duplicates")
print(df_weather.info())
print("#################################################")
print("check the unique of each coloum")
print("Year=",df_weather.Year.unique())
print("Day=",df_weather.Day.unique())
print("Month=",df_weather.Month.unique())
print("Hour=",df_weather.Hour.unique())
print("Weather_Condition=",df_weather.Weather_Condition.unique())
print("Selected=",df_weather.Selected.unique())
print("#################################################")
print('---------------File: holidays.xml-------------------')
cols = ["date", "description"]
rows = []
# Parsing the XML file
print("#################################################")
xmlparse = Xet.parse('/kaggle/input/car-crashes-severity-prediction/holidays.xml')
root = xmlparse.getroot()
print("#################################################")
for i in root:
    date = i.find("date").text
    description = i.find("description").text 
    rows.append({"date": date,"description": description})
df_holidays = pd.DataFrame(rows, columns=cols)
print(df_holidays)
print("#################################################")
"""
We've got 6407 examples in the dataset with 14 featues, 1 ID, and the `Severity` of the crash.

By looking at the features and a sample from the data, the features look of numerical and catogerical types. What about some descriptive statistics?
"""
"""
### 2 - Data Preparation
"""
"""
#### 2-1 Statistics and correlations
"""
print('---------------File: train.csv-------------------')
print("#################################################")
print("Statistics describtion:")
print(df.drop(columns='ID').describe())
print("#################################################")
print("correlation matrix=")
print(df.corr())
print("#################################################")
print('---------------File: train.csv-------------------')

# using Kmeans Clustering for Lat and Lng:
X = df[['Lat','Lng']].to_numpy()


#Finding the optimum number of clusters for k-means classification
from sklearn.cluster import KMeans
css = []

for i in range(1, 11):
    kmeans = KMeans(n_clusters = i, init = 'k-means++', max_iter = 300, n_init = 10, random_state = 0)
    kmeans.fit(X)
    css.append(kmeans.inertia_)
    
#Plotting the results onto a line graph, allowing us to observe 'The elbow'
plt.plot(range(1, 11), css)
plt.title('The elbow method')
plt.xlabel('Number of clusters')
plt.ylabel('CSS') #within cluster sum of squares
plt.show()


#Applying kmeans to the dataset / Creating the kmeans classifier
kmeans = KMeans(n_clusters = 10, init = 'k-means++', max_iter = 500, n_init = 11, random_state = 0)
y_kmeans = kmeans.fit_predict(X)




#Visualising the clusters
plt.scatter(X[y_kmeans == 0, 0], X[y_kmeans == 0, 1], s = 50, c = 'red', label = 'City-1')
plt.scatter(X[y_kmeans == 1, 0], X[y_kmeans == 1, 1], s = 50, c = 'blue', label = 'City-2')
plt.scatter(X[y_kmeans == 2, 0], X[y_kmeans == 2, 1], s = 50, c = 'green', label = 'City-3')
plt.scatter(X[y_kmeans == 3, 0], X[y_kmeans == 3, 1], s = 50, c = 'purple', label = 'City-4')
plt.scatter(X[y_kmeans == 4, 0], X[y_kmeans == 4, 1], s = 50, c = 'yellow', label = 'City-5')

#Plotting the centroids of the clusters
plt.scatter(kmeans.cluster_centers_[:, 0], kmeans.cluster_centers_[:,1], s = 250, c = 'Black', label = 'Centroids', marker='*')

plt.legend()
plt.xlabel("Lat")
plt.ylabel("Lng")



df["City No."]=kmeans.fit_predict(X)
df.head()
print('---------------File: weather-sfcsv.csv-------------------')
print("#################################################")
print("Statistics describtion:")
print(df_weather.describe())
print("#################################################")
print("correlation matrix=")
print(df_weather.corr())
print("#################################################")
print("removing NaN from Weather_Condition")
df_weather.dropna(subset=['Weather_Condition'], inplace=True)
print("#################################################")
print("split Weather_Condition to Weather and Is_Windy")
df_weather[['Weather','Is_Windy']]=df_weather.Weather_Condition.str.split(" / ",expand=True)
df_weather["Is_Windy"] = df_weather["Is_Windy"].notnull().astype('int')
print("#################################################")
print("Encoding Weather_Condition:")
Weather_Condition_coding={"Weather":{'Clear':1,'Fair':2,'Partly Cloudy':3,'Mostly Cloudy':4,'Cloudy':5,'Overcast':6,
                                               'Smoke':7,'Scattered Clouds':8,'Light Drizzle':9,'Light Rain':10,'Rain':11,
                                               'Heavy Rain':12,'Haze':13,'Mist':14,'Patches of Fog':15,'Fog':16,'Shallow Fog':17,
                                               'Squalls':18,'Light Thunderstorms and Rain':19}}
df_weather = df_weather.replace(Weather_Condition_coding)
df_weather.info()
#print("#################################################")
"""
#### 2-2 Drop Unnecessary Columns & Convert & Encoding & merge
"""
print('---------------File: train.csv-------------------')
print("Convert bool Type to int Type:")
print("'Crossing','Give_Way','Junction','No_Exit','Railway','Stop','Amenity'")
df['Crossing'] = df['Crossing'].astype('int')
df['Give_Way'] = df['Give_Way'].astype('int')
df['Junction'] = df['Junction'].astype('int')
df['No_Exit'] = df['No_Exit'].astype('int')
df['Railway'] = df['Railway'].astype('int')
df['Stop'] = df['Stop'].astype('int')
df['Amenity'] = df['Amenity'].astype('int')
print("#################################################")
print("Convert 'timestamp' to datetime:")
df.timestamp = pd.to_datetime(df.timestamp)
print("#################################################")
print("Encoding 'Side' to bool Type:")
side_encod={"Side":{"R":0, "L":1}}
df = df.replace(side_encod)
print("#################################################")
print("Drop Unnecessary Columns")
df= df.drop(['Bump','Roundabout'],axis=1)
print("#################################################")
print('---------------File: weather-sfcsv.csv-------------------')
print("#################################################")
print('Removing Null values from preceptation and Build Model to free Nulls')
clean_weather = df_weather.dropna()
train_df_weather, val_df_weather = train_test_split(clean_weather, test_size=0.2, random_state=42) 
X_train = train_df_weather.drop(columns=['Weather_Condition','Precipitation(in)','Year', 'Day', 'Month', 'Hour', 'Wind_Chill(F)','Selected'])
y_train = train_df_weather['Precipitation(in)']
X_val = val_df_weather.drop(columns=['Weather_Condition','Precipitation(in)', 'Year', 'Day', 'Month', 'Hour', 'Wind_Chill(F)', 'Selected'])
y_val = np.array(val_df_weather['Precipitation(in)'])
print("#################################################")
print('LinearRegression Model')
Weather_model = LinearRegression()
Weather_model.fit(X_train,y_train)
Weather_model.score(X_train,y_train)
print("#################################################")
print('dataset null')
dataset = df_weather[df_weather['Precipitation(in)'].isnull()]
x = dataset.drop(columns=['Weather_Condition','Precipitation(in)','Year', 'Day', 'Month', 'Hour', 'Wind_Chill(F)','Selected'])
y = dataset['Precipitation(in)']
x.fillna(x.mean(), inplace=True)
df_weather.loc[df_weather["Precipitation(in)"].isnull(), "Precipitation(in)"] = Weather_model.predict(x)
print("#################################################")
print('free NAN from Wind_Speed(mph)')
df_weather.loc[df_weather["Wind_Speed(mph)"].isnull(),"Wind_Speed(mph)"] = df_weather["Wind_Speed(mph)"].mean()
print("#################################################")
print('---------------File: train.csv-------------------')
print("#################################################")
print("Drop Unnecessary Columns")
df_weather=df_weather.drop(['Wind_Chill(F)','Selected'],axis=1)
print("#################################################")
#print("Drop Null Rows")
#df_weather=df_weather.dropna()
print("#################################################")
print("Extract year, month, day, hour from timestamp")
def get_my_date(row):
    return pd.Timestamp(year=row.Year, month=row.Month, day=row.Day, hour=row.Hour)
df_weather["timestamp"] = df_weather.apply(get_my_date, axis=1)
print("#################################################")
print("Remove duplication from timestamp")
df_weather.drop_duplicates(subset="timestamp", inplace=True)
df_weather.reset_index(inplace=True, drop=True)
df_weather.info()
print("#################################################")
print("round Hours in timestamp")

def reset(row):
    return row.round(freq="min")
df_weather.timestamp = df_weather.timestamp.apply(reset)
df.timestamp = df.timestamp.apply(reset)
print("#################################################")
print("merge year, month, day, hour to timestamp")
merged = pd.merge(df, df_weather, how="left", on="timestamp")
print("#################################################")
merged.info()
print('---------------File: weather-sfcsv.csv-------------------')
print("1#################################################")
df_holidays['date'] = pd.to_datetime(df_holidays['date']).dt.date
merged['date'] = pd.to_datetime(merged['timestamp']).dt.date
final_merged = pd.merge(merged, df_holidays, how="left", on="date")
print(df_weather.info())
print("2#################################################")
final_merged.loc[:,"is_holiday"] = final_merged["description"].notnull()
final_merged.head()
print(final_merged.info())
print("3#################################################")
final_merged['is_holiday'] = final_merged['is_holiday'].astype('int')
print("#################################################")
final_merged= final_merged.drop(["description",'date','Precipitation(in)'],axis=1)
print("#################################################")
final_merged_free=final_merged.dropna()  
#final_merged.fillna(final_merged.mean(), inplace=True)
print("#################################################")
print(final_merged_free.info())
#Normalization
scaler = preprocessing.MinMaxScaler()
columns = ["Temperature(F)","Humidity(%)","Wind_Speed(mph)","Visibility(mi)"]
for col in columns:
        final_merged_free[col] = scaler.fit_transform(final_merged_free[col].values.reshape(-1, 1))
int_cols = ["Is_Windy","Weather","Year","Day","Month","Hour"]
for c in int_cols:
    final_merged_free[c] = final_merged_free[c].astype(int)

print(final_merged.info())
print(final_merged_free['Lng'])
"""
#### 2-3 Data Visualization
"""
print('---------------File: train.csv-------------------')
print("scatter_matrix for Lat, Lng, Distance(mi),Severity: ")
scatter_matrix(df[['Lat', 'Lng', 'Distance(mi)','Severity']], figsize=(10,10));
plt.show()
print("#################################################")
print("Histogram:")
df.Lat.hist();plt.title('Lat Histogram');plt.xlabel('Lat');plt.ylabel('Values');plt.show()
df.Lng.hist();plt.title('Lng Histogram');plt.xlabel('Lng');plt.ylabel('Values');plt.show()
df.Severity.hist();plt.title('Severity Histogram');plt.xlabel('Severity');plt.ylabel('Values');plt.show()
df.timestamp.hist();plt.title('timestamp Histogram');plt.xlabel('timestamp');plt.ylabel('Values');plt.show()
print("#################################################")
print('---------------File: weather-sfcsv.csv-------------------')
print("#################################################")
print("scatter_matrix for 'Year', 'Day', 'Month','Hour','Temperature(F)','Humidity(%)','Wind_Speed(mph)','Visibility(mi)': ")
scatter_matrix(df_weather[['Year', 'Day', 'Month','Hour','Temperature(F)','Humidity(%)','Wind_Speed(mph)','Visibility(mi)']], figsize=(10,10));
plt.show()
print("#################################################")
print("Histogram:")
df_weather.Year.hist();plt.title('Year Histogram');plt.xlabel('Year');plt.ylabel('Values');plt.show()
df_weather.Day.hist();plt.title('Day Histogram');plt.xlabel('Day');plt.ylabel('Values');plt.show()
df_weather.Month.hist();plt.title('Month Histogram');plt.xlabel('Month');plt.ylabel('Values');plt.show()
#df.Temperature(F).hist();plt.title('Temperature(F) Histogram');plt.xlabel('Temperature(F)');plt.ylabel('Values');plt.show()
#df.Humidity(%).hist();plt.title('Humidity(%) Histogram');plt.xlabel('Humidity(%)');plt.ylabel('Values');plt.show()
#df.Wind_Speed(mph).hist();plt.title('Wind_Speed(mph) Histogram');plt.xlabel('Wind_Speed(mph)');plt.ylabel('Values');plt.show()
#df.Visibility(mi).hist();plt.title('Visibility(mi) Histogram');plt.xlabel('Visibility(mi)');plt.ylabel('Values');plt.show()
print("#################################################")
"""
The output shows desciptive statistics for the numerical features, `Lat`, `Lng`, `Distance(mi)`, and `Severity`. I'll use the numerical features to demonstrate how to train the model and make submissions. **However you shouldn't use the numerical features only to make the final submission if you want to make it to the top of the leaderboard.**
"""
"""
## Data Splitting

Now it's time to split the dataset for the training step. Typically the dataset is split into 3 subsets, namely, the training, validation and test sets. In our case, the test set is already predefined. So we'll split the "training" set into training and validation sets with 0.8:0.2 ratio. 

*Note: a good way to generate reproducible results is to set the seed to the algorithms that depends on randomization. This is done with the argument `random_state` in the following command* 
"""
from sklearn.model_selection import train_test_split
train_df, val_df = train_test_split(final_merged_free, test_size=0.2, random_state=42) # Try adding `stratify` here  , shuffle=True, stratify=final_merged_free[['Severity']]
X_train = train_df.drop(columns=['ID', 'Severity','Weather_Condition','timestamp','Year','Day','Weather','Temperature(F)','Give_Way','Railway','City No.'])
y_train = train_df['Severity']
X_val = val_df.drop(columns=['ID', 'Severity','Weather_Condition','timestamp','Year','Day','Weather','Temperature(F)','Give_Way','Railway','City No.'])
y_val = val_df['Severity']

#is_holiday  ,'Year','Day','Weather','Temperature(F)','Give_Way','Railway'
"""
As pointed out eariler, I'll use the numerical features to train the classifier. **However, you shouldn't use the numerical features only to make the final submission if you want to make it to the top of the leaderboard.** 
"""
## This cell is used to select the numerical features. IT SHOULD BE REMOVED AS YOU DO YOUR WORK.
#X_train = X_train[['Lat', 'Lng', 'Distance(mi)']]
#X_val = X_val[['Lat', 'Lng', 'Distance(mi)']]
"""
## Model Training

Let's train a model with the data! We'll train a Random Forest Classifier to demonstrate the process of making submissions. 
"""
from sklearn.ensemble import RandomForestClassifier

# Create an instance of the classifier
classifier = RandomForestClassifier(max_depth=2, random_state=0)

# Train the classifier
classifier = classifier.fit(X_train, y_train)
"""
Now let's test our classifier on the validation dataset and see the accuracy.
"""
print("The accuracy of the classifier on the validation set is ", (classifier.score(X_val, y_val)))
#0.8823529411764706
"""
Well. That's a good start, right? A classifier that predicts all examples' `Severity` as 2 will get around 0.63. You should get better score as you add more features and do better data preprocessing.
"""
"""
## Submission File Generation

We have built a model and we'd like to submit our predictions on the test set! In order to do that, we'll load the test set, predict the class and save the submission file. 

First, we'll load the data.
"""
test_df = pd.read_csv(os.path.join(dataset_path, 'test.csv'))
test_df.head()
"""
Note that the test set has the same features and doesn't have the `Severity` column.
At this stage one must **NOT** forget to apply the same processing done on the training set on the features of the test set.

Now we'll add `Severity` column to the test `DataFrame` and add the values of the predicted class to it.

**I'll select the numerical features here as I did in the training set. DO NOT forget to change this step as you change the preprocessing of the training data.**
"""
X = test_df[['Lat','Lng']].to_numpy()
kmeans = KMeans(n_clusters = 10, init = 'k-means++', max_iter = 500, n_init = 11, random_state = 0)
test_df["City No."]=kmeans.fit_predict(X)
test_df.timestamp = pd.to_datetime(test_df.timestamp)
side_encod={"Side":{"R":0, "L":1}}
test_df = test_df.replace(side_encod)


test_df['Crossing'] = test_df['Crossing'].astype('int')
test_df['Give_Way'] = test_df['Give_Way'].astype('int')
test_df['Junction'] = test_df['Junction'].astype('int')
test_df['No_Exit'] = test_df['No_Exit'].astype('int')
test_df['Railway'] = test_df['Railway'].astype('int')
test_df['Stop'] = test_df['Stop'].astype('int')
test_df['Amenity'] = test_df['Amenity'].astype('int')
test_df.timestamp = pd.to_datetime(test_df.timestamp)

test_df= test_df.drop(['Bump','Roundabout'],axis=1)

test_merged = pd.merge(test_df, df_weather, how="left", on="timestamp")
df_holidays['date'] = pd.to_datetime(df_holidays['date']).dt.date
test_merged['date'] = pd.to_datetime(test_merged['timestamp']).dt.date

final_test_merged = pd.merge(test_merged, df_holidays, how="left", on="date")
final_test_merged.loc[:,"is_holiday"] = final_test_merged["description"].notnull()
final_test_merged['is_holiday'] = final_test_merged['is_holiday'].astype('int')
final_test_merged= final_test_merged.drop(["description",'date','Precipitation(in)'],axis=1)
final_test_merged.fillna(final_test_merged.mean(), inplace=True)
#final_test_merged_free=final_test_merged.fillna(final_test_merged.mean())
#Normalization
scaler = preprocessing.MinMaxScaler()
columns = ["Temperature(F)","Humidity(%)","Wind_Speed(mph)","Visibility(mi)"]
for col in columns:
        final_test_merged[col] = scaler.fit_transform(final_test_merged[col].values.reshape(-1, 1))
int_cols = ["Is_Windy","Weather","Year","Day","Month","Hour"]
for c in int_cols:
    final_test_merged[c] = final_test_merged[c].astype(int)

print(final_test_merged.info())
X_test = final_test_merged.drop(columns=['ID','Weather_Condition','timestamp', 'Distance(mi)','Year','Day','Weather','Temperature(F)','Give_Way','Railway'])
# You should update/remove the next line once you change the features used for training
#X_test = X_test[['Lat', 'Lng', 'Distance(mi)']]
y_test_predicted = classifier.predict(X_test)
test_df['Severity'] = y_test_predicted
test_df.head()
#X_test = test_df.drop(columns=['ID'])

## You should update/remove the next line once you change the features used for training
#X_test = X_test[['Lat', 'Lng', 'Distance(mi)']]

#y_test_predicted = classifier.predict(X_test)

#test_df['Severity'] = y_test_predicted

#test_df.head()
"""
Now we're ready to generate the submission file. The submission file needs the columns `ID` and `Severity` only.
"""
test_df[['ID', 'Severity']].to_csv('/kaggle/working/submission.csv', index=False)
"""
The remaining steps is to submit the generated file and are as follows. 

1. Press `Save Version` on the upper right corner of this notebook.
2. Write a `Version Name` of your choice and choose `Save & Run All (Commit)` then click `Save`.
3. Wait for the saved notebook to finish running the go to the saved notebook.
4. Scroll down until you see the output files then select the `submission.csv` file and click `Submit`.

Now your submission will be evaluated and your score will be updated on the leaderboard! CONGRATULATIONS!!
"""
"""
## Conclusion

In this notebook, we have demonstrated the essential steps that one should do in order to get "slightly" familiar with the data and the submission process. We chose not to go into details in each step to keep the welcoming notebook simple and make a room for improvement.

You're encourged to `Fork` the notebook, edit it, add your insights and use it to create your submission.
"""