"""
# A Message to readers

Given the importance of Data Preprocessing in ML, I as a beginner used to be inconsiderate of its effect in model performance. Neglecting it always put me behind in competitions until I explored the methods comprehensively and learned what data science really is. I aim to cover the methods in my notebooks so that I can enlist the prevalent methods for my and your future reference.

Below is my take on how handle outliers in your data. My target is to make it comprehensive and include all the research done in the field. Yet the notebook is not extensive but will be will in the near future. Stay tuned for future versions. 


Please provide your suggestions and compliments in comments! Feel to point out any mistakes I might have made. They help a lot in the learning process!!

Upvote and follow the notebook for updates on future versions! Happy Learning :)
"""
"""
# Problem Statement
The problem statement was posted on [Analytics Vidhya](https://datahack.analyticsvidhya.com/contest/all/) as a competition - [JanataHack: Mobility Analytics](https://datahack.analyticsvidhya.com/contest/janatahack-mobility-analytics/). 
You can access the data there while I will post the gist of the problem statement which will make you ready to go!

Welcome to Sigma Cab Private Limited - a cab aggregator service. Their customers can download their app on smartphones and book a cab from any where in the cities they operate in. They, in turn search for cabs from various service providers and provide the best option to their client across available options. They have been in operation for little less than a year now. During this period, they have captured surge_pricing_type from the service providers.

You have been hired by Sigma Cabs as a Data Scientist and have been asked to build a predictive model, which could help them in predicting the surge_pricing_type pro-actively. This would in turn help them in matching the right cabs with the right customers quickly and efficiently.

![image.png](attachment:image.png)

I had submitted my score and got a rank of roughly 170 (as far as I remember). I would share the approach in a future notebook and stick with outlier handling in this one!

### It is a multi-class classification problem.
"""
"""
# IMPORTS & necessary data preprocessing before getting into Outlier handling
"""
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline
plt.rcParams['figure.figsize'] = (8,6) # setting a default preferred size for plots
plt.style.use('fivethirtyeight')
sns.set(style='darkgrid')
import scipy
from scipy import stats
df = pd.read_csv('../input/mobility/train.csv')
df
df.drop('Trip_ID', axis=1,inplace=True); target = 'Surge_Pricing_Type'
numfeat, catfeat = list(df.select_dtypes(include=np.number)), list(df.select_dtypes(exclude=np.number)); numfeat.remove(target)
df
df.info()
"""
### Dont like tabular ugly datas hence dont understand a thing?
[This](https://www.kaggle.com/twinkle0705/a-comprehensive-guide-to-handle-missing-values) notebook tells you about beautiful insightful visualizations for missing values. Lets be done with it and move to the main methods of outlier handling.
"""
import missingno as msno; msno.bar(df, figsize=(16,6)); plt.show()
"""
So simple and pretty already.

Understand what it tells? Well, obviously...
"""
df[numfeat].skew()
"""
See, the numeric features do not have much skew. We can safely assume mean ~= median and fill with median. 

You get what I am saying? Well, obviously...
"""
for col in catfeat:
    df[col].fillna(df[col].mode()[0], inplace=True)
for col in numfeat:
    df[col].fillna(df[col].median(), inplace=True)
"""
# Now, outliers - what are they?
Outlier is defined as an observation point that is distant from the mainstream data. The presence of outliers can break a model’s analysis ability. 
One of the most efficient ways to identify outliers may be data visualization. But we'll go beyond that in this notebook.

### Definition of Hawkins [Hawkins 1980]:
“An outlier is an observation which deviates so much from the other observations as to arouse suspicions that it was generated by a different mechanism”


Usually, features are expected to follow some statistical process - a generating mechanism. A lot of times we observe the feature distributions deviating from this "generating mechanism". 

ML models generally expect **Normal Distribution** of the data hence we proceed assuming our features should be Normally Distributed. 

*(Ever wonder "WHY NORMAL DISTRIBUTION EVERYTIME!!!"? Here's [Cédric Villani](https://www.springer.com/birkhauser?SGWID=0-40290-6-986722-0) in his [TED Talk explaining why](https://www.youtube.com/watch?v=Kc0Kthyo0hU)).*

Outlier handling involves 2 steps:-
 1. Detecting the outliers
 2. Treating the detected outliers
 
Lets proceed with various methods of detection of the outliers. 
"""
"""
## Detecting outliers using Univariate Analysis

There are numerous ways one can go about detecting outliers. The one which we usually use - visualizations of univariate, bivariate, trivariate.....multivariate, and statistical calculations. 

Univariate visualizations would be the case of checking the distribution of data. 


"""
"""
### Distribution plots
"""
f, a = plt.subplots(2,4, figsize=(24,12))
a = a.flatten().T
for i, col in enumerate(df[numfeat].columns):
    sns.distplot(df[col],ax=a[i],kde=False).set_title('Skew: {:.4f}'.format(df[col].skew()))
plt.show()
"""
Do you see any outliers for any feature? Well, with plots it can be a bit difficult more so when the data is huge. Plus there is an argument of artificial vs natural outliers. How do we go on about deciding which outlier is which? 

It is hard to identify outliers if the data range is not specified.
"""
"""
### Normal Probability Plot
#### The normal probability plot is formed by:
   *  Vertical axis: Ordered response values
   *  Horizontal axis: Normal order statistic medians
Well, what do they mean? Check it out [here](https://online.stat.psu.edu/stat501/lesson/4/4.6).
   
The crux is - **The further the points vary from this line, the greater the indication of departures from normality.**
"""
f, a = plt.subplots(2,4, figsize=(24,12))
a = a.flatten().T
for i, col in enumerate(df[numfeat].columns):
    stats.probplot(df[col], plot=a[i])
    a[i].set_title(col)
plt.show()
"""
Now, the assumption that out data should be Normally Distributed as the generating mechanism was so, we can clearly see from the probability plots that there exist values which are responsible for deviation from Normal Distribution. 

So, one can nitpick values from the visualizations and classify them as "Outliers" and treat accordingly.

#### Example:-
*  Similarly, values beyond 160 can be treated as **Outliers** for feature 'Var1'.
*  It can be clearly seen that values between 50 and 60, and values beyond 150 are responsible for the deviation and hence can be classified as **Outliers** for feature 'Var3'.
*  Values beyond 4.0 are **Outliers** for feature 'Life_Style_Index' and so on for other features too..

## Treatment of outliers
*  Removing observations which involve these outliers 
*  Or transform the data 
*  Or treat them as we would treat missing values and impute accordingly
"""
## REMOVING OBSERVATIONS BASED ON CLASSIFICATION IN 'EXAMPLE OUTLIERS' FOR FEATURE 'Var1'
df.drop(df[df['Var1']>160].index).reset_index(drop=True)
## TRANSFORMING THE FEATURE TO TREAT OUTLIERS USING BOXCOX TRANSFORMATION FOR FEATURE 'Var3'
temp = pd.Series(stats.boxcox(df['Var3'],lmbda=stats.boxcox_normmax(df['Var3'])))
stats.probplot(temp, plot=plt); plt.show()
## TREATING OUTLIERS AS MISSING VALUES FOR FEATURE 'Life_Style_Index'
df['Life_Style_Index'].where(df['Life_Style_Index']<4.0, df['Life_Style_Index'].median())
"""
So, these was the most basic (and comprehensive) way of treating outliers based on univariate analysis of numeric features. Doing so might yield you better results already. But we aren't done yet, **are we?**

## Detecting (more) outliers using Multivariate Analysis
Your data might still have outliers so we are not done yet! How do we detect them in the first place?

Well, we are still left with bivariate (involving two variables/features) analysis of the data. These will the last kind of analysis we'll be able to plot (unless you delve into 3D plotting or 2D plots with hue).

Starting off with bivariate visualizations we realize that we can now have 
* Numeric-Numeric analysis
* and Numeric-Categorical analysis 

for which we will use scatterplots and boxplots respectively.

### Scatterplots
A scatterplot is a graphic tool used to display the relationship between two quantitative variables. They are nice to the eye and easy to interpret and draw conclusions from. Lets see how!

"""
sns.pairplot(df[numfeat]); plt.show()
sns.scatterplot('Var2', 'Var3', data=df, hue = target).set_title('Correlation: {:.4f}'.format(df['Trip_Distance'].corr(df['Life_Style_Index']))); plt.show()
from scipy.spatial import ConvexHull

extremes = df[['Var2', 'Var3']].to_numpy()

hull = ConvexHull(extremes)

# print(extremes[hull.vertices])

plt.plot(df["Var2"], df["Var3"], 'ok')
plt.plot(extremes[hull.vertices, 0], extremes[hull.vertices,1], 'r--', lw = 2)
plt.plot(extremes[hull.vertices, 0], extremes[hull.vertices,1], 'ro', lw = 2)
plt.show()
"""
## Now, what is this doing for us? Thats for you to answer. I will upload the answer in the next version of the notebook which will also include:
* ###  Complete bi-variate and multivariate analysis
* ###  Un-supervised learning for outlier-inlier clustering
* ###  Outliers and Marginal Objects Detection
* ###  Do categorical variables have any outliers?

### I am sure we are gonna learn a lot from this notebook. I would learn from your experience and suggestions and you might learn a bit from my research and coverage of the topic so crucial to Data Preprocessing! See ya!
"""
"""
References:-
*  [Sets up the pipeline for you](https://www.analyticsvidhya.com/blog/2016/01/guide-data-exploration/)
*  [A great comprehensive resource](https://www.itl.nist.gov/div898/handbook/eda/section3/eda35h.htm) to learn statistical methods
*  [Pandas documentation](https://pandas.pydata.org/pandas-docs/stable/reference/frame.html) comes in very handy
*  [Scipy Stats documentation](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.probplot.html) is worth a read after your stats lecture
*  [Google Scholar search on published papers](https://scholar.google.co.in/scholar?q=outlier+detection+papers&hl=en&as_sdt=0&as_vis=1&oi=scholart) 
"""