"""
# Welcome to my notebook!
This competition is a classic machine learning classification problem. And like with most data science competition, **dealing with noise and data pre-processing is a major part of the process.** I will walk you through my entire thought process in dealing with noise present within the data. It should be easy enough for beginners to understand but don't hope to achieve a high score with this! :)


"""
"""
# Importing libraries
"""
import pandas as pd
import numpy as np
import re

from sklearn.metrics import matthews_corrcoef
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from xgboost import XGBRFClassifier

import matplotlib.pyplot as plt
import seaborn as sns
"""
# Read and inspect the csv files
"""
train_df = pd.read_csv("../input/open-shopee-code-league-marketing-analytics/train.csv").set_index('row_id')
test_df = pd.read_csv("../input/open-shopee-code-league-marketing-analytics/test.csv").set_index('row_id')
user_df = pd.read_csv("../input/open-shopee-code-league-marketing-analytics/users.csv")
train_df.head()
user_df.head()
"""
# Data Anomalies

With the help of [@soappp9527's EDA work](http://www.kaggle.com/soappp9527/marketing-analytics-eda-by-r-data-table-ggplot2), and inspecting them within the csv in the excel program, here are the basic anomalies that requires processing.

**1. Never open/login/checkout in 3 of the columns**

**2. Missing values in attr_1,2 and 3**

**3. Missing values in age**
"""
"""
# Merging Users Dataframe with Train and Test Dataframes

Every feature is important to me unless otherwise told by the random forest algorithm's feature importance. Since we are given the user's information, why not make use of the data? Especially when you think about it, age might probably play a factor in opening e-mails. So let's do a left join on both the train and test dataframes to add additional features for prediction.
"""
train_df = train_df.merge(user_df, left_on = 'user_id', right_on = 'user_id')
test_df = test_df.merge(user_df, left_on = 'user_id', right_on = 'user_id')
train_df.head()
"""
Now we see that every row has been added 5 additional features that could potentially help with the robustness of the prediction.
"""
"""
# Adding Additional Features by Adding Date Parts

When we look at the ``grass_date`` feature in the dataset, it may not appear to be a great predictive feature on its own. However, if we dissect the dates and separate it based on the day of week, day of month, etc.. Some of these additional features might serve as a great predictor.

For example, some might open an e-mail relating to e-commerce during the weekends in their own personal e-mails because that is when they are most likely to be free.

So I will make use of fast.ai's ``add_datepart`` function to help with the addition of these features.
"""
def add_datepart(df, fldname, drop=True):
    fld = df[fldname]
    if not np.issubdtype(fld.dtype, np.datetime64):
        df[fldname] = fld = pd.to_datetime(fld, infer_datetime_format=True)
    targ_pre = re.sub('[Dd]ate$', '', fldname)
    for n in ('Year', 'Month', 'Week', 'Day', 'Dayofweek', 'Dayofyear',
            'Is_month_end', 'Is_month_start', 'Is_quarter_end', 'Is_quarter_start', 'Is_year_end', 'Is_year_start'):
        df[targ_pre+n] = getattr(fld.dt,n.lower())
    if drop: df.drop(fldname, axis=1, inplace=True)
    
add_datepart(train_df, 'grass_date')
add_datepart(test_df, 'grass_date')
train_df.head()
"""
Now that we have separated the date into multiple features, it's time to remove features that are unnecessary for obvious reasons.
"""
def remove_datecols(df):
    fld = ['grass_Year','grass_Is_quarter_end','grass_Is_quarter_start','grass_Is_year_start','grass_Is_year_end']
    df.drop(fld, axis=1, inplace=True)
    
remove_datecols(train_df)
remove_datecols(test_df)
train_df.head()
"""
# Removing User ID
"""
#Remove userid because they are not required for prediction
train_df.drop('user_id', axis = 1, inplace = True)
test_df.drop('user_id', axis = 1, inplace = True)
"""
# One-Hot Encoding E-mail Domain and Country Code

I do this because these are categorical features and should not have any cardinality within the random forest algorithm.
"""
train_df = pd.concat([train_df.drop('country_code', axis=1), pd.get_dummies(train_df['country_code'])], axis=1)
train_df = pd.concat([train_df.drop('domain', axis=1), pd.get_dummies(train_df['domain'])], axis=1)
test_df = pd.concat([test_df.drop('country_code', axis=1), pd.get_dummies(test_df['country_code'])], axis=1)
test_df = pd.concat([test_df.drop('domain', axis=1), pd.get_dummies(test_df['domain'])], axis=1)
"""
# Processing Never Open/Login/Checkout and Other Missing Values
"""
#Replace never open, never checkout and never login to nan
train_df.replace(["Never open", "Never login", "Never checkout"], np.nan, inplace = True)
test_df.replace(["Never open", "Never login", "Never checkout"], np.nan, inplace = True)
train_df[['last_open_day','last_login_day','last_checkout_day']] = train_df[['last_open_day','last_login_day','last_checkout_day']].apply(pd.to_numeric)
test_df[['last_open_day','last_login_day','last_checkout_day']] = test_df[['last_open_day','last_login_day','last_checkout_day']].apply(pd.to_numeric)

#Fill missing values
train_df.fillna({"last_open_day":train_df['last_open_day'].max(),
                 "last_login_day":train_df['last_login_day'].max(),
                 "last_checkout_day":train_df['last_checkout_day'].max(),
                 "attr_1": 2,
                 "attr_2": 2,
                 "attr_3": 2,
                 "age": train_df['age'].median()}, inplace = True)

test_df.fillna({"last_open_day":test_df['last_open_day'].max(),
                 "last_login_day":test_df['last_login_day'].max(),
                 "last_checkout_day":test_df['last_checkout_day'].max(),
                 "attr_1": 2,
                 "attr_2": 2,
                 "attr_3": 2,
                 "age": test_df['age'].median()}, inplace = True)
"""
# Split Training Dataset for Modelling

Credits to [Nathaniel Ng's notebook](https://www.kaggle.com/nathaniel/marketing-analytics-baseline-0-41795) for the following chunk of code, saved me the trouble from typing it out on my own!
"""
def get_xy(df, target_col='open_flag', **kwargs):
    feature_cols = [ col for col in df.columns if col != target_col ]
    if target_col in df:
        X = df[feature_cols]
        y = df[target_col]
        X_train, X_valid, y_train, y_valid = train_test_split(X, y)
        return X_train, X_valid, y_train, y_valid
    else:
        X_test = df[feature_cols]
        return X_test
    
X_train, X_valid, y_train, y_valid = get_xy(train_df, test_size=0.2, random_state=123)
X_test = get_xy(test_df)
"""
# Modelling using Random Forest Classifier
"""
model = RandomForestClassifier(max_depth = 200, n_estimators = 300, n_jobs = -1, bootstrap = True, random_state = 123)
rf = model.fit(X_train, y_train)
"""
# Evaluating the Model
"""
y_pred = rf.predict(X_valid)
matthews_corrcoef(y_valid,y_pred)
"""
# Predicting the Test Set and Submitting to Kaggle
"""
# Make predictions on test set with RF
y_pred = model.predict(X_test)
submission_df = pd.DataFrame({'row_id':np.arange(len(test_df)),
                              'open_flag':y_pred})
submission_df.to_csv('submission.csv', index = False)
"""
# [Optional] Inspecting Feature Importance
"""
def imp_df(column_names, importances):
  df = pd.DataFrame({'variable': column_names,
                     'variable_importance': importances}) \
         .sort_values('variable_importance', ascending = False) \
         .reset_index(drop = True)
  return df
base_imp = imp_df(X_train.columns, rf.feature_importances_)

plt.figure(figsize = (15,8))
fig = sns.barplot(x = 'variable_importance', y = 'variable', data = base_imp, orient = 'h', color = 'royalblue')
plt.title("Variable Importance using RFC", fontsize = 20)
plt.xlabel('Variable Importance', fontsize = 16)
plt.ylabel('')
plt.xticks(fontsize = 14)
plt.yticks(fontsize = 14)
plt.show()
"""
# Conclusion

I am clearly still a beginner in fine-tuning the random forest classifier algorithm. It's been awhile since I messed around with machine learning algorithms, but I'm still happy I managed to tackle this competition!

There is still much to learn including using more sophisticated algorithms such as LightGBM and XGBoost as well as hyperparameter tuning using GridSearchCV.

There are still a lot of things you can do in data preparation such as, and not limited to:

***1. Dealing with outliers (negative age and age hitting 118)***

***2. Removing certain features that appear redundant with either Recursive Feature Elimination or simply deciding from the feature importance chart***

***3. More feature engineering, especially with the top 3 features.***

I hope you find this notebook useful especially in the data pre-processing part for your participation in this competition! :)
"""