"""
# General information about this notebook
This notebook has empty output cells. It ran outside of Kaggle for computation reasons:
- Kaggle has many libraries, so dependencies
- long notebook runtime

However you can find the full notebook with full output on Github:
https://github.com/ThomasMeissnerDS/e2e_ml/blob/develop/Example%20notebooks/Auto%20exploration%20using%20Timewalk/Kaggle%20credit%20card%20fraud%20with%20Timewalk.ipynb
"""
"""
This notebook did not run on Kaggle, but a local system.
We are running on e2eml version 2.10.1
"""
"""
# Introduction

Hi,

I am Thomas, creator of the automl library e2eml. In this notebook I give you a little walkthrough and example run using a feature called Timewalk.

I created this library for personal development, but also to give something back to the data science community. You can install the library using !pip install e2eml.


# What does e2eml offer?
e2eeml has two major goals: You can either speed up your prototyping and exploration (with Timewalk) or create a full pipeline with just a few lines of code. e2eml will take care of:
- data preprocessing
- model training
- model fine-tuning
- model evaluation
- logging file

It can handle datetime, categorical and numerical data and allows you to build classification & regression models. For NLP tasks you can even let it create a full BERT model for you.

# Why e2eml and not any other automl framework?
This decision is fully up to you. They all have their ins & outs. Chose what suits your needs best. e2eml is just an option for you.


# What is the spirit of e2eml?
This library tries to maximize the model performance. It shall help to see how far you can get given your data.
As a characteristic it creates huge notebooks, but we really wanted to print out a lot for. We want you to be able to actually see what is happening under the hood. That comes at a cost here. If you prefer very elegant and silent implementations, check out the fantastic Pycaret library.
Additionally being able to fine-tune BERT models for you can be a life saver. We also provide some GPU acceleration with RAPIDS. Currently this is implemented in a few spots only however.
"""
"""
# Importing libraries
"""
# import libraries and custom Postgres connector
from e2eml.full_processing import postprocessing
from e2eml.classification import classification_blueprints as cb
from e2eml.timetravel import timetravel

import pandas as pd
from sklearn import model_selection
from sklearn.metrics import matthews_corrcoef
from sklearn.metrics import classification_report
import re
import numpy as np
pd.set_option('display.max_colwidth', None)

import pickle
import gc
"""
# Import needed data
"""
target = 'Class'
creditcard = pd.read_csv('creditcard.csv')
creditcard
creditcard[target].value_counts()
creditcard[target].isna().sum()
len(creditcard.index)
"""
# Test train split
"""
"""
First we sort the dataset by timestamp and then hold back the newest data points as unseen holdout data for validation.
This has two reasons:
- Without any unseen data we cannot see overfitting
- Fraud does not consist of static behaviour. In real world applications fraudsters change their behaviour to adapt for anti-fraud mechanisms. For model prototyping (not final training!) it makes sense to keep newest data as holdout, so we can see, if our model can adapt to changing patterns (given they even change in our training data)
"""
creditcard = creditcard.sort_values(by=["Time"], ascending=[True])
train_df = creditcard.head(230000)
"""
Here we go crazy actually. Timewalk should be used with smaller samples, but we want to illustrate what might happen, if the sample is very big.
"""
train_df = train_df.sample(100000, random_state=1000)
train_df_target = train_df[target]
train_df_target.value_counts()
# actual holdout
holdout_df = creditcard.tail(50000)
hold_df_target = holdout_df[target]
del holdout_df[target]
hold_df_target.value_counts()
del creditcard
_ = gc.collect()
"""
# Automl using e2eml

We use e2eml. There are plenty of fantastic frameworks. Chose whatever you like.
"""
"""
Using e2eml we first instantiate a class, here a ClassificationBluePrint.
"""
automl_pipeline = cb.ClassificationBluePrint(datasource=train_df,
                                       target_variable=target,
                                       preferred_training_mode='cpu',
                                       tune_mode='accurate',
                                       ml_task='binary',
                                       rapids_acceleration=False
                                        )
"""
From here we have two general options. We can:
- run a certain blueprint straight away (this is great to create a ready-to-use pipeline for prediction)
- run Timewalk to fully explore many algorithms and preprocessing combinations

Here we assume that we don't know much about what works for this dataset. So we chose Timewalk. Timewalk takes a long time to run (this can be controlled by manually chosing algorithms and preprocessing steps to use). As we have plenty of data Timewalk will automatically switch off some algorithms. I.e. Xgboost will not run here (this has been designed due to problems of releasing the memory again and also due to high consumption of system resources).

As our data is imbalanced, we add this flag as a parameter. This will add an additional step for imbalanced data only.
"""
results = timetravel.timewalk_auto_exploration(class_instance=automl_pipeline,
                                   holdout_df=holdout_df,
                                   holdout_target=hold_df_target,
                                   is_imbalanced=True,
                                   algs_to_test=None,
                                   experiment_name="timewalk_creditcard_fraud.pkl")
results = pd.read_pickle("timewalk_creditcard_fraud.pkl")
results.sort_values(by=["Matthews"], ascending=[False]).head(30).drop_duplicates(subset=['Trial number', 'Algorithm', 'Matthews'], keep='last')
"""
# Results overview
Timewalk returns a result dataframe. Here we can sort the results by Matthews correlation to see our best results.

During our trial we ran out of memory. As Timewalk is still in an early stage we still try to aim for optimization. Timewalk tries to clean memory, but sometimes it does not clean properly., However we had a different reason here. For some reason memory consumption exploded after transforming the data.

However it is not a big problem here. We already saw very good results. If we would like to continue Timewalk we could pass the parameter "preprocess_checkpoints", which expects a list. On default this includes:
- "default"
- "delete_low_variance_features"
- "automated_feature_selection"
- "autotuned_clustering"
- "cardinality_remover"
- "delete_high_null_cols"
- "early_numeric_only_feature_selection"
- "fill_infinite_values"

We failed during checkpoint "cardinality_remover". So we could provide the list 
["default", 
 "delete_high_null_cols",
 "early_numeric_only_feature_selection",
 "fill_infinite_values"] instead. We exclude the failing step as well as we would trigger the same issue again.
 
 We can also chose to run less algorithms. SGD & Quadratic Discriminent Analysis did not look promising at all. TabNet has been very good, but extremely slow.
"""
"""
# Continue the trial
We continue our trial, but change the parameters (for this step we restartet Jupyter Kernel and rerun the code until ClassificationBlueprint instantiation. This is for demonstation purposes only).. We explicitly call certain algorithms and also reduce checkpoints ("default" is mandatory here). We also provide the name of an existing experiment file, so Timewalk can append new to previous results. This is kind of a mini MLOps utility.

What exactly happens here?
Timetravel will run the "default" step. During this step it creates different save points of our preprocessed data and stores them locally. In it's current implementation it does this every time we run Timnetravel. Please be aware, that this might consume much disk space (depending on dataset size).
Then Timewalk will load a previous preprocessing checkpoint from disk, adjust some preprocessing decisions (i.e. different feature selection) and run the models again. In our case this should run a lot faster than in the attempt above, because we do not make use of all steps and algorithms anymore. Especially TabNet has been very slow. 
"""
results = timetravel.timewalk_auto_exploration(class_instance=automl_pipeline,
                                   holdout_df=holdout_df,
                                   holdout_target=hold_df_target,
                                   is_imbalanced=True,
                                    preprocess_checkpoints=["default", 
                                                            "delete_high_null_cols",
                                                            "early_numeric_only_feature_selection",
                                                            "fill_infinite_values"],
                                   algs_to_test=["lgbm", "lgbm_focal", "ridge", "logistic_regression", "vowpal_wabbit"],
                                   name_of_exist_experiment="timewalk_creditcard_fraud.pkl",
                                   experiment_name="timewalk_creditcard_fraud.pkl")
"""
After the delete_high_null_cols step we would run into the same issues as every following step would use feature transformation as well.
Here we could cose to run this npart with a smaller sample size, but LGBM focal seems to be the winner anyway.
"""
results = pd.read_pickle("timewalk_creditcard_fraud.pkl")
results.sort_values(by=["Matthews"], ascending=[False]).head(30).drop_duplicates(subset=['Trial number', 'Algorithm', 'Matthews'], keep='last')
import plotly.express as px
fig = px.line(
            results,
            x="Total runtime",
            y="Matthews",
            color="Algorithm",
            text="Trial number",
            title='Performance vs runtime comparison of algorithms'
        )
fig.update_traces(textposition="bottom right")
fig.show()
"""
LGBM focal is has been fast and superior. This makes sense as focal loss is most suitable for imbalanced data (binary and multiclass).
"""
"""
# Creating the final pipeline
We:
- restart the notebook
- reimport libraries
- reload the data, but this time use 230k rows for training instead of 100k
- load in the results from Timnewalk
- instantiate a ClassificationBlueprint class
- take the winning parameters from results and overwrite the class default preprocessing steps
- run the lgbm_focal_loss pipeline
- run the function agsin (from here it is in prediction mode and will predict on new data)
- do a final evaluation

Let's see, if more data can help improving the score. It could also hurt performance, if we sudeenly have more bad data points in the set. 
"""
target = 'Class'
creditcard = pd.read_csv('creditcard.csv')
creditcard = creditcard.sort_values(by=["Time"], ascending=[True])
train_df = creditcard.head(230000)
train_df = train_df.sample(230000, random_state=1000)
train_df_target = train_df[target]
# actual holdout
holdout_df = creditcard.tail(50000)
hold_df_target = holdout_df[target]
del holdout_df[target]
del creditcard
_ = gc.collect()
results = pd.read_pickle("timewalk_creditcard_fraud.pkl")
results.sort_values(by=["Matthews"], ascending=[False]).head(30).drop_duplicates(subset=['Trial number', 'Algorithm', 'Matthews'], keep='last')
winner_set = results[(results["Accuracy"].isna() == False)].sort_values(by=["Matthews"], ascending=[False]).head(1)
winner_set
automl_pipeline = cb.ClassificationBluePrint(datasource=train_df,
                                       target_variable=target,
                                       preferred_training_mode='cpu',
                                       tune_mode='accurate',
                                       ml_task='binary',
                                       rapids_acceleration=False
                                        )
# showing the preprocessing steps of our best iteration and model
best_params = winner_set["Preprocessing applied"].values.tolist()[0]

# use best prarams from timewalk
for key, value in best_params.items():
    automl_pipeline.blueprint_step_selection_non_nlp[key] = value
automl_pipeline.ml_bp14_multiclass_full_processing_lgbm_focal()
automl_pipeline.ml_bp14_multiclass_full_processing_lgbm_focal(holdout_df)
algorithms = ["lgbm_focal"]

def get_matthews(algorithm):
    # Assess prediction quality on holdout data
    print(classification_report(hold_df_target, automl_pipeline.predicted_classes[algorithm]))
    try:
        matthews = matthews_corrcoef(hold_df_target, automl_pipeline.predicted_classes[algorithm])
    except Exception:
        print("Matthew failed.")
        matthews = 0
    print(matthews)
    
for i in algorithms:
    print(f"---------Start evaluating {i}----------")
    get_matthews(i)
"""
More data did not improve our model.
"""
"""
# Summary

It seems like we created a good model here. However some notes:
- Accuracy is not a good metric for imbalanced data. Matthews correlation is a lot better as a general metric
- Our model could still fail on production! The training data might be from a wrong season or just too old and would suffer a lot seeing new patterns in production. So monitoring ongoing performance of your model is extremely important.
- Does automl replace data scientists? No, it empowers them. Instead of building stuff from scratch, data scientists can focus on finding better features. Also it needs expertise to properly evaluate a model. Automl cannot replace domain knowledge at all.

I hope you enjoyed the notebook. Upvotes are very welcome :-) 
"""