"""
<div style="font-size: 16px; font-weight: 500; line-height: 1.4; font-style: italic; margin-bottom: 1em;">
  Pitting My Neural Network Against a Corporate Benchmark
</div>

1. **[Introduction](#Introduction)**
2. **[Ground rules](#Ground-rules)**
3. **[Test metric](#Test-metric)**
4. **[LendingClub's turn](#LendingClub's-turn)**
5. **[My turn](#My-turn)**
6. **[Victory!](#Victory!)**
7. **[Further reading](#Further-reading)**

## Introduction

In case you missed it, I [built a neural network to predict loan risk](https://www.kaggle.com/tywmick/building-a-neural-network-to-predict-loan-risk) using a [public dataset](https://www.kaggle.com/wordsforthewise/lending-club) from [LendingClub](https://www.lendingclub.com/). Then I built a [public API](https://tywmick.pythonanywhere.com/) to serve the model's predictions. That's nice and all, but&hellip; how _good_ is my model?

Today I'm going to put it to the test, pitting it against the risk models of the very institution who issued those loans. That's right, LendingClub included their own calculated loan grades (and sub-grades) in the dataset, so all the pieces are in place for the most thrilling risk modeling smackdown of the ~~century~~ week. May the best algorithm win!
"""
import joblib

prev_notebook_folder = "../input/building-a-neural-network-to-predict-loan-risk/"
loans = joblib.load(prev_notebook_folder + "loans_for_eval.joblib")
loans.shape
loans.head()
"""
## Ground rules

This is going to be a clean fight—my model won't use any data LendingClub wouldn't have access to at the point they calculate a loan's grade (including the grade itself).

I'm going to sort the dataset chronologically (using the `issue_d` column, the month and year the loan was issued) and split it into two parts. The first 80% I'll use for training my competition model, and I'll compare performance on the last 20%.
"""
from sklearn.model_selection import train_test_split

loans["date"] = loans["issue_d"].astype("datetime64[ns]")
loans.sort_values("date", axis="index", inplace=True, kind="mergesort")

train, test = train_test_split(loans, test_size=0.2, shuffle=False)
train, test = train.copy(), test.copy()
print(f"The test set contains {len(test):,} loans.")
"""
At the earlier end of the test set my model may have a slight informational advantage, having been trained on a few loans that may not have closed yet at the point LendingClub was grading those ones. On the other hand, LendingClub may have a slight informational advantage on the later end of the test set, since they would have known the outcomes of some loans on the earlier end of the test set by that time.

I have to give credit to Michael Wurm, by the way, for [the idea](https://towardsdatascience.com/intelligent-loan-selection-for-peer-to-peer-lending-575dfa2573cb#fac8) of comparing my model's performance to LendingClub's loan grades, but my approach is pretty different. I'm not trying to simulate the performance of an investment portfolio; I'm just evaluating how well my predictions of simple risk compare.

## Test metric

The test: who can pick the best set of grade A loans, judged on the basis of the independent variable from [my last notebook](https://www.kaggle.com/tywmick/building-a-neural-network-to-predict-loan-risk), the fraction of an expected loan return that a prospective borrower will pay back (which I engineered as `fraction_recovered`).

LendingClub will take the plate first. I'll gather all their grade A loans from the test set, count them, and calculate their average `fraction_recovered`. That average will be the metric my model has to beat.

Then I'll train my model on the training set using the same [pipeline and parameters](https://www.kaggle.com/tywmick/building-a-neural-network-to-predict-loan-risk#Building-the-neural-networks) I settled on in my last notebook. Once it's trained, I'll use it to make predictions on the test set, then gather the number of top predictions equal to the number of LendingClub's grade A loans. Finally, I'll calculate the same average of `fraction_recovered` on that subset, and we'll have ourselves a winner!

## LendingClub's turn
"""
from statistics import mean

lc_grade_a = test[test["grade"] == "A"]
print(f"LendingClub gave {len(lc_grade_a):,} loans in the test set an A grade.")

print("\nAverage `fraction_recovered` on LendingClub's grade A loans:")
print(format(mean(lc_grade_a["fraction_recovered"]), ".5f"))
"""
That's a pretty high percentage. I'm a bit nervous.

## My turn

First, I'll copy over my `run_pipeline` function from [my previous notebook](https://www.kaggle.com/tywmick/building-a-neural-network-to-predict-loan-risk#Building-the-neural-networks). I'll hide the input here since it isn't new information, but feel free to take a peek by clicking the "Input" button.
"""
from sklearn.model_selection import train_test_split
from sklearn_pandas import DataFrameMapper
from sklearn.preprocessing import OneHotEncoder, OrdinalEncoder, StandardScaler
from tensorflow.keras import Sequential, Input
from tensorflow.keras.layers import Dense, Dropout


def run_pipeline(
    data,
    onehot_cols,
    ordinal_cols,
    batch_size,
    validate=True,
):
    X = data.drop(columns=["fraction_recovered"])
    y = data["fraction_recovered"]
    X_train, X_valid, y_train, y_valid = (
        train_test_split(X, y, test_size=0.2, random_state=0)
        if validate
        else (X, None, y, None)
    )

    transformer = DataFrameMapper(
        [
            (onehot_cols, OneHotEncoder(drop="if_binary")),
            (
                list(ordinal_cols.keys()),
                OrdinalEncoder(categories=list(ordinal_cols.values())),
            ),
        ],
        default=StandardScaler(),
    )

    X_train = transformer.fit_transform(X_train)
    X_valid = transformer.transform(X_valid) if validate else None

    input_nodes = X_train.shape[1]
    output_nodes = 1

    model = Sequential()
    model.add(Input((input_nodes,)))
    model.add(Dense(64, activation="relu"))
    model.add(Dropout(0.3, seed=0))
    model.add(Dense(32, activation="relu"))
    model.add(Dropout(0.3, seed=1))
    model.add(Dense(16, activation="relu"))
    model.add(Dropout(0.3, seed=2))
    model.add(Dense(output_nodes))
    model.compile(optimizer="adam", loss="mean_squared_logarithmic_error")

    history = model.fit(
        X_train,
        y_train,
        batch_size=batch_size,
        epochs=100,
        validation_data=(X_valid, y_valid) if validate else None,
        verbose=2,
    )

    return history.history, model, transformer


onehot_cols = ["term", "application_type", "home_ownership", "purpose"]
ordinal_cols = {
    "emp_length": [
        "< 1 year",
        "1 year",
        "2 years",
        "3 years",
        "4 years",
        "5 years",
        "6 years",
        "7 years",
        "8 years",
        "9 years",
        "10+ years",
    ]
}
# Train the model
_, model, transformer = run_pipeline(
    train.drop(columns=["issue_d", "date", "grade", "sub_grade", "expected_return"]),
    onehot_cols,
    ordinal_cols,
    batch_size=128,
    validate=False,
)

# Make predictions
X_test = transformer.transform(
    test.drop(
        columns=[
            "fraction_recovered",
            "issue_d",
            "date",
            "grade",
            "sub_grade",
            "expected_return",
        ]
    )
)
test["model_predictions"] = model.predict(X_test)

# Gather top predictions
test_sorted = test.sort_values("model_predictions", axis="index", ascending=False)
ty_grade_a = test_sorted.iloc[0:len(lc_grade_a)]

# Display results
print("\nAverage `fraction_recovered` on Ty's grade A loans:")
print(format(mean(ty_grade_a["fraction_recovered"]), ".5f"))
"""
## Victory!

Phew, that was a close one! My win might be too small to be statistically significant, but hey, it's cool seeing that I can keep up with LendingClub's best and brightest.

## Further reading

- [Natural Language Processing for Loan Risk](https://www.kaggle.com/tywmick/natural-language-processing-for-loan-risk)

-----

What I'd really like to know now is what quantitative range of estimated risk each LendingClub grade and sub-grade corresponds to, but it looks like [that's proprietary](https://www.lendingclub.com/foliofn/rateDetail.action). Does anyone know if loans grades generally correspond to certain percentage ranges like letter grades in academic classes? If not, have any ideas for better benchmarks I could use to evaluate my model's performance? Go ahead and chime in in the comments below.
"""