"""
### [Context of this work : Défi IA](https://www.kaggle.com/c/defi-ia-insa-toulouse/leaderboard)

This edition of the Défi IA pertains to NLP. The task is straightforward: assign the correct **job category** to a **job description**.

### Desciption

The approach used here to make the model **fair** is to use as many descriptions of jobs of both sexes.
This is similar to when you have an unbalanced dataset, and you decide to reduce the number of samples in each class to the minimum sample.

### Results of this notebook

|Metric       |precision       |recall       |f1-score       |support       |
|:-:    |:-:    |:-:    |:-:    |--:    |
|accuracy       |       |       |0.80       |2423       |
|macro avg       |0.81       |0.80       |0.80       |2423       |
|weighted avg       |0.81       |0.80       |0.80       |2423       |
|fairness predict       |       |       |       |2.817007881684417       |
|fairness valid       |       |       |       |2.8221477674508764       |

"""
"""
# Introduction
"""
"""

## History

2018 was a breakthrough year in NLP. Transfer learning, particularly models like Allen AI's ELMO, OpenAI's Open-GPT, and Google's BERT allowed researchers to smash multiple benchmarks with minimal task-specific fine-tuning and provided the rest of the NLP community with pretrained models that could easily (with less data and less compute time) be fine-tuned and implemented to produce state of the art results. Unfortunately, for many starting out in NLP and even for some experienced practicioners, the theory and practical application of these powerful models is still not well understood.

"""
"""
## What is Bert ?

**BERT** (Bidirectional Encoder Representations from Transformers), released in late 2018, is the model we will use in this notebook to provide readers with a better understanding of and practical guidance for using transfer learning models in NLP. 

**BERT** is a method of pretraining language representations that was used to create models that NLP practicioners can then download and use for free. You can either use these models to extract high quality language features from your text data, or you can fine-tune these models on a specific task (classification, entity recognition, question answering, etc.) with your own data to produce state of the art predictions.

This notebook will explain how you can modify and fine-tune BERT to create a powerful NLP model that quickly gives you state of the art results. 

"""
"""

## Advantages of Fine-Tuning

"""
"""

I will use Bert to train a text classifier. Specifically, we will take the pre-trained Bert model, add an untrained layer of neurons on the end, and train the new model for our classification task. Why do this rather than train a train a specific deep learning model (a CNN, BiLSTM, etc.) that is well suited for the specific NLP task you need? 

1. **Quicker Development**

    * First, the pre-trained Bert model weights already encode a lot of information about our language. As a result, it takes much less time to train our fine-tuned model - it is as if we have already trained the bottom layers of our network extensively and only need to gently tune them while using their output as features for our classification task. In fact, the authors recommend only 2-4 epochs of training for fine-tuning Bert on a specific NLP task (compared to the hundreds of GPU hours needed to train the original Bert model or a LSTM from scratch!). 

2. **Less Data**

    * In addition and perhaps just as important, because of the pre-trained weights this method allows us to fine-tune our task on a much smaller dataset than would be required in a model that is built from scratch. A major drawback of NLP models built from scratch is that we often need a prohibitively large dataset in order to train our network to reasonable accuracy, meaning a lot of time and energy had to be put into dataset creation. By fine-tuning Bert, we are now able to get away with training a model to good performance on a much smaller amount of training data.

3. **Better Results**

    * Finally, this simple fine-tuning procedure (typically adding one fully-connected layer on top of Bert and training for a few epochs) was shown to achieve state of the art results with minimal task-specific adjustments for a wide variety of tasks: classification, language inference, semantic similarity, question answering, etc. Rather than implementing custom and sometimes-obscure architetures shown to work well on a specific task, simply fine-tuning BERT is shown to be a better (or at least equal) alternative.

"""
"""

### A Shift in NLP

This shift to transfer learning parallels the same shift that took place in computer vision a few years ago. 

Creating a good deep learning network for computer vision tasks can take millions of parameters and be very expensive to train. Researchers discovered that deep networks learn hierarchical feature representations (simple features like edges at the lowest layers with gradually more complex features at higher layers). Rather than training a new network from scratch each time, the lower layers of a trained network with generalized image features could be copied and transfered for use in another network with a different task. It soon became common practice to download a pre-trained deep network and quickly retrain it for the new task or add additional layers on top - vastly preferable to the expensive process of training a network from scratch. For many, the introduction of deep pre-trained language models in 2018 (ELMO, BERT, ULMFIT, Open-GPT, etc.) signals the same shift to transfer learning in NLP that computer vision saw.

Let's get started!
"""
import numpy as np # linear algebra
import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)

import os
for dirname, _, filenames in os.walk('../input'):
    for filename in filenames:
        print(os.path.join(dirname, filename))
import os
import pickle
from matplotlib import pyplot as plt
"""
Use of GPU
"""
%reload_ext autoreload
%autoreload 2
%matplotlib inline
import os
os.environ["CUDA_DEVICE_ORDER"]="PCI_BUS_ID";
os.environ["CUDA_VISIBLE_DEVICES"]="0";
"""
### Import ktrain

**[Ktrain](https://github.com/amaiya/ktrain)** is a lightweight wrapper for the deep learning library TensorFlow Keras (and other libraries) to help build, train, and deploy neural networks and other machine learning models. 

Inspired by ML framework extensions like fastai and ludwig, ktrain is designed to make deep learning and AI more accessible and easier to apply for both newcomers and experienced practitioners. With only a few lines of code, ktrain allows you to easily and quickly.
"""
!pip -q install ktrain
import ktrain
from ktrain import text
"""
# 2. Loading Defi IA Dataset

"""
"""
We'll use [Defi IA dataset for job classification](https://www.kaggle.com/c/defi-ia-insa-toulouse/data) dataset for single sentence classification. It's a set of sentences labeled as 28 jobs classes. The data has been retrieved from [CommonCrawl](https://www.wikiwand.com/en/Common_Crawl). The latter has been famously used to train OpenAI's GPT-3 model. The data is therefore representative of what can be found on the English speaking part of the Internet, and thus contains a certain amount of bias. One of the goals of this competition is to design a solution that is both accurate as well as fair, as explained in the Evaluation section.

"""
DATA_PATH = "../input/defi-ia-insa-toulouse"
train_df = pd.read_json(DATA_PATH+"/train.json").set_index('Id')
test_df = pd.read_json(DATA_PATH+"/test.json").set_index('Id')
train_label = pd.read_csv(DATA_PATH+"/train_label.csv").set_index('Id')
categories_string = pd.read_csv(DATA_PATH+"/categories_string.csv")
# template_submissions = pd.read_csv(DATA_PATH+"/template_submissions.csv").set_index('Id')
names = categories_string['0'].to_dict()
jobs = train_label['Category']
jobs = jobs.map(names)
jobs = jobs.rename('job')
genders = train_df["gender"]
people = pd.concat((jobs, genders), axis='columns')
people.head()
"""
### Split data
Split data into train and validation data
"""
train = people[["gender", "job"]].reset_index(   # need to keep the index as a column
        ).groupby(["gender", "job"]                  # split by "group"
        ).apply(lambda x: x.sample(500, replace=True) # in each group, do the random split
        ).reset_index(drop=True              # index now is group id - reset it
        ).set_index("Id")                 # reset the original index
val = people.drop(train.index)[["gender", "job"]].reset_index(   # need to keep the index as a column
        ).groupby(["gender", "job"]                  # split by "group"
        ).apply(lambda x: x.sample(50, replace=True) # in each group, do the random split
        ).reset_index(drop=True              # index now is group id - reset it
        ).set_index("Id")                 # reset the original index

train = train.sample(frac=1.)
train.head()
"""
### Remove duplicated

"""
print(train.shape, val.shape)
train = train[~train.index.duplicated(keep='first')]
val = val[~val.index.duplicated(keep='first')]
print(train.shape, val.shape)
train_df.head()
import warnings

warnings.filterwarnings("ignore")
train["text"] = train_df.loc[train.index]["description"].values
val["text"] = train_df.loc[val.index]["description"].values
train.head()
"""
### Load pre-trained model

We will use **distilbert-base-uncased**
"""
model_name = "distilbert-base-uncased"
t = text.Transformer(model_name, maxlen=125)
trn = t.preprocess_train(train["text"].values, train["job"].values)
vl = t.preprocess_test(val["text"].values, val["job"].values)
import gc

gc.collect()
"""
### Model and Learner
"""
model = t.get_classifier()
learner = ktrain.get_learner(model, train_data=trn, val_data=vl, batch_size=64)
"""
### Model summary
"""
model.summary()
"""
### Hyperparmeter learning rate

Simulating training for different learning rates... this may take a few moments...
"""
learner.lr_find(max_epochs=3, suggest=True, show_plot=True)
"""
### Suggestion

Suggest learning
"""
learner.lr_plot(suggest=True)
"""
### Fit one cylcle

Fit one cycle on 5 epochs
"""
learner.fit_onecycle(5e-5, 5)
"""
### Visualize fit results
"""
learner.history.history.keys()
learner.plot()
plt.plot(learner.history.history['accuracy'])
plt.plot(learner.history.history['val_accuracy'])
plt.title('Model Accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['train', 'validation'], loc='upper left')
plt.show()
"""
## Model Performance
"""
corr = learner.validate(class_names=t.get_classes())
"""
### View model top losses
"""
learner.view_top_losses(n=3, preproc=t)
print(val.loc[val.index[1223]]["text"])
print(val.loc[val.index[1431]]["text"])
predictor = ktrain.get_predictor(learner.model, preproc=t)
predictor.predict("She also teaches an after-hours Krav Maga class for interested Freelancers and she's loosely affiliated with the Russian mob. With her livelihood up in smoke, \
                  she wants answers. The Russians want answers, and it feels like not everyone is being completely honest.")
pd.DataFrame(predictor.predict_proba('After working for three months in a Google lab in the USA. He is now a cloud computing engineer at IBM.').reshape(1, -1),
            columns=np.array(predictor.get_classes()))
predictor.save("./output/fairness")
load_predictor = ktrain.load_predictor("./output/fairness")
"""
### reload the predictor
reloaded_predictor = ktrain.load_predictor('./output/predictor_ktrain')
"""
!pip install -q git+https://github.com/amaiya/eli5@tfkeras_0_10_1
import warnings

warnings.filterwarnings("ignore")
predictor.explain("She also teaches an after-hours Krav Maga class for interested Freelancers and she's loosely affiliated with the Russian mob. With her livelihood up in smoke, she wants answers. \
The Russians want answers, and it feels like not everyone is being completely honest.")
predictor.explain("He was also the editor of the Sunday Datebook for 14 years during a 34-year newspaper career at the San Francisco Chronicle.\
He has worked on “The Voice” at the Marsh Theater with David Ford, Ann Randolph and Mark Kenward for the last two years. \
                  The full piece will be present at the Marsh San Francisco on May 10.")
props = predictor.predict_proba(val["text"].values)
props.shape
def get_nd(arr):
    return np.array([arr==i for i in np.array(predictor.get_classes())])        
def reverse_job(df):
    return df.replace({j:i for i,j in names.items()})
preds = predictor.predict(val["text"].values)
preds[:5]
val["job_pred"] = preds
val.sample(20).head(10)
counts = val[['job_pred', 'gender']].groupby(['job_pred', 'gender']).size().unstack('gender')
counts['disparate_impact'] = counts[['M', 'F']].max(axis='columns') / counts[['M', 'F']].min(axis='columns')
counts
counts["disparate_impact"].mean()
counts = val[['job', 'gender']].groupby(['job', 'gender']).size().unstack('gender')
counts['disparate_impact'] = counts[['M', 'F']].max(axis='columns') / counts[['M', 'F']].min(axis='columns')
counts["disparate_impact"].mean()