%%capture                      
#delete the first line to see this cells output
import numpy as np
import pandas as pd 
import matplotlib.pyplot as plt
import matplotlib.gridspec as gridspec
%matplotlib inline
import seaborn as sns 
from scipy.sparse import csr_matrix, hstack
from scipy.stats import probplot
import re
from category_encoders import TargetEncoder
!pip install transformers
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestRegressor
from xgboost import XGBRegressor
from sklearn.metrics import mean_absolute_error
from sklearn.metrics import mean_squared_error
from sklearn.model_selection import GridSearchCV
from sklearn.model_selection import cross_val_score
import torch
import transformers as ppb
from transformers import DistilBertTokenizer
import warnings
import gc
import tensorflow as tf

#installig NVIDEA Apex for running Bert on GPU
! pip install -v --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" ../input/nvidiaapex/repository/NVIDIA-apex-39e153a

#assigning GPU to Apax package
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")


color = sns.color_palette()
sns.set_style("whitegrid")
sns.set_context("paper")
sns.palplot(color)

import os
for dirname, _, filenames in os.walk('/kaggle/input'):
    for filename in filenames:
        print(os.path.join(dirname, filename))
"""
# Table of Contents

1. [Introduction](#Introduction)
2. [Text embedding - DistilBERT](#Text-embedding---DistilBERT)
    - [ML models using CLS tokens](#ML-models-using-CLS-tokens)
    - [Predicting on test data for submission](#Predicting-on-test-data-for-submission)
3. [Header analysis](#Header-analysis)
    - [ML model using CLS tokens](#ML-model-using-CLS-tokens)
    - [Predicting on test data for submission](#Predicting-on-test-data-for-submission)
4. [Random-Forest model for numeric analysis](#Random-Forest-model-for-numeric-analysis)
5. [Ensembling of text, header and numeric predictions](#Ensembling-of-text,-header-and-numeric-predictions)
    - [Predictions on the test dataset](#Predictions-on-the-test-dataset)
6. [Conclusion](#Conclusion)
7. [Bibliography](#Bibliography)
"""
"""
# Introduction
This notebook constitutes an attempt to predict "claps" of articles from medium.com. Frameworks for Natural Language Processing (NLP) will be used, combined with Machine learning models and numeric data analysis will be added where possible. As an indicator of the prediction quality, the mean absolute error (MAE) or the mean squared error (MSE) will be used.

The first step will be data pre-processing and exploratory data analysis. This analysis should inform the decision on which NLP frameworks could be used and what numeric data from the train- and test-dataset can be combined. Both will be covered in the linked/submitted notebook (https://www.kaggle.com/benediktbnedikt/adams-dataprep). 

**The notebooks can either be executed locally, with the ADAMS-DataPrep/EDA running first and generating output that is taken as input for this notebook, or they can be viewed and executed on kaggle.com, for saving time and taking advantage of the provided hardware.**

First, we should take a look at medium.com. It is a publishing platform for all kinds of technological topics, from working with Raspberry Pis to AI frameworks. Publishing networks and independent authors can both be found on the platform and code can sometimes be part of an article. It can also be used for self-publishing. Because of the wide variety of topics and authors, it might be quite difficult to find meaningful categories and structures in the text data.  

This notebook will take the processed train and test data frame as input and therefore will be focused on implementing NLP and ML frameworks for predicting "claps".

Citations will be used where extended concepts are used and modified. Code and concepts shown in a package documentation will not be mentioned, when using a package one can assume, that the documentation was used as well. 
"""
train = pd.read_csv('../input/adams-dataprep-eda/train_processed.csv')
test = pd.read_csv('../input/adams-dataprep-eda/test_processed.csv')
Sample_submission = pd.read_csv('../input/adams-df/ADAMS_NLPtask_SS20/Sample_submission.csv')
test.info()
#log transform the target variable because it is scewed towards low values as shown in the EDA
train['totalClapCount'] = np.log1p(train['totalClapCount'])

#np.expm1() for reversing log transformation
#change to dtype string as tokenizers require
train['text'] = train['text'].astype(str)
train['title'] = train['title'].astype(str)
test['text'] = train['text'].astype(str)
test['Header'] = test['Header'].astype(str)

#validation dataset for every model
train_test = train.sample(5000, random_state = 45)
train = train.drop(labels = train_test.index, errors = 'ignore')

#for generating CLS-tokens from text variable
train_text = train.sample(10000, random_state = 22)

#for generating CLS-tokens from header variable
train_header = train.sample(20000, random_state = 777)
"""
# Text embedding - DistilBERT
Because the given task is, to use NLP for "clap" prediction, we will create a numeric representation of the articles using a pre-trained BERT model. 

The Bert model, published by Google in 2018, might currently be the most popular choice for language processing. It is using the concept of self-attention, which helps the model to combine the meaning of multiple words into one vector embedding, representing relationships between words more accurately [1]. 

The main problem is, that language processing models are usually designed for classification, question answering and text prediction tasks. Because we are trying to predict numeric values, we have to interpret the output with a different model like regressions.

When taking a closer look at the distribution of the target variable it becomes clear, that linear regression models do not fit this task. To achieve good results, on one hand, the model has to predict "normal" values between 0 and some hundred claps, on the other hand, it has to predict which articles are going "viral". Predictions for lower values could be performed with one regression model. However, also predicting values that are more than 100 times the standard deviation, cannot be performed by a singular regression model. To achieve a low mean squared error, it is likely more important to catch outliners, than to have really accurate predictions on low values. For a high MAE the outliners are not as important. 

The concept of using CLS-tokens from the Bert model for a regression task and the implementation is based on numerous informative articles by Jay Alammar, especially his notebook [2].

CLS, short for classification, meaning that those tokens are used to classify the meaning of a sentence while also representing the words used in a sentence. 

The text data will be pre-processed as required by Bert in the pre-processing notebook.


We are going to use DestilBert, which is a distilled down version of Bert. This is achieved by removing some weights in the neural network and decreasing the number of weights by approximating multiple weights to one, which is slightly less accurate but a lot faster and more memory efficient [3].
Because the *DestilBertTokenizer* and *DestilBertModel* calculate using vectors, the GPU should be used for encoding the text data. GPUs are designed for working with vectors, thus encoding can be done faster or using more text data. The Nvidia Apex module can be used for distributed processing like running the Bert model on the GPUs provided by Kaggle.

Because the DistilBERT model will run in a pyTorch environment on the GPU, a pyTorch implementation of DistilBERT could have been used, removing processing and conversion steps. 

The neural network of DestilBert will not be trained on the training data. It will be used to create a "machine-readable" representation of the texts provided. Those embeddings will be used for training different regression models. This should not be a problem because we are using a pre-trained model. Because of limitations of computing power and RAM we can not give all text data into the model at once. This could cause DestilBert to have some unknown words but the pre-trained vocabulary is so big, that this effect should be negligible. 
"""
def get_cls(max_length, text_source): 
    #pipeline for converting text data to CLS-tokens, concept and implementation partly from source [2]
    
    #initialize Bert tokenizer
    tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-uncased', do_lower_case=True, add_special_tokens=True,
                                                max_length=max_length, pad_to_max_length=True)
    #initialize DistilBertModel
    model = ppb.DistilBertModel.from_pretrained('distilbert-base-uncased')
    
    #save model on GPU
    model.to(device)
    
    
    ###tokenize, padding, attention_mask, CLS tokens, 
    encoded_dict = tokenizer.batch_encode_plus(
                    text_source,                       # Sentence to encode.
                    add_special_tokens = True,      # Add '[CLS]' and '[SEP]'
                    max_length = max_length,                # Pad & truncate all sentences.
                    padding=True,
                    pad_to_max_length=True,                     
                    return_attention_mask = True,   # Construct attn. masks.
                    return_tensors = 'pt',          # Return pytorch tensors.
                    )

    #put encoded_dict on GPU for faster/more memory efficient processing
    encoded_dict.to(device)
   

    #model = model.train()

    with torch.no_grad():
        last_hidden_states = model(encoded_dict.input_ids, attention_mask= encoded_dict.attention_mask)
    
    
    #read CLS-tokens from the last layer before the output layer
    features = last_hidden_states[0][:,0,:]
    
    
    #print("features extracted", flush=True)
    
    #free up memory to avoid crashing the notebook due to RAM overflow
    del last_hidden_states
    del model
    del encoded_dict
    gc.collect()
    return features
def append_cls_text(text, counter, factor):

    #cocatenate multiple bert outputs
    g = get_cls(450, train.text[counter : counter + 500]).cpu()
    counter += 500
    for i in range(factor):
        g = np.concatenate((g, get_cls(450, text[counter  : counter + 500]).cpu() ), 0)
        counter += 500
    return g
#generate cls tokens for training
fit_text = append_cls_text(train_text['text'], 0, 19)
#generate cls tokens for testing
predict_text = append_cls_text(train_test['text'], 0, 9)
torch.cuda.empty_cache()
gc.collect()
"""
# ML models using CLS tokens
Now we have prepared the text data for the DestilBertModell and used it to generate CLS tokens.

Two sets of data have been generated, one for training and one for testing. Because we have dropped all entries of the validation dataset from the training dataset, we don't have to be concerned about data overlap and train-test-splitting. 

This data will be analysed with a RandomForestRegressor and an XGBRegressor. Both ML models are chosen because they are easy to use and prior personal experience using both models. Different models likely can achieve a similar or maybe even better score. However the scores of both models no not deviate too much from each other, thus making high improvements with other models not as likely.  
"""
my_model = XGBRegressor(eval_metric= 'mae')
my_model.fit(fit_text, train_text['totalClapCount'], verbose=False)
text_preds = my_model.predict(predict_text)
print(mean_absolute_error(np.expm1(train_test['totalClapCount']), np.expm1(text_preds)))
print(mean_squared_error(np.expm1(train_test['totalClapCount']), np.expm1(text_preds)))
"""
The MAE reaching a fraction of the standard deviation is a first indicator of the model being somewhat task-specific. The high MSE indicates a low prediction accuracy on articles with a high clap count.

This throws the important question of training goals. The RF-R and the XGB-R can use different evaluation metrics and can therefore be trained to minimize either the MAE or the MSE. Using the MSE could help catching outliners, or rather articles that went "viral" and using the MAE can help to get most predictions accurate while accepting, that outliners will most likely not be detected. However, a significant difference could not be shown working with the text data.

It might prove helpful to combine both evaluation metrics, text data could be used to predict using the MAE and numeric values could be evaluated using the MSE. 
"""
forest_model = RandomForestRegressor(criterion='mse', random_state=9)
forest_model.fit(fit_text, train_text['totalClapCount'])
text_preds = forest_model.predict(predict_text)
    
print(mean_absolute_error(np.expm1(train_test['totalClapCount']), np.expm1(text_preds)))
print(np.expm1(train_test['totalClapCount']), np.expm1(text_preds))
print(mean_squared_error(np.expm1(train_test['totalClapCount']), np.expm1(text_preds)))
"""
Using a sample of 700 texts, the XGBRegressor predicts with a lower MAE than the RandomForestRegressor. Using a sample size of 10.000, the RF-R is predicting slightly better. Because a big part of parameter tuning can be seen as trial and error, this will not be the focus of this notebook. Also, improvements achieved using the small dataset can't be replicated using the bigger dataset. With the available computing power on Kaggle or Colab it is possible to do fine-tuning, the most efficient approach would be automating it and using between 20-60 iterations with each iteration taking around 10 minutes. Fine-tuning would take 2-4 hours per model, this would add up to 6-12 hours of parameter tuning for the three base models. This is possible but neither does it improve my understanding of NLP significantly, nor does it show my understanding of it.

In general, prediction accuracy that is better than the standard deviation of the target variable is not a bad sign for accuracy. When we try to predict outliners which are more than 100 times the standard deviation, exceptional results cannot be expected, though. There are also some other reasons, why prediction accuracy on the text data cannot be too accurate.

a) Using text data and NLP to predict likes is not the most sensible approach for a robust and accurate model. Language models can be used for classification tasks, sentiment analysis or text generation, not so much for numeric predictions.

b) The variety of the provided data is big, text data alone cannot be used to quantify the popularity. The same articles posted by one really popular publishing entity at 9 pm is likely going to be 1000 times more successful than the same articles posted by a self-publishing professional posted at 6 am.
"""
"""
# Predicting on test data for submission
"""
features_sub = get_cls(450, test['text'])


features_sub = features_sub.cpu()
features_sub = features_sub.numpy()

text_sub = my_model.predict(features_sub)

#map negative predictions to zero
#text_sub = text_sub.clip(min = 0)
"""
# Header analysis <a id="5"></a>
"""
def append_cls_header(text, counter, factor):

    #cocatenate multiple bert outputs:
    g = get_cls(100, train.text[counter : counter + 2500]).cpu()
    counter += 2500
    for i in range(factor):
        g = np.concatenate((g, get_cls(100, text[counter  : counter + 2500]).cpu() ), 0)
        counter += 2500
    return g
torch.cuda.empty_cache()
gc.collect()
#header predictions
fit_header = append_cls_header(train_header['title'], 0, 7)
predict_header = append_cls_header(train_test['title'], 0, 1)
"""
# ML model using CLS tokens
"""
my_model = XGBRegressor()
my_model.fit(fit_header, train_header['totalClapCount'], verbose=False)
header_preds = my_model.predict(predict_header)
print(mean_absolute_error(np.expm1(train_test['totalClapCount']), np.expm1(header_preds)))
print(mean_squared_error(np.expm1(train_test['totalClapCount']), np.expm1(header_preds)))
"""
Predictions using the header data achieve a similar accuracy as with the text data, this is a positive indicator for reproducibility of this approach. Again, the MAE is indicating an acceptable prediction quality, considering the distribution of the data. However, the MSE accentuates the low prediction accuracy on high clap counts, again. 
"""
"""
# Preparing test data for submission
"""
features_title = get_cls(100, test['Header'])
features_title = features_title.cpu()
features_title = features_title.numpy()

header_sub = my_model.predict(features_title)

#map negative values to zero
#header_sub = header_sub.clip(min = 0)
"""
# Random-Forest model for numeric analysis

The variables have been pre-processed in the linked notebook. Train and test data are in the same format. We still have to apply target encoding on training and testing datasets. Each author will be assigned the average clap count over their published articles.
"""
te = TargetEncoder()
X_target_encoded = te.fit(train['author'], train['totalClapCount'], handle_missing='return_nan', handle_unknown='return_nan')

test['author'] = X_target_encoded.transform(test['author'], y=None, override_return_df=False, )

test['author'].value_counts()

train['author'] = X_target_encoded.transform(train['author'], y=None, override_return_df=False,)
train_test['author'] = X_target_encoded.transform(train_test['author'], y=None, override_return_df=False,)
features_num = ['responsesCreatedCount', 'year_2017', 'year_2018', 'author']
labels_num = train['totalClapCount']
features_num_sub = ['Responses', 'year_2017', 'year_2018', 'author']
"""
After fitting a model with the MAE and one with the MSE as a target, we can see that both models do not deviate far from each other, due to faster computing, we will choose the MSE implementation. The MAE deviates by 0.5, the MSE by around 200, which is not much when looking at values of 280.000. 
"""
forest_model = RandomForestRegressor(criterion='mse')
forest_model.fit(train[features_num], labels_num)
num_preds = forest_model.predict(train_test[features_num])

#map negative predictins to zero
#num_preds = num_preds.clip(min = 0)
print(mean_absolute_error(np.expm1(train_test['totalClapCount']), np.expm1(num_preds)))
print(mean_squared_error(np.expm1(train_test['totalClapCount']), np.expm1(num_preds)))
#predicting on test features
forest_model.fit(train[features_num], train['totalClapCount'])

num_sub = forest_model.predict(test[features_num_sub])

#map negative predictions to zero
#num_sub = num_sub.clip(min = 0)
"""
# Ensembling of text, header and numeric predictions
Finally, we will combine the predictions from the text, the header and the numeric model by combining those in a pandas data frame and try to improve the prediction quality further. The biggest problem is the small size of the training data. 
"""
ensemble = [text_preds, header_preds, num_preds]
ensemble_features = pd.DataFrame(data = ensemble)
ensemble_features = ensemble_features.transpose()
train_features, test_features, train_labels, test_labels = train_test_split(ensemble_features, train_test['totalClapCount'], random_state = 16)
    
forest_model = RandomForestRegressor(n_estimators=110, criterion='mae', max_depth=None, min_samples_split=2, min_samples_leaf=1)
forest_model.fit(train_features, train_labels)
ensemble_preds = forest_model.predict(test_features)
print(mean_absolute_error(np.expm1(test_labels), np.expm1(ensemble_preds)))
print(mean_squared_error(np.expm1(test_labels), np.expm1(ensemble_preds)))
ensemble_preds = np.expm1(ensemble_preds)
#mapping all values below 50 is an attempt to fit the predictions to the distribution ot the target variable 

#clap = []
#for i in ensemble_preds:
#    if i < 50:
#        clap.append(0)
#    else:
#        clap.append(i)  

#print(mean_absolute_error(np.expm1(test_labels), clap))
"""
The mapping did not show improvements of the MAE. It will not be used for the final submission. Also the ensembling is predicting a slightly higher MAE than the numeric model itself. For further improvement other ensembling methods could be tested.

Now we train the RF-R on the biggest available dataset after showing the prediction accuracy. This trained model will then be applied to the test dataset to make the final predictions for the submission. Lets's see if we still get negative values. 
"""
"""
# Making predictions on the test dataset
Now we will combine the predictions of the text, header and numeric value models for a final prediction. We can combine the three lists to a pandas data frame and use the columns as input features for a RF-Regressor. We are using the predictions and features from the *train* dataset to train/fit the model and then use the predictions from the *test* dataset to make final predictions for the submission

The test/header/numeric predictions contain some negative values. However, negative values don't cause the ensembling model to predict negative values. If we map the negative values to zero, the accuracy decreases from an MAE of 81 to an MAE of 108. However, this was only proven using a rather small sample size. 
"""
forest_model = RandomForestRegressor()
forest_model.fit(ensemble_features, train_test['totalClapCount'])
ensemble = [text_sub, header_sub, num_sub]
ensemble_features = pd.DataFrame(data = ensemble)
ensemble_features = ensemble_features.transpose()
ensemble_features.info()
ensemble_features.columns = ['text_sub', 'header_sub', 'num_sub' ]
ensemble_features
ensemble_preds = forest_model.predict(ensemble_features)
ensemble_preds = np.expm1(ensemble_preds)
#round to integer because claps cannot be float
ensemble_preds = [round(x) for x in ensemble_preds]
"""
# CSV file for submission
"""
Sample_submission['Claps'] = ensemble_preds
Sample_submission.drop(labels = ['index'], axis = 1)
Sample_submission.to_csv('Sample_submission.csv',index=False)
"""
# Conclusion
In general, we were able to achieve a good but not outstanding accuracy when looking at the MAE as the prediction parameter. The numeric model was the only one capable of catching very high values, thus achieving a lower MSE than the text models. This would also be the strongest criticism of this implementation, that especially the NLP model is not designed and not suitable for catching those outliners. Training the implemented models using MSE as a target would still improve the prediction quality of high values. 

Also, the improvement from the numeric baseline to the ensemble is so small, that it might not justify the necessary computing power for the NLP model, this would depend strongly on the real-world use case.

Because this is the first time I am working with real-world data, the first time working with language data and my experience with full neural networks is limited, I decided on this ML-heavy implementation. Due to this inexperience with working with big, real-world datasets, the approach of working with two notebooks was chosen, this could have easily been done using one notebook. 

For future work, a model with BERT for sequence classification in combination with the mapped target variable would be an interesting approach. 

It is not likely, that an implementation with the full BERT model would increase the accuracy because we are not training the model and the CLS tokens of the texts should not deviate much or not at all.

This notebook is missing a cross-fold validation, which is especially necessary when using target encoding. Also, the positive influence of the target encoding is likely going to be smaller predicting on the *test* dataset because the author variable is less accurate here. This will make the predictions made by the NLP model more relevant, though. 
"""
"""
# Bibliography

**ADAMS_Submission:**

[1] Raimi Karim. Towards Data Science. Illustrated: Self-Attention. Retrieved 20.8.2020 from https://towardsdatascience.com/illustrated-self-attention-2d627e33b20a

[2] Jay Alammar. A Visual Notebook to Using BERT for the First Time. Retrieved 20.8.2020 from https://colab.research.google.com/github/jalammar/jalammar.github.io/blob/master/notebooks/bert/A_Visual_Notebook_to_Using_BERT_for_the_First_Time.ipynb#scrollTo=q1InADgf5xm2

[3] Victor Sanh. Hugging Face. Smaller, faster, cheaper, lighter: Introducing DistilBERT, a distilled version of BERT. Retrieved 20.8.2020 from https://medium.com/huggingface/distilbert-8cf3380435b5)






**ADAMS_DataPrep/EDA:**

[1] Daniel Jeffries. Hackernoon (medium.com). The Cryptocurrency Trading Bible. Retrieved 20.8.2020 from https://medium.com/hackernoon/the-cryptocurrency-trading-bible-43d0c57e3fe6

[2]  Alex Martelli. Stackoverflow. How to make a flat list out of list of lists?. Retrieved 20.8.2020 from https://stackoverflow.com/questions/952914/how-to-make-a-flat-list-out-of-list-of-lists

[3] "Ωmega". Stackoverflow. How to remove any URL within a string in Python. Retrieved 20.8.2020 from https://stackoverflow.com/questions/11331982/how-to-remove-any-url-within-a-string-in-python
"""