"""
# <center> Personality Profile Prediction </center>
"""
"""
![](https://collegepublicspeakingresources.files.wordpress.com/2014/09/myersbriggs1.jpg)
"""
"""
# Table of contents
"""
"""
# <center> Classify a person's MBTI personality type using text from what they post online. </center><a id="index"></a>
"""
"""
[](http://)1. [Introduction](#1)
2. [Importing Packages & Data](#2)
3. [Text Preprocessing](#3)
4. [Text to Features ](#4)
5. [Text Classification](#5)
    * [Model Validation](#)
    * [Other Models Used](#)
    * [Final model](#)
    
6. [Submission](#6)
7. [Conclusion](#7)

"""
"""
<a id="1"></a>
"""
"""
# 1. Introduction
"""
"""
In this challenge, you will be required to build and train a model (or many models) capable of predicting a person's MBTI label using only what they post in online forums.

This challenge will require the use of Natural Language Processing to convert the data into machine learning format. This data will then be used to train a classifier capable of assigning MBTI labels to a person's online forum posts.

Each person will have only one of the two categories for each variable above. Combining the four variables gives the final personality type. For example, a person who is **Extraverted**, **Intuitive**, **Thinking** and **Judging** will get the ENTJ personality type.

You will need to build and train a model that is capable of predicting labels for each of the four MBTI variables - i.e. predict four separate labels for each person which, when combined, results in that person's personality type.
"""
"""
<a id="2"></a>
"""
"""
# 2. Importing Packages & Data
"""
"""
Importing all the necessary packages and input data thats going to be necessary for our predictions.
"""
# This Python 3 environment comes with many helpful analytics libraries installed
# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python
# For example, here's several helpful packages to load in 

import numpy as np # linear algebra
import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)
import seaborn as sns
import matplotlib.pyplot as plt
import re
import nltk
from wordcloud import WordCloud
from sklearn.metrics import accuracy_score, f1_score
from sklearn.metrics import precision_score, recall_score
from sklearn.linear_model import LogisticRegressionCV
from sklearn.multiclass import OneVsRestClassifier
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.feature_extraction.text import TfidfTransformer
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.model_selection import train_test_split
from sklearn import metrics
from sklearn.metrics import classification_report
import warnings
warnings.filterwarnings('ignore')

# Input data files are available in the "../input/" directory.
# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory

import os
print(os.listdir("../input"))

# Any results you write to the current directory are saved as output.
# import train and test set
train = pd.read_csv('../input/train.csv')
test = pd.read_csv('../input/test.csv')
train.head()
test.head()
"""
<a id="3"></a>
"""
"""
# 3. Text Preprocessing
"""
"""
Since, our text is in an unstructured form, various types of noise are present in it and the data is not readily analyzable without any pre-processing. In this section we go through the entire process of cleaning and standardization of text, making it noise-free and ready for analysis.
"""
# visualise the occurances/distribution of all the class types.
num_c = train['type'].value_counts()
f, ax = plt.subplots(figsize=(12,6))
sns.barplot( num_c.values, num_c.index, palette="Blues_d")
ax.xaxis.grid(False)
ax.set(xlabel="Number of Occurrences")
ax.set(ylabel="Personality Types")
ax.set(title="Personality Occurances")
sns.despine(trim=True, left=True, bottom=True)
# drop the  'Id' colum since it's unnecessary for the prediction process.
train_type = train[['type']]
test_ID = test['id']
train.drop(['type'], axis=1, inplace=True)
test.drop(['id'], axis=1, inplace=True)
# put all the features together to enable quick transformation process
post_features = pd.concat([train, test],sort=False).reset_index(drop=True)
post_features.head()
# using regular expressions for dealing with special patterns of noise.
pattern = r'http[s]?://(?:[A-Za-z]|[0-9]|[$-_@.&+]|[!*\(\),]|(?:%[0-9A-Fa-f][0-9A-Fa-f]))+'
post_features = post_features.replace(to_replace = pattern, value = ' ', regex = True)
post_features.head()
"""
Here we are going to go through the three steps of **Noise Removal**(Any piece of text which is not relevant to the context of the data and the end-output can be specified as the noise), **Lexicon Normalization**(Another type of textual noise is about the multiple representations exhibited by single word), **Object Standardization**(Text data often contains words or phrases which are not present in any standard lexical dictionaries. These pieces are not recognized by models). These will enable us to get our data into a format that we can work with.
"""
# creating a cleaned corpus
from nltk.stem.wordnet import WordNetLemmatizer 
from nltk.stem.porter import PorterStemmer
from nltk.corpus import stopwords

lem = WordNetLemmatizer() # returns the actual root word of a text token
# ps = PorterStemmer() # cuts off the surfix but doesnt necessarily return the root word
corpus =[]
# cleaning text
for i in range(0,len(post_features['posts'])):

    post = re.sub('[^a-zA-Z]', ' ',str(post_features.iloc[i].values)) 
    post = post.lower()
    post = post.split()
    # ps.stem(word) #MethodAttempt
    stop_wrds = set(stopwords.words('english'))
    post = [lem.lemmatize(word, "v") for word in post if not word in stop_wrds]
    post = ' '.join(post)
    corpus.append(post)

post_features['posts_clean'] = corpus
post_features.drop(['posts'], axis=1, inplace=True)              
post_features.head()
"""
<a id="4"></a>
"""
"""
# 4. Text to Features (Feature Engineering on text data)
"""
"""
To analyse a preprocessed data, it needs to be converted into features. Depending upon the usage, text features can be constructed using assorted techniques. Here we are going to use **Statistical features**. Here text data is quantified directly into numbers using several techniques implemented in this section. **Term Frequency – Inverse Document Frequency (TF – IDF**) and **CountVectorizer**.
"""
# Create Ranked Statistical Features with TfidfVectorizer, best choice because insignificant words recieve a lower rank
obj = TfidfVectorizer()
post_final_features = obj.fit_transform(post_features['posts_clean'])
print(post_final_features[0])

#AnotherMethodAttempt
# Create Ranked Statistical Features with CountVectorizer has no ranking machanism
# obj = CountVectorizer() 
# post_final_features = obj.fit_transform(post_features['posts_clean']).toarray()
# print(post_final_features[0:5])
# label extraction and creating our target variables from personality types
pd.options.mode.chained_assignment = None
yc = pd.DataFrame(train_type['type'])
train_type['I-E'] = train_type['type'].astype(str).str[0]
train_type['I-E'] = train_type['I-E'].map({"I": 0, "E": 1})
train_type['S-N'] = train_type['type'].astype(str).str[1]
train_type['S-N'] = train_type['S-N'].map({"S": 0, "N": 1})
train_type['F-T'] = train_type['type'].astype(str).str[2]
train_type['F-T'] = train_type['F-T'].map({"F": 0, "T": 1})
train_type['P-J'] = train_type['type'].astype(str).str[3]
train_type['P-J'] = train_type['P-J'].map({"P": 0, "J": 1})
train_type.drop('type', axis=1, inplace=True) 
y = train_type
print(y[0:5])
y = np.array(y)
print(y[0:5])
"""
Separating the data into a proper training(X,y) and the testing(X_final_test) set.
"""
# Spliting the data back to train(X,y) and test(X_sub)
X = post_final_features[:len(y), :]
X_final_test = post_final_features[len(y):, :]
print('Features size for train(X,y) and test(X_final_test):')
print('X', X.shape, 'y', y.shape, 'X_final_test', X_final_test.shape)
# wordcloud of the most frequently used words by each personality
yc['posts_clean'] = post_features['posts_clean']
labels = yc['type'].unique()
row, col = 5, 3
wc = WordCloud(stopwords = ['infj','entp','intp','intj', 'isfps','istps','isfjs','istjs',
                             'entjs','enfjs','infps','enfps','entj','enfj','infp','enfp',
                             'estps','esfps','estjs','esfjs','isfp','istp','isfj','istj',
                             'estp','esfp','estj','esfj','infjs','entps','intps','intjs'])
fig, ax = plt.subplots(5, 3, figsize=(20,15))
for i in range(5):
    for j in range(3):
        c_type = labels[i*col+j]
        c_ax = ax[i][j]
        df = yc[yc['type'] == c_type]
        wordc = wc.generate(df['posts_clean'].to_string())
        c_ax.imshow(wordc)
        c_ax.axis('off')
        c_ax.set_title(label=c_type,fontdict = {'fontsize': 20})
"""
<a id="5"></a>
"""
"""
# 5. Text Classification
"""
"""
Text classification, in common words is defined as a technique to systematically classify a text object (document or sentence) in one of the fixed category. In these case we have 8 categories(**(I-E)**, **(S-N)**, **(F-T)**, **(P-J)**) that we are classifiying. 
"""
"""
* **Model Bulding and Validation**
"""
# Predicting model
model = OneVsRestClassifier(LogisticRegressionCV(Cs=30, solver = 'saga',
                                                       multi_class = 'multinomial', cv=5), n_jobs =-1)
# split the train set to create a validation set
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.20)
# train model with 80% of train set
model.fit(X_train, y_train)
# predict 20% of train set
y_pred = model.predict(X_test)
# performance matrics and model eveluation using sklearn.metrics inbuilt classification metric
print(classification_report(y_test, y_pred, target_names=['Mind', 'Energy', 'Nature', 'Tactics']))
"""
*  **Other Models Used**
"""
"""
These models did not work very well compared to the LogisticRegressionCV. These was based on accuracy score and execution time. LogisticRegressionCV had a much better accuracy and with its build in cross validation method, helped to avoid overfitting.
"""

#AnotherMethodAttempt
# from sklearn.naive_bayes import MultinomialNB
# model = OneVsRestClassifier(MultinomialNB())
#AnotherMethodAttempt
# from sklearn.neighbors import KNeighborsClassifier
# model = OneVsRestClassifier(KNeighborsClassifier(n_neighbors=3))
"""
* **Final Model**
"""
"""
The hyperparameters where selected manually through trial and error. Other hyperparameter tuning models like GridSearchCV did not work due to multi-class classification nature of our model.
"""
# Reinitialise final model
model = OneVsRestClassifier(LogisticRegressionCV(Cs=30, solver = 'saga',
                                                 multi_class = 'multinomial', cv=10), n_jobs =-1)
# train final model with full train set 
model.fit(X, y)
# pridicting the actual test set (X_final_test)
y_predicted = model.predict(X_final_test)
# Final model results
y_predicted[0:5]
"""
Visualising Accuracy and Distribution of each class/category of our test data. This is the best alternative to visualise the model accuracy as we cannot do it with the usual accuracy matrics methods as they require the target variable y which we do not have for the test dataset.
"""
# Plot the accuracy for each classifier
f, ax = plt.subplots(figsize=(20, 10))
sns.set_color_codes(palette='deep')
plt.subplots_adjust(wspace = 0.5)
ax = plt.subplot(2, 4, 1)
plt.pie([sum(y_predicted[:,0]), len(y_predicted[:,0]) - sum(y_predicted[:,0])],
        labels = ['Extraverted', 'Introverted'],explode = (0, 0.1),autopct='%1.1f%%', colors=['y','orange'])
ax.set(title="Mind")

ax = plt.subplot(2, 4, 2)
plt.pie([sum(y_predicted[:,1]), len(y_predicted[:,1]) - sum(y_predicted[:,1])], 
        labels = ['Sensing', 'Intuitive'],explode = (0, 0.1),autopct='%1.1f%%', colors=['y','orange'])
ax.set(title="Energy")

f, ax2 = plt.subplots(figsize=(20, 10))
plt.subplots_adjust(wspace = 0.5)

ax2 = plt.subplot(2, 4, 1)
plt.pie([sum(y_predicted[:,2]), len(y_predicted[:,2]) - sum(y_predicted[:,2])], 
        labels = ['Thinking', 'Feeling'],explode = (0, 0.1),autopct='%1.1f%%', colors=['salmon','y'])
ax2.set(title="Nature")

ax2 = plt.subplot(2, 4, 2)
plt.pie([sum(y_predicted[:,3]),  len(y_predicted[:,3]) - sum(y_predicted[:,3])], 
        labels = ['Judging', 'Perceiving'], explode = (0, 0.1), autopct='%1.1f%%', colors=['y','salmon'])
ax2.set(title="Tactics")
plt.show()

"""
<a id="6"></a>
"""
"""
# 6. Submission
"""
"""
Submission of the predicted results from our test data. The predicted array is sliced into 4 categories(**mind** (I-E), **energy** (S-N), **nature** (F-T), **tactics** (P-J) to match the required prediction format.
"""
# format submission of the predicted classes
submission = pd.DataFrame({'id' : np.array(test_ID),'mind' : y_predicted[:,0], 
                           'energy' : y_predicted[:,1], 'nature' : y_predicted[:,2], 
                           'tactics' : y_predicted[:,3]})
print('Save submission')
# save DataFrame to csv file for submission
# submission.to_csv("new_submission.csv", index=False)
"""
# 7. Conclusion
"""
"""
<a id="7"></a>
"""
"""
From our analysis we can conclude that the most common personality type based on our test set is **ISFP** (Introverted Sensing Feeling Perceiving). These is based on comments that are obtained from the test set that does not necessarily represent the whole population of users because we have a limited dataset and its only those who participate in comments.
"""