"""
![](https://www.themavencircle.com/wp-content/uploads/2016/03/Neuro-linguistic-Programming-650x435.jpg)
"""
"""
<a id="0"></a> <br>
## Kernel Headlines
1. [Introduction](#1)
    1.  [Concept of Text Mining](#2)
    2.  [Text Mining Importance and Difficulties](#3)
    3.  [Text Mining In Buisiness](#4)
    4.  [Dealing With Unstructured Data](#5)
    5.  [Dealing With Dirty Space](#6)
    6.  [Why Representation is Complex ?!](#7)
    7. [Wath Does BagOfWords Means?!](#8)
    8. [Ignoring Grammer and NLP-Based Features](#9)
    9. [Feature Extraction Using BOW](#10)
    10. [Term Frequency (TF)](#11)
    11. [Inverse Term Frequency (IDF)](#12)
    12. [Simple Example (TF-IDF)](#13)
    13. [Preprocessing In Traditional Manners](#14)
2. [Embedding Methods](#15)
     1. [What Does Embedding Means?!](#16)
     2. [What Does N-Gram Means?!](#17)
     3. [Main Disadvantages of N-Gram](#18)
     4. [What Does Embeddings Provide For Text Mining](#19)
3. [Take a Glance to Data](#20)
     1. [WordCloud](#21)
     2. [Target Distribution](#22)
4. [Start Vectorizing by Simple CountVectorizer](#23)
    1. [Make Your Hands Dirty;-) ](#24)
    2. [Some Suggestions About Text Vectorizing ](#25)
    3. [Classification](#26)
        4. [Naive Bayes](#27)
        5. [RandomForest](#28)
        6. [MLP](#29)
        7. [LogicRegression](#30)
        8. [Classifiers Accuracy Comparisons on CountVectorizer](#31)
        9. [Curse of Dimentionality](#32)
5. [Embedding Methods_Training Our Word2Vec Model](#33)
	1. [Introduction](#34)
	2. [Python Tip. Using Generators To Avoid Memory Error](#35)
	3. [Texts and Vectors Relations](#36)
	4. [Training Using Pandas Piplines](#37)
	5. [Classifier Accuracy Comparison On Our Word2Vec Model](#38)
6. [Embedding Methods_Glove](#39)
	1. [Introduction](#40)
	2. [Be careful about our method](#41)
	3. [GLOVE initiation](#42)
7. [Embedding Methods_Program_300Vectors](#43)
8. [Embedding Methods_GoogleNewsVectors](#44)
9. [Embedding Methods_Wiki_Vectors](#45)
10. [Conclusion on Traditonal Methods](#46)
11. [Moving to DeepMoldes](#47)
    1. [Introduction and Roadmap](#48)
	2. [Prepairing Data for Using in Keras](#49)
12. [References](#100)
"""
"""
<a id="1"></a> <br>
#  1-Introduction

In this competition we are dealing with text processing and embedding vectors. First of all, lets recap some basic and fundamental concepts.
<a id="2"></a> <br>
* **A. Concept of Text Mining**

In principle, text is just another form of data, and text processing is just a special case of representation engineering. In reality, dealing with text requires dedicated pre-processing steps and sometimes specific expertise on the part of the data science In this introduction we can only scratch the surface, to give a basic overview of the techniques and issues involved in typical business applications. First, let’s discuss why text is so important and why it’s difficult.
<a id="3"></a> <br>
* **B. Text Mining Importance and Difficulties**

Exploiting this vast amount of data requires converting it to a meaningful form. The Internet may be the home of “new media,” but much of it is the same form as old media. It contains a vast amount of text in the form of personal web pages, Quera, Twitter feeds, email, Facebook status updates, product descriptions, Reddit comments, blog postings—the list goes on. Underlying the search engines (Google and Bing) that we use everyday are massive amounts of text-oriented data science. Indeed, the thrust of Web 2.0 was about Internet sites allowing users to interact with one another as a community, and to generate much added content of a site. This user-generated content and interaction usually takes the form of text.

<a id="4"></a> <br>
* **C. TEXT MINING IN BUSINESS**

In businesses such as our case study (Quera), understanding customer feedback often requires understanding text. This isn’t always the case; admittedly, some important consumer attitudes are represented explicitly as data or can be inferred through behavior, for example via five-star ratings, click-through patterns, conversion rates, and so on. We can also pay to have data collected and quantified through focus groups and online surveys. But in many cases if we want to “listen to the customer” we’ll actually have to read what she’s written—in product reviews, customer feedback forms, opinion pieces, and email messages.
<a id="5"></a> <br>
* **D. DEALING WITH UNSTRUCTURED DATA**

Why Text Is Difficult Text is often referred to as “unstructured” data. This refers to the fact that text does not have the sort of structure that we normally expect for data: tables of records with fields having fixed meanings (essentially, collections of feature vectors), as well as links between the tables. Words can have varying lengths and text fields can have varying numbers of words.
<a id="6"></a> <br>
* **E. DEALING WITH DIRTY SPACE**

Sometimes word order matters, sometimes not. As data, text is relatively dirty .People write ungrammatically, they misspell words, they run words together, they abbreviate unpredictably, and punctuate  randomly. Even when text is flawlessly expressed it may contain synonyms (multiple words with the same meaning) and homographs (one spelling shared among multiple words with different meanings).
<a id="7"></a> <br>
* **F. WHY REPRESENTATION IS COMPLEX !**

Representation Having discussed how difficult text can be, let’s go through the basic steps to transform a body of text into a set of data that can be fed into a data mining algorithm. The general strategy in text mining is to use the simplest (least expensive) or small. A document is composed of individual tokens or terms. For now, think of a token or term as just a word; as we go on we’ll show how they can be different from what are customarily thought of as words. A collection of documents is called a corpus.
<a id="8"></a> <br>
* **G. WATH DOES BAG OF WORDS MAENS?!**

Bag of Words It is important to keep in mind the purpose of the text representation task. In essence, we are taking a set of documents—each of which is a relatively free-form sequence of words—and turning it into our familiar feature-vector form. Each document is one instance but we don’t know in advance what the features will be. The approach we introduce first is called “bag of words.” As the name implies, the approach is to treat every document as just a collection of individual words.
<a id="9"></a> <br>
* **H. IGNORING GRAMMER AND NLP-BASED FEATURES**

BOW approach ignores grammar, word order, sentence structure, and (usually) punctuation. It treats every word in a document as a potentially important keyword of the document. The representation is  straightforward and inexpensive to generate, and tends to work well for many tasks.
<a id="10"></a> <br>
* **I. FEATURE EXTRACTION USING BOW**

So if every word is a possible feature, what will be the feature’s value in a given document? There are several approaches to this. In the most basic approach, each word is a token, and each document is represented by a one (if the token is present in the document) or a zero (the token is not present in the document). This approach simply reduces a document to the set of words contained in it.
<a id="11"></a> <br>
* **J. TERM-FREQUENCY (TF)**

The next step up is to use the word count (frequency) in the document instead of just a zero or one. This allows us to differentiate between how many times a word is used; in some applications, the importance of a term in a document should increase with the number of times that term occurs. This is called the term frequency representation. It could simply defined by the following equation: 
*TF(t) = (Number of times term t appears in a document) / (Total number of terms in the document).*
<a id="12"></a> <br>
* **K. INVERSE DOCUMENT FREQUENCY (IDF)**

which measures how important a term is. While computing TF, all terms are considered equally important. However it is known that certain terms, such as "is", "of", and "that", may appear a lot of times but have little importance. Thus we need to weigh down the frequent terms while scale up the rare ones, by computing the following: 
*IDF(t) = log_e(Total number of documents / Number of documents with term t in it).*
idf is on the best methods for Measuring Sparseness
<a id="13"></a> <br>
* **L. Example FROM (tfidf.com)**

Consider a document containing 100 words wherein the word cat appears 3 times. The term frequency (i.e., tf) for cat is then (3 / 100) = 0.03.
Now, assume we have 10 million documents and the word cat appears in one thousand of these. Then, the inverse document frequency (i.e., idf) is
calculated as log(10,000,000 / 1,000) = 4. Thus, the Tf-idf weight is the product of these quantities: 0.03 * 4 = 0.12.
<a id="14"></a> <br>
* **M. PRE-PROCESSING WHICH ARE CONVENTIONAL (SPECIALLY FOR TRADITIONAL MANNERS).**

First, the case has been normalized: every term is in lowercase. This is so that words like Skype and SKYPE are counted as the same thing. Second, many words have been stemmed : their suffixes removed, so that verbs like announces , announced and announcing are all reduced to the term announce. Finally, stopwords have been removed. A stopword is a very common word in English (or whatever language is being parsed). The words the , and , of , and on are considered stopwords in English so they are typically removed.

<a id="15"></a> <br>
#  2-EMBEDDING METHODS
<a id="16"></a> <br>
* **A. WHAT DOES EMBEDDING MEANS ?!**

An embedding is a relatively low-dimensional space into which you can translate high-dimensional vectors. Embeddings make it easier to do machine learning on large inputs like sparse vectors representing words. Ideally, an embedding captures some of the semantics of the input by placing semantically similar inputs close together in the embedding space. An embedding can be learned and reused across models.
<a id="17"></a> <br>
* **B. What Does N-GRAM Means?!**

As presented, the bag-of-words representation treats every individual word as a term, discarding word order entirely. In some cases, word order is important and you want to preserve some information about it in the representation. A next step up in complexity is to include sequences of adjacent words as terms. For example, we could include pairs of adjacent words so that if a document contained the sentence “The quick brown fox jumps.” it would be transformed into the set of its constitutent words { quick , brown , fox , jumps }, plus the tokens quick_brown , brown_fox , and fox_jumps . This general representation tactic is called n-grams . Adjacent pairs are commonly called bi-grams. If you hear a data scientist mention representing text as “bag of n-grams up to three” it simply means she’s representing each document using as
features its individual words, adjacent word pairs, and adjacent word triples. N-grams are useful when particular phrases are significant but their component words may not be. In a business news story, the appearance of the tri-gram exceed_analyst_expectation is more meaningful than simply knowing that the individual words analyst , expectation , and exceed appeared somewhere in a story. An advantage of using n-grams is that they are easy to generate; they require no linguistic knowledge or complex parsing algorithm.
<a id="18"></a> <br>
* **C. MAIN DISADVANTAGE OF N-GRAM**

The main disadvantage of n-grams is that they greatly increase the size of the feature set. There are far more word pairs than individual words, and still more word triples. The number of features generated can quickly get out of hand. Data mining using n-grams almost always needs some special consideration for dealing with massive numbers of features, such as a feature selection stage or special consideration to computational storage space. 
<a id="19"></a> <br>
* **D. WHAT DOES EMBEDDING PROVIDE FOR US IN TEXT MINING ?!**

In the most simplest way, we can say embedding means replacing each word of our corpus with the correspondence feature space that pre-trained embedding models have provided. Some organizations have trained a deep learning models on popular datasets such as wikipedia. If you want to know how they have train the models check [this url](https://radimrehurek.com/gensim/tut1.html#from-strings-to-vectors). Training a deep learning model needs process and time. So, we cant do it by our personal laptops. Fortunately, some of these pretrained embeddings (i.e models which are existed in data directory of this competition) are published with opensource licences. We need only convert our text to the target feature spaces and do a classification task with wide range of classifier we have known.

<a id="20"></a> <br>
#  3-Take a Glance To Data

Ok. lets do simple WordCloud for easily getting start ... 
"""
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from wordcloud import WordCloud
import warnings
import collections
import sklearn as sklearn
from sklearn.feature_extraction.text import CountVectorizer, TfidfTransformer,TfidfVectorizer
from sklearn.model_selection import cross_val_score, KFold, train_test_split
from sklearn.feature_extraction import text 
from sklearn.naive_bayes import MultinomialNB
from sklearn.ensemble import RandomForestClassifier,VotingClassifier
from sklearn.neural_network import MLPClassifier
from sklearn.linear_model import LogisticRegression
from sklearn import svm, tree
from sklearn.svm import LinearSVC
from sklearn.metrics import accuracy_score
import gensim
from collections import defaultdict
from itertools import islice
from sklearn.ensemble import ExtraTreesClassifier
from sklearn.pipeline import Pipeline
from sklearn.naive_bayes import GaussianNB
from keras import Sequential
from keras.layers import Bidirectional, GlobalMaxPool1D,Dense, Input, Embedding, Dropout, LSTM, CuDNNGRU
from keras.models import Model
from keras.preprocessing.sequence import pad_sequences
from keras.preprocessing.text import Tokenizer


import time
%matplotlib inline
warnings.filterwarnings('ignore')
train_df = pd.DataFrame.from_csv("../input/train.csv")
test_df =  pd.DataFrame.from_csv("../input/test.csv")
train_df.head(5)
"""
<a id="21"></a> <br>
**A. TEST WORDCLOUD**
"""
text =" ".join(train_df.question_text)
 # Create the wordcloud object
wordcloud = WordCloud(width=1024, height=1024, margin=0).generate(text)
 
# Display the generated image:
fig,ax = plt.subplots(1,1,figsize=(10,10))
ax.imshow(wordcloud, interpolation='bilinear')
ax.axis("off")
ax.margins(x=0, y=0)
plt.show()
"""
<a id="22"></a> <br>
**B. TARGET DISTRIBUTION**
"""
fig, ax = plt.subplots(1,1, figsize=(8,8))
ax.set_title("Target Status")
explode=(0,0.1)
labels ='0','1'
ax.pie(list(dict(collections.Counter(list(train_df.target))).values()), explode=explode, labels=labels, autopct='%1.1f%%',shadow=True, startangle=90)
del text
"""
<a id="23"></a> <br>
#  4-Start Vectorizing by Simple CountVectorizer
"""
"""
<a id="24"></a> <br>
**A. MAKE YOUR HANDS DIRTY ;-) **

Starting with CountVectorizer.

As we mentioned in introduction, for converting string to feature spaces, we need do some transforms on string. As a result of this sort of transonsformations, the string will be converted to vectors space. One of teh most simplest ways for vectorizing the text and converting it to a formal dataset (which could be passed to classification algorithms) is countVectorizer. Simply, the count of each word will be calculated. In the second step, by using TermFrequency transformations, for any word and for any document, the relative proportions will be form a formal dataset.

Lets start with a countVectorizer...
"""
X_train, X_test, y_train, y_test = train_test_split(train_df.question_text, train_df.target, test_size=0.33, random_state=42)
#initiation of countVectorizer
count_vectorizer = CountVectorizer(min_df=5, stop_words='english')
vect = count_vectorizer.fit(train_df.question_text)

X_vect_train = vect.transform(X_train) # documents-terms matrix of training set
X_vect_test = vect.transform(X_test) # documents-terms matrix of testing set

tf_train_transformer = TfidfTransformer(use_idf=False).fit(X_vect_train)
tf_test_transformer =  TfidfTransformer(use_idf=False).fit(X_vect_test)

xtrain_tf = tf_train_transformer.transform(X_vect_train)
xtest_tf = tf_test_transformer.transform(X_vect_test)
type(xtrain_tf),xtrain_tf.shape, train_df.shape
"""
Because of high dimentional problems we have encountered with, by default it returns the matrix in sparse mode. For representing in normal feature space, you can use todense method.
"""
xtrain_tf[:5].todense()
"""
Any of these rows, reveals the vector of each document. The columns are also correspondence to words existed in the dataset. You can access the feature names by 'get_feature_names()' method.
Lets look around some sections of features.
"""
count_vectorizer.get_feature_names()[0:5]
count_vectorizer.get_feature_names()[30000:30005]
count_vectorizer.get_feature_names()[-5:]
"""
There are also non-english featuers. These sort of features can reduce the accuracy.
"""
[{k:count_vectorizer.vocabulary_[k]} for k in list(count_vectorizer.vocabulary_)[:10]]
list(sklearn.feature_extraction.text.ENGLISH_STOP_WORDS)[:10]
"""
<a id="25"></a> <br>
**B. SOME SUGGESTIONS ABOUT TEXT VECTORIZING  **

Ok. Good.

Lets review some facts about text processing in this special case.

As it is obvious you can see there are various range of features (from foreign language to astonishing digits and numbers) in feature space.
We can do some modifications on the count_vectorizer. Some preprocessing could be done are:

* Cleaning meaningless words from dataset.

* Using threshold for repetition of words. for example discarding the words which are not repeated in whole records. it  could be setted by min_df in count_vectorizer

* Defining threshold for minimum repetition of words. for example any word that has repetitioin fewer than 10 times, should be ignored.

* Removing Stopwords. Stop words are common words that have low information for classification because they exist on any sentences. For example, words such as 'the','is', 'by' and this sort of words, have belonged to stop_words list. Removing them from dataset can increase the performance accuracy. Count_vectorizer supports using stop words by defatult. You can use it by CountVectorizer(stop_words=LIST_OF_STOP_WORDS). It is wise to see improvements in classifier performance by adding effect of stop_words to it.
"""
"""
Now, We have the dataset which is ready for classification.
Lets do simple classifications for starting classification on first vectorized space (count vectorize).
"""
"""
<a id="26"></a> <br>
**C. CLASSIFICATION**
"""
results_df = pd.DataFrame()
"""
<a id="27"></a> <br>
**D. Multinominal NB**
"""
# MULTINOMINA_NAIVE_BAYES
nb_ = MultinomialNB()
nb_clf = nb_.fit(X=xtrain_tf, y=y_train)
results_df.set_value("NB" , "countVectorizer" , accuracy_score(y_test,nb_clf.predict(xtest_tf)))
"""
<a id="28"></a> <br>
**E. RF**
"""
# RANDOM_FORES_CLASSFIER
rf_clf = RandomForestClassifier(n_estimators=25, max_depth=15,random_state=42)
rf_clf.fit(X=xtrain_tf,y=y_train)
results_df.set_value("RF" , "countVectorizer" , accuracy_score(y_test,rf_clf.predict(xtest_tf)))
"""
<a id="29"></a> <br>
**F. MLP**
"""
#MLP_CLASSIFIER
mlp_clf = MLPClassifier(solver='lbfgs', alpha=1e-4,hidden_layer_sizes=(20,10, 2), random_state=42)
mlp_clf.fit(X=xtrain_tf, y=y_train)                         
results_df.set_value("MLP" , "countVectorizer" , accuracy_score(y_test,mlp_clf.predict(xtest_tf)))
"""
<a id="30"></a> <br>
**G. LoggicReg**
"""
#LoggicRegression
lreg_clf = LogisticRegression(solver='lbfgs', multi_class='multinomial',random_state=42)
lreg_clf.fit(X=xtrain_tf, y=y_train)                         
results_df.set_value("LREG" , "countVectorizer" , accuracy_score(y_test,lreg_clf.predict(xtest_tf)))
"""
<a id="31"></a> <br>
**H. CLASSIFIERS ACCURACY COMPARISON ON COUNT_VECTORIZER**

Congratulation ! 

You have tested large variety of classifiers.
Lets visualize the result of tested classifiers.
"""
results_df
fig,axes=plt.subplots(1,1,figsize=(8,8))
axes.set_ylabel("Accuracy")
plt.ylim((.92,.97))
results_df.plot(kind="bar",ax=axes)
"""
Don't forget we have used limit number of layers and neurons. Results absolutely could be better by tuning parameters in more precise way.

RandomForest also could revealed better results if it trains with more decision trees.

**Simply, we can improve accuracy of these classifiers with more supervised training.**
"""
"""
<a id="32"></a> <br>
**I. CURSE OF DIMENTIONALITY**


Look again at the dimension of train data. It has 49650 features !!! Dealing with such a huge feature space is really challenging.
If you do classification (similar to what we have done in previous part) you will get MemoryError. 
One of the most challenging factors in data mining is dimensionality. Read about curse of dimensionality [here](https://en.wikipedia.org/wiki/Curse_of_dimensionality) if you want to know more about it.

For our example we have 49650 features. Absolutely classifiers such as DecisionTree, KNeighborsClassifier or SVM will return MemoryError.  

"""
"""
<a id="33"></a> <br>
#  5-EMBEDDING METHODS_WORD2VEC

<a id="34"></a> <br>
**A. INTRODUCTION**

Now you know how to deal with texts. At first step, we need to extract features from text; Then there is a need to investigate classifiers and tune them for getting better accuracy.  In previous sections you trained a classifier based on the count vectorizer. Now, lets move on to newer vectorizing method. The name is Word2Vec. As its name is simply representing, by using this manner any word will be replaced by a correspondence vectors. In this manner, training parameters are playing important role. lets try word2vec on our dataset to understand what does it means.

**As these processes are time consuming, I have simulated these classifiers on my local machine. Committing these codes (In each commit waiting until finishing whole the  classifiers was really annoying for me) so I have commented the lines which are representing training process. You can easily uncomment them and try to  train classifier by yourself ;-). **

<a id="35"></a> <br>
**B. PYTHON TIP. USE GENERATORS TO AVOID MEMORY ERROR**

To train a model on text, we need pass following steps: 
 Reading text and keeping in memory,
 Tokenizing the text
 Iterating on specified window
 
 In use cases such as texts (especially large texts) keeping whole the text in memory is memory consumable. Using generators, can reduce the overhead of
 iterating. Python generators are a simple way of creating iterators. All the overhead we mentioned above are automatically handled by generators in
 Python. Simply speaking, a generator is a function that returns an object (iterator) which we can iterate over (one value at a time).

"""
text_ = train_df.question_text
targets_ = train_df.target
class GetSentences(object):
    def __iter__(self):
        counter = 0
        for sentence_iter in text_:
            tmp_sentence = sentence_iter
            counter += 1
            yield tmp_sentence.split()
len(text_)
"""
Training Word2Vec. 
We will ignore words which have repetition less than 5 times and scrolling default window size equal to 5. Changing these parameters can change our model accuracy. We will check it in next steps.
"""
num_features = 200  # Word vector dimensionality
min_word_count = 5  # Minimum word count
num_workers = 4  # Number of threads to run in parallel
context = 10  # Context window size
downsampling = 1e-3  # Downsample setting for frequent words
get_sentence = GetSentences()
model = gensim.models.Word2Vec(sentences=get_sentence, min_count=min_word_count, size=num_features, workers=4)
w2v = dict(zip(model.wv.index2word, model.wv.syn0))
next(iter(w2v.items()))
list(islice(model.wv.vocab.items(), 5))
"""
As you can see, we have a vector for word 'the'. It means we can replace 'the' with correspondence vector. Lets do some interesting calculations with our word2vec model ;-)
"""
"""
<a id="36"></a> <br>
**C. TEXTS and VECTORS RELATIONS**
"""
"""
    **-. MOST SIMILAR WORDS TO ... **
"""
"MOST SIMILAR WORD TO MAN: {} AND MOST SIMILAR WORD TO QUERA:{} ".format(model.most_similar(['man'],topn=1),model.most_similar(["quora"],topn=3))
model.most_similar(positive=["united","state"])
"""
Or in more related way to competition challenges...
"""
# most similar words to 'sex'
model.most_similar(positive=["sex"])
model.most_similar(positive=['gangbang', 'sex'], negative=['man'], topn=5) 
"""
    **-.  [ X+Y ]  - [ Z ] ... **
"""
model.most_similar(positive=['woman', 'king'], negative=['man'], topn=2) 
"""
As you can see there is vector for each word. The better trained models the higher accuracy we can get. Now you can understand what does the embedding models can do for us.
"""
"""
    **-. SAVING THE MODEL **
"""
model.save('word2vec.model')
"""
<a id="37"></a> <br>
**D. TRAINING USING PANDAS PIPLINES**
"""
"""
In this section we will use pipline facilities in training the data.

About the pipelines there are two points should be mentioned. 

From the documentation:
1. Pipelines can Sequentially apply a list of transforms and a final estimator. Intermediate steps of the pipeline must be ‘transforms’, that is, they must implement fit and transform methods. The final estimator only needs to implement fit.
2. The purpose of the pipeline is to assemble several steps that can be cross-validated together while setting different parameters. 

"""
"""
Until now, we have vector for each word. How we can convert this vectors to formal dataset which could be passed to traning algorithms ?!

Simply manner could be mean of vectors in any dataset record. Another manner could be the same tfidf transformers. 

Lets implement them for using in our pipeline.

Lets split Train and Test for training the classfieirs.
"""
class MeanEmbeddingVectorizer(object):
    def __init__(self, word2vec):
        self.word2vec = word2vec
        # if a text is empty we should return a vector of zeros
        # with the same dimensionality as all the other vectors
        # self.dim = len(word2vec.itervalues().next())
        self.dim = len(next(iter(self.word2vec.items()))[1])

    def fit(self, X, y):
        return self

    def transform(self, X):
        return np.array([np.mean([self.word2vec[w] for w in words if w in self.word2vec] or [np.zeros(self.dim)], axis=0) for words in X])


class TfidfEmbeddingVectorizer(object):
    def __init__(self, word2vec):
        self.word2vec = word2vec
        self.word2weight = None
        # self.dim = len(word2vec.itervalues().next())
        self.dim = len(next(iter((word2vec.items()))))

    def fit(self, X, y):
        tfidf = TfidfVectorizer(analyzer=lambda x: x)
        tfidf.fit(X)
        # if a word was never seen - it must be at least as infrequent
        # as any of the known words - so the default idf is the max of
        # known idf's
        max_idf = max(tfidf.idf_)
        self.word2weight = defaultdict(
            lambda: max_idf,
            [(w, tfidf.idf_[i]) for w, i in tfidf.vocabulary_.items()])

        return self

    def transform(self, X):
        return np.array([
            np.mean([self.word2vec[w] * self.word2weight[w]
                     for w in words if w in self.word2vec] or
                    [np.zeros(self.dim)], axis=0)
            for words in X
        ])


"""
NOTE:
As these processes are time consuming, I have simulated these classifiers on my local machine. Committing these codes (In each commit waiting until finishing whole the 
classifiers was really annoying for me) so I have commented the lines which are representing training process. You can easily uncomment them and try to
train classifier by yourself ;-).
"""

print("EXTRA TREE ...")
# 1- MEAN VECTORIZER
etree_w2v = Pipeline([("w2v mean vectorizer", MeanEmbeddingVectorizer(w2v)), ("extra trees", ExtraTreesClassifier(n_estimators=25))])
# etree_w2v.fit(X=X_train, y=y_train)
# results_df.set_value("ExtraTree", "w2v_mean", accuracy_score(y_test, etree_w2v.predict(X_test)))

# 2- TFIDF VECTORIZER
etree_w2v_tfidf = Pipeline([("w2v tfidf vectorizer", TfidfEmbeddingVectorizer(w2v)), ("extra trees", ExtraTreesClassifier(n_estimators=25))])
# etree_w2v_tfidf.fit(X=X_train, y=y_train)
# results_df.set_value("ExtraTree", "w2v_tfidf", accuracy_score(y_test, etree_w2v_tfidf.predict(X_test)))

####SVM####
print("SVM ... ")
# 1- MAIN VECTORIZER
svm_w2v = Pipeline([("w2v mean vectorizer", MeanEmbeddingVectorizer(w2v)), ("SVM", LinearSVC(random_state=0, tol=1e-4))])
# svm_w2v.fit(X=X_train, y=y_train)
# results_df.set_value("SVM", "w2v_mean", accuracy_score(y_test, etree_w2v_tfidf.predict(X_test)))

# 2- TFIDF VECTORIZER
svm_w2v_tfidf = Pipeline([("word2vec vectorizer", TfidfEmbeddingVectorizer(w2v)), ("SVM", LinearSVC(random_state=0, tol=1e-4))])
# svm_w2v_tfidf.fit(X=X_train, y=y_train)
# results_df.set_value("SVM", "w2v_tfidf", accuracy_score(y_test, svm_w2v_tfidf.predict(X_test)))

####MLP####
print("MLP ... ")
# 1- MAIN VECTORIZER
mlp_w2v = Pipeline(
    [("w2v mean vectorizer", MeanEmbeddingVectorizer(w2v)),
     ("MLP", MLPClassifier(solver='lbfgs', alpha=1e-5, hidden_layer_sizes=(20,10, 2), random_state=1))])
# mlp_w2v.fit(X=X_train, y=y_train)
# results_df.set_value("MLP", "w2v_mean", accuracy_score(y_test, mlp_w2v.predict(X_test)))

# 2- TFIDF VECTORIZER
mlp_w2v_tfidf = Pipeline(
    [("word2vec vectorizer", TfidfEmbeddingVectorizer(w2v)),
     ("MLP", MLPClassifier(solver='lbfgs', alpha=1e-5, hidden_layer_sizes=(20,10,2), random_state=1))])
# mlp_w2v_tfidf.fit(X=X_train, y=y_train)
# results_df.set_value("MLP", "w2v_tfidf", accuracy_score(y_test, mlp_w2v_tfidf.predict(X_test)))
results_df.set_value("NB","w2v_mean",0.476575)
results_df.set_value("ExtraTree","w2v_mean",0.939119)
results_df.set_value("SVM","w2v_mean",0.939144)
results_df.set_value("MLP","w2v_mean",0.939035)
results_df.set_value("LREG","w2v_mean",0.938836)

results_df.set_value("NB","w2v_tfidf",0.260479)
results_df.set_value("ExtraTree","w2v_tfidf",0.939144)
results_df.set_value("SVM","w2v_tfidf",0.939033)
results_df.set_value("MLP","w2v_tfidf",0.939035)
results_df.set_value("LREG","w2v_tfidf",0.953311)
"""
<a id="38"></a> <br>
**E. CLASSIFIERS ACCURACY COMPARISON ON OUR WORD2VEC MODEL**

"""
results_df
fig,axes=plt.subplots(1,1,figsize=(8,8))
axes.set_ylabel("Accuracy")
axes.set_title("word2vec results for 67% training and 23% testing")
# plt.ylim((.93,.95))
results_df[["w2v_mean","w2v_tfidf"]].dropna().plot(kind="bar",ax=axes)
del w2v
del model
import gc; gc.collect()
time.sleep(10)
"""
<a id="39"></a> <br>
#  6-EMBEDDING METHODS_GLOVE

<a id="40"></a> <br>
**A. INTRODUCTION**

In prevous section we represented the results of own word2vec model. Now, kets do a classification with glov embedded models.
In the most simplest way, we can say for every embedding models we have a dictionary containing key (word) and values (list of double values which are correspondence to relation to other words which are exist on that dataset).


<a id="41"></a> <br>
**B. BE CAREFUL ABOUT OUR MANNER.**

**The point should be mentioned is that because of time and memory limitations, we have limit our training data to 67 percent of samples. The most reason for the low accuracies and a gap which is existed in most of the classifiers is related to type of our selection. In most of the classifiers, we also limit the parameters. for example in ExtraTree we use only 25 classifiers which is absolutely low. Or in the neural network MLP we have used two layer perceptron model which is approximately unsatisfied model. By the way, lets continue the process. We will try to extract all the information which is needed and use them in final step.**

"""
"""
<a id="42"></a> <br>
**C. GLOVE INITIATIONS.**

"""
EMBEDDING_FILE = '../input/embeddings/glove.840B.300d/glove.840B.300d.txt'
def get_coefs(word,*arr): return word, np.asarray(arr, dtype='float32')
glov = dict(get_coefs(*o.split(" ")) for o in open(EMBEDDING_FILE))
next(iter(glov.values()))
"""
As you can see, the process is completely similar to the previous section.
"""
"""
NOTE:
As these processes are time consuming, I have simulated them on my local machine. Committing these codes (Waiting until finishing whole the trainin process for
any commit really bothers me) so I have commented the lines which have belonged to training process. You can easily uncomment them and try to
train classifier by yourself ;-).
"""
# GLOVE MODEL

###NB####
print("Multinomina NB ...")
nb_glov = Pipeline([("glov mean vectorizer", MeanEmbeddingVectorizer(glov)), ("Guassian NB", GaussianNB())])
# nb_glov.fit(X=X_train, y=y_train)
# results_df.set_value("GB", "glov_mean", accuracy_score(y_test, nb_glov.predict(X_test)))

# 2- TFIDF VECTORIZER
nb_glov_tfidf = Pipeline([("glov tfidf vectorizer", TfidfVectorizer(glov)), ("transform", TfidfTransformer()), ("Guassian NB", MultinomialNB())])
# nb_glov_tfidf.fit(X=X_train, y=y_train)
# results_df.set_value("GB", "glov_tfidf", accuracy_score(y_test, nb_glov_tfidf.predict(X_test)))

###EXTRA TREE####
print("EXTRA TREE ...")
# 1- MEAN VECTORIZER
etree_glov = Pipeline([("glov mean vectorizer", MeanEmbeddingVectorizer(glov)), ("extra trees", ExtraTreesClassifier(n_estimators=25))])
# etree_glov.fit(X=X_train, y=y_train)
# results_df.set_value("ExtraTree", "glov_mean", accuracy_score(y_test, etree_glov.predict(X_test)))

# 2- TFIDF VECTORIZER
etree_glov_tfidf = Pipeline(
    [("glov tfidf vectorizer", TfidfVectorizer(glov)), ("transform", TfidfTransformer()), ("extra trees", ExtraTreesClassifier(n_estimators=25))])
# etree_glov_tfidf.fit(X=X_train, y=y_train)
# results_df.set_value("ExtraTree", "glov_tfidf", accuracy_score(y_test, etree_glov_tfidf.predict(X_test)))

####SVM####
print("SVM ... ")
# 1- MAIN VECTORIZER
svm_glov = Pipeline([("glov mean vectorizer", MeanEmbeddingVectorizer(glov)), ("SVM", LinearSVC(random_state=42, tol=1e-5))])
# svm_glov.fit(X=X_train, y=y_train)
# results_df.set_value("SVM", "glov_mean", accuracy_score(y_test, svm_glov.predict(X_test)))

# 2- TFIDF VECTORIZER
svm_glov_tfidf = Pipeline(
    [("glov tfidf vectorizer", TfidfVectorizer(glov)), ("transform", TfidfTransformer()), ("SVM", LinearSVC(random_state=0, tol=1e-5))])
# svm_glov_tfidf.fit(X=X_train, y=y_train)
# results_df.set_value("SVM", "glov_tfidf", accuracy_score(y_test, svm_glov_tfidf.predict(X_test)))

####MLP####
print("MLP ... ")
# 1- MAIN VECTORIZER
mlp_glov = Pipeline(
    [("glov mean vectorizer", MeanEmbeddingVectorizer(glov)),
     ("MLP", MLPClassifier(solver='lbfgs', alpha=1e-5, hidden_layer_sizes=(20, 10, 2), random_state=42))])
# mlp_glov.fit(X=X_train, y=y_train)
# results_df.set_value("MLP", "glov_mean", accuracy_score(y_test, mlp_glov.predict(X_test)))

# 2- TFIDF VECTORIZER
mlp_glov_tfidf = Pipeline(
    [("glov tfidf vectorizer", TfidfVectorizer(glov)), ("transform", TfidfTransformer()),
     ("MLP", MLPClassifier(solver='lbfgs', alpha=1e-5, hidden_layer_sizes=(20, 10, 2), random_state=42))])
# mlp_glov_tfidf.fit(X=X_train, y=y_train)
# results_df.set_value("MLP", "glov_tfidf", accuracy_score(y_test, mlp_glov_tfidf.predict(X_test)))


####LREG####
print("LREG ... ")
# 1- MAIN VECTORIZER
lreg_glov = Pipeline(
    [("glov mean vectorizer", MeanEmbeddingVectorizer(glov)),
     ("LREG", LogisticRegression(solver='lbfgs', multi_class='multinomial', random_state=42))])
# lreg_glov.fit(X=X_train, y=y_train)
# results_df.set_value("LREG", "glov_mean", accuracy_score(y_test, lreg_glov.predict(X_test)))

# 2- TFIDF VECTORIZER
lreg_glov_tfidf = Pipeline(
    [("glov tfidf vectorizer", TfidfVectorizer(glov)), ("transform", TfidfTransformer()),
     ("LREG", LogisticRegression(solver='lbfgs', multi_class='multinomial', random_state=42))])
# lreg_glov_tfidf.fit(X=X_train, y=y_train)
# results_df.set_value("LEREG", "glov_tfidf", accuracy_score(y_test, lreg_glov_tfidf.predict(X_test)))
results_df.set_value("NB","glov_mean",0.533364)
results_df.set_value("ExtraTree","glov_mean",0.939131)
results_df.set_value("SVM","glov_mean",0.939140)
results_df.set_value("MLP","glov_mean",0.938973)
results_df.set_value("LREG","glov_mean",0.938836)

results_df.set_value("NB","glov_tfidf",0.941720)
results_df.set_value("ExtraTree","glov_tfidf",0.945798)
results_df.set_value("SVM","glov_tfidf",0.953854)
results_df.set_value("MLP","glov_tfidf",0.939035)
results_df.set_value("LREG","glov_tfidf",0.953311)
# fig,axes=plt.subplots(1,1,figsize=(15,8))
# axes.set_ylabel("Accuracy")
# axes.set_title("GLOVE results for 67% training and 23% testing")
# results_df.plot(kind="bar",ax=axes)
del glov
import gc; gc.collect()
time.sleep(10)
"""
<a id="43"></a> <br>
#  7-EMBEDDING METHODS_PROGRAM_300
"""
EMBEDDING_FILE = '../input/embeddings/paragram_300_sl999/paragram_300_sl999.txt'
def get_coefs(word,*arr): return word, np.asarray(arr, dtype='float32')
program = dict(get_coefs(*o.split(" ")) for o in open(EMBEDDING_FILE,encoding="latin1"))
next(iter(program.values()))
"""
NOTE:
As these processes are time consuming, I have simulated them on my local machine. Committing these codes (Waiting until finishing whole the trainin process for
any commit really bothers me) so I have commented the lines which have belonged to training process. You can easily uncomment them and try to
train classifier by yourself ;-).
"""

print("Multinomina NB ...")
nb_program = Pipeline([("program mean vectorizer", MeanEmbeddingVectorizer(program)), ("Guassian NB", GaussianNB())])
# nb_program.fit(X=X_train, y=y_train)
# results_df.set_value("GB", "program_mean", accuracy_score(y_test, nb_program.predict(X_test)))

# 2- TFIDF VECTORIZER
nb_program_tfidf = Pipeline([("program tfidf vectorizer", TfidfVectorizer(program)), ("transform", TfidfTransformer()), ("Guassian NB", MultinomialNB())])
# nb_program_tfidf.fit(X=X_train, y=y_train)
# results_df.set_value("GB", "program_tfidf", accuracy_score(y_test, nb_program_tfidf.predict(X_test)))


# PROGRAM-300 MODEL
####EXTRA TREE####
print("EXTRA TREE ...")
# 1- MEAN VECTORIZER
etree_program = Pipeline([("program mean vectorizer", MeanEmbeddingVectorizer(program)), ("extra trees", ExtraTreesClassifier(n_estimators=20))])
# etree_program.fit(X=X_train, y=y_train)
# results_df.set_value("ExtraTree", "program_mean", accuracy_score(y_test, etree_program.predict(X_test)))


# 2- TFIDF VECTORIZER
etree_program_tfidf = Pipeline([("program tfidf vectorizer", TfidfEmbeddingVectorizer(program)), ("extra trees", ExtraTreesClassifier(n_estimators=20))])
# etree_program_tfidf.fit(X=X_train, y=y_train)
# results_df.set_value("ExtraTree", "program_tfidf", accuracy_score(y_test, etree_program_tfidf.predict(X_test)))

####SVM####
print("SVM ... ")
#1- MAIN VECTORIZER
svm_program = Pipeline([("program mean vectorizer", MeanEmbeddingVectorizer(program)), ("SVM", LinearSVC(random_state=0, tol=1e-5))])
# svm_program.fit(X=X_train, y=y_train)
# results_df.set_value("SVM", "program_mean", accuracy_score(y_test, svm_program.predict(X_test)))

# 2- TFIDF VECTORIZER
svm_program_tfidf = Pipeline([("word2vec vectorizer", TfidfEmbeddingVectorizer(program)), ("SVM", LinearSVC(random_state=0, tol=1e-5))])
# svm_program_tfidf.fit(X=X_train, y=y_train)
# results_df.set_value("SVM", "program_tfidf", accuracy_score(y_test, svm_program_tfidf.predict(X_test)))

####MLP####
print("MLP ... ")
# 1- MAIN VECTORIZER
mlp_program = Pipeline(
    [("program mean vectorizer", MeanEmbeddingVectorizer(program)),
     ("MLP", MLPClassifier(solver='lbfgs', alpha=1e-5, hidden_layer_sizes=(20,10, 2), random_state=42))])
# mlp_program.fit(X=X_train, y=y_train)
# results_df.set_value("MLP", "program_mean", accuracy_score(y_test, mlp_program.predict(X_test)))

# 2- TFIDF VECTORIZER
mlp_program_tfidf = Pipeline(
    [("word2vec vectorizer", TfidfEmbeddingVectorizer(program)),
     ("MLP", MLPClassifier(solver='lbfgs', alpha=1e-5, hidden_layer_sizes=(20,10, 2), random_state=42))])
# mlp_program_tfidf.fit(X=X_train, y=y_train)
# results_df.set_value("MLP", "program_tfidf", accuracy_score(y_test, mlp_program_tfidf.predict(X_test)))

####LREG####
print("LREG ... ")
# 1- MAIN VECTORIZER
lreg_program = Pipeline(
    [("program mean vectorizer", MeanEmbeddingVectorizer(program)),
     ("LREG", LogisticRegression(solver='lbfgs', multi_class='multinomial', random_state=42))])
# lreg_program.fit(X=X_train, y=y_train)
# results_df.set_value("LREG", "program_mean", accuracy_score(y_test, lreg_program.predict(X_test)))

# 2- TFIDF VECTORIZER
lreg_program_tfidf = Pipeline(
    [("program tfidf vectorizer", TfidfVectorizer(program)), ("transform", TfidfTransformer()),
     ("LREG", LogisticRegression(solver='lbfgs', multi_class='multinomial', random_state=42))])
# lreg_program_tfidf.fit(X=X_train, y=y_train)
# results_df.set_value("LREG", "program_tfidf", accuracy_score(y_test, lreg_program_tfidf.predict(X_test)))
results_df.set_value("NB","program_mean",0.559782)
results_df.set_value("ExtraTree","program_mean",0.939124)
results_df.set_value("SVM","program_mean",0.939026)
results_df.set_value("MLP","program_mean",0.939035)
results_df.set_value("LREG","program_mean",0.938989)

results_df.set_value("NB","program_tfidf",0.941720)
results_df.set_value("ExtraTree","program_tfidf",0.945717)
results_df.set_value("SVM","program_tfidf",0.953854)
results_df.set_value("MLP","program_tfidf",0.939035)
results_df.set_value("LREG","program_tfidf",0.953311)
# fig,axes=plt.subplots(1,1,figsize=(15,8))
# axes.set_ylabel("Accuracy")
# axes.set_title("PROGRAM results for 67% training and 23% testing")
# results_df.plot(kind="bar",ax=axes)
del program
import gc; gc.collect()
time.sleep(10)
"""
<a id="44"></a> <br>
#  8-EMBEDDING METHODS_GOOGLE_NEWS_VECTORS
"""
model = gensim.models.KeyedVectors.load_word2vec_format(
    fname='../input/embeddings/GoogleNews-vectors-negative300/GoogleNews-vectors-negative300.bin',
    binary=True)
google = dict(zip(model.wv.index2word, model.wv.syn0))
next(iter(google.values()))
"""
NOTE:
As these processes are time consuming, I have simulated them on my local machine. Committing these codes (Waiting until finishing whole the trainin process for
any commit really bothers me) so I have commented the lines which have belonged to training process. You can easily uncomment them and try to
train classifier by yourself ;-).
"""



print("Multinomina NB ...")
google_google = Pipeline([("google mean vectorizer", MeanEmbeddingVectorizer(google)), ("Guassian NB", GaussianNB())])
# google_google.fit(X=X_train, y=y_train)
# results_df.set_value("GB", "google_mean", accuracy_score(y_test, google_google.predict(X_test)))

# 2- TFIDF VECTORIZER
google_google_tfidf = Pipeline([("google tfidf vectorizer", TfidfVectorizer(google)), ("transform", TfidfTransformer()), ("Guassian NB", MultinomialNB())])
# google_google_tfidf.fit(X=X_train, y=y_train)
# results_df.set_value("GB", "google_tfidf", accuracy_score(y_test, google_google_tfidf.predict(X_test)))


# GOOGLE_NEWS_VEC MODEL
####EXTRA TREE####
print("EXTRA TREE ...")
# 1- MEAN VECTORIZER
etree_google = Pipeline([("google mean vectorizer", MeanEmbeddingVectorizer(google)), ("extra trees", ExtraTreesClassifier(n_estimators=20))])
# etree_google.fit(X=X_train, y=y_train)
# results_df.set_value("ExtraTree", "google_mean", accuracy_score(y_test, etree_google.predict(X_test)))

# 2- TFIDF VECTORIZER
etree_google_tfidf = Pipeline([("google tfidf vectorizer", TfidfEmbeddingVectorizer(google)), ("extra trees", ExtraTreesClassifier(n_estimators=20))])
# etree_google_tfidf.fit(X=X_train, y=y_train)
# results_df.set_value("ExtraTree", "google_tfidf", accuracy_score(y_test, etree_google_tfidf.predict(X_test)))

####SVM####
print("SVM ... ")
#1- MAIN VECTORIZER
svm_google = Pipeline([("google mean vectorizer", MeanEmbeddingVectorizer(google)), ("SVM", LinearSVC(random_state=0, tol=1e-5))])
# svm_google.fit(X=X_train, y=y_train)
# results_df.set_value("SVM", "google_mean", accuracy_score(y_test, svm_google.predict(X_test)))

# 2- TFIDF VECTORIZER
svm_google_tfidf = Pipeline([("word2vec vectorizer", TfidfEmbeddingVectorizer(google)), ("SVM", LinearSVC(random_state=0, tol=1e-5))])
# svm_google_tfidf.fit(X=X_train, y=y_train)
# results_df.set_value("SVM", "google_tfidf", accuracy_score(y_test, svm_google_tfidf.predict(X_test)))

####MLP####
print("MLP ... ")
# 1- MAIN VECTORIZER
mlp_google = Pipeline(
    [("google mean vectorizer", MeanEmbeddingVectorizer(google)),
     ("MLP", MLPClassifier(solver='lbfgs', alpha=1e-5, hidden_layer_sizes=(5, 2), random_state=1))])
# mlp_google.fit(X=X_train, y=y_train)
# results_df.set_value("MLP", "google_mean", accuracy_score(y_test, mlp_google.predict(X_test)))

# 2- TFIDF VECTORIZER
mlp_google_tfidf = Pipeline(
    [("word2vec vectorizer", TfidfEmbeddingVectorizer(google)),
     ("MLP", MLPClassifier(solver='lbfgs', alpha=1e-5, hidden_layer_sizes=(5, 2), random_state=1))])
# mlp_google_tfidf.fit(X=X_train, y=y_train)
# results_df.set_value("MLP", "google_tfidf", accuracy_score(y_test, mlp_google_tfidf.predict(X_test)))

####LREG####
print("LREG ... ")
# 1- MAIN VECTORIZER
lreg_google = Pipeline(
    [("google mean vectorizer", MeanEmbeddingVectorizer(google)),
     ("LREG", LogisticRegression(solver='lbfgs', multi_class='multinomial', random_state=42))])
# lreg_google.fit(X=X_train, y=y_train)
# results_df.set_value("LREG", "google_mean", accuracy_score(y_test, lreg_google.predict(X_test)))

# 2- TFIDF VECTORIZER
lreg_google_tfidf = Pipeline(
    [("google tfidf vectorizer", TfidfVectorizer(google)), ("transform", TfidfTransformer()),
     ("LREG", LogisticRegression(solver='lbfgs', multi_class='multinomial', random_state=42))])
# lreg_google_tfidf.fit(X=X_train, y=y_train)
# results_df.set_value("LREG", "google_tfidf", accuracy_score(y_test, lreg_google_tfidf.predict(X_test)))
results_df.set_value("NB","google_mean",0.527487)
results_df.set_value("ExtraTree","google_mean",0.939126)
results_df.set_value("SVM","google_mean",0.939038)
results_df.set_value("MLP","google_mean",0.939035)
results_df.set_value("LREG","google_mean",0.938806)

results_df.set_value("NB","google_tfidf",0.941720)
results_df.set_value("ExtraTree","google_tfidf",0.945448)
results_df.set_value("SVM","google_tfidf",0.953854)
results_df.set_value("MLP","google_tfidf",0.939035)
results_df.set_value("LREG","google_tfidf",0.953311)
# fig,axes=plt.subplots(1,1,figsize=(15,8))
# axes.set_ylabel("Accuracy")
# axes.set_title("GOOGLE_NEWS results for 67% training and 23% testing")
# results_df.plot(kind="bar",ax=axes)
del google
del model
import gc; gc.collect()
time.sleep(10)
"""
<a id="45"></a> <br>
#  9-EMBEDDING METHODS_WIKI_NEWS
"""
model = gensim.models.KeyedVectors.load_word2vec_format(fname='../input/embeddings/wiki-news-300d-1M/wiki-news-300d-1M.vec',encoding='utf-8')
wiki = dict(zip(model.wv.index2word, model.wv.syn0))
next(iter(wiki.values()))
"""
NOTE:
As these processes are time consuming, I have simulated them on my local machine. Committing these codes (Waiting until finishing whole the trainin process for
any commit really bothers me) so I have commented the lines which have belonged to training process. You can easily uncomment them and try to
train classifier by yourself ;-).
"""


print("Multinomina NB ...")
wiki_wiki = Pipeline([("wiki mean vectorizer", MeanEmbeddingVectorizer(wiki)), ("Guassian NB", GaussianNB())])
# wiki_wiki.fit(X=X_train, y=y_train)
# results_df.set_value("GB", "wiki_mean", accuracy_score(y_test, wiki_wiki.predict(X_test)))

# 2- TFIDF VECTORIZER
wiki_wiki_tfidf = Pipeline([("wiki tfidf vectorizer", TfidfVectorizer(wiki)), ("transform", TfidfTransformer()), ("Guassian NB", MultinomialNB())])
# wiki_wiki_tfidf.fit(X=X_train, y=y_train)
# results_df.set_value("GB", "wiki_tfidf", accuracy_score(y_test, wiki_wiki_tfidf.predict(X_test)))

####EXTRA TREE####
print("EXTRA TREE ...")
# 1- MEAN VECTORIZER
etree_wiki = Pipeline([("wiki mean vectorizer", MeanEmbeddingVectorizer(wiki)), ("extra trees", ExtraTreesClassifier(n_estimators=20))])
# etree_wiki.fit(X=X_train, y=y_train)
# results_df.set_value("ExtraTree", "wiki_mean", accuracy_score(y_test, etree_wiki.predict(X_test)))

# 2- TFIDF VECTORIZER
etree_wiki_tfidf = Pipeline([("wiki tfidf vectorizer", TfidfVectorizer(wiki)), ("transform", TfidfTransformer())
                             , ("extra trees", ExtraTreesClassifier(n_estimators=20))])
# etree_wiki_tfidf.fit(X=X_train, y=y_train)
# results_df.set_value("ExtraTree", "wiki_tfidf", accuracy_score(y_test, etree_wiki_tfidf.predict(X_test)))

####SVM####
print("SVM ... ")
#1- MAIN VECTORIZER
svm_wiki = Pipeline([("wiki mean vectorizer", MeanEmbeddingVectorizer(wiki)), ("SVM", LinearSVC(random_state=0, tol=1e-5))])
# svm_wiki.fit(X=X_train, y=y_train)
# results_df.set_value("SVM", "wiki_mean", accuracy_score(y_test, svm_wiki.predict(X_test)))

# 2- TFIDF VECTORIZER
svm_wiki_tfidf = Pipeline([("wiki tfidf vectorizer", TfidfVectorizer(wiki)), ("transform", TfidfTransformer()),
                           ("SVM", LinearSVC(random_state=0, tol=1e-5))])
# svm_wiki_tfidf.fit(X=X_train, y=y_train)
# results_df.set_value("SVM", "wiki_tfidf", accuracy_score(y_test, svm_wiki_tfidf.predict(X_test)))

####MLP####
print("MLP ... ")
# 1- MAIN VECTORIZER
mlp_wiki = Pipeline(
    [("wiki mean vectorizer", MeanEmbeddingVectorizer(wiki)),
     ("MLP", MLPClassifier(solver='lbfgs', alpha=1e-5, hidden_layer_sizes=(5, 2), random_state=1))])
# mlp_wiki.fit(X=X_train, y=y_train)
# results_df.set_value("MLP", "wiki_mean", accuracy_score(y_test, mlp_wiki.predict(X_test)))

# 2- TFIDF VECTORIZER
mlp_wiki_tfidf = Pipeline(
    [("wiki tfidf vectorizer", TfidfVectorizer(wiki)), ("transform", TfidfTransformer()),
     ("MLP", MLPClassifier(solver='lbfgs', alpha=1e-5, hidden_layer_sizes=(5, 2), random_state=1))])
# mlp_wiki_tfidf.fit(X=X_train, y=y_train)
# results_df.set_value("MLP", "wiki_tfidf", accuracy_score(y_test, mlp_wiki_tfidf.predict(X_test)))

####LREG####
print("LREG ... ")
# 1- MAIN VECTORIZER
lreg_wiki = Pipeline(
    [("wiki mean vectorizer", MeanEmbeddingVectorizer(wiki)),
     ("LREG", LogisticRegression(solver='lbfgs', multi_class='multinomial', random_state=42))])
# lreg_wiki.fit(X=X_train, y=y_train)
# results_df.set_value("LREG", "wiki_mean", accuracy_score(y_test, lreg_wiki.predict(X_test)))

# 2- TFIDF VECTORIZER
lreg_wiki_tfidf = Pipeline(
    [("wiki tfidf vectorizer", TfidfVectorizer(wiki)), ("transform", TfidfTransformer()),
     ("LREG", LogisticRegression(solver='lbfgs', multi_class='multinomial', random_state=42))])
# lreg_wiki_tfidf.fit(X=X_train, y=y_train)
# results_df.set_value("LREG", "wiki_tfidf", accuracy_score(y_test, lreg_wiki_tfidf.predict(X_test)))
results_df.set_value("NB","wiki_mean",0.939035)
results_df.set_value("ExtraTree","wiki_mean",0.939249)
results_df.set_value("SVM","wiki_mean",0.925018)
results_df.set_value("MLP","wiki_mean",0.074982)
results_df.set_value("LREG","wiki_mean",0.939035)

results_df.set_value("NB","wiki_tfidf",0.941720)
results_df.set_value("ExtraTree","wiki_tfidf",0.944973)
results_df.set_value("SVM","wiki_tfidf",0.953854)
results_df.set_value("MLP","wiki_tfidf",0.950935)
results_df.set_value("LREG","wiki_tfidf",0.953311)
results_df
del wiki
del model
import gc; gc.collect()
time.sleep(10)
"""
<a id="46"></a> <br>
#  10-CONCLUSION ON TRADITIONAL METHODS

"""
fig,axes=plt.subplots(1,1,figsize=(15,8))
plt.ylim((.5,1))
axes.set_ylabel("Accuracy")
axes.set_title("Traditional Classfieris Results for 67% Training and 23% Testing with Two Types of Embedding")
results_df[results_df.index != "RF"].plot(kind="bar",ax=axes)
"""
The point could be mentioed is approximately whole the classifiers and whole the embedding methods (except our trained w2v) have the similar accuracy another point could be mentioned is in overall range of accuracy in tfidf model is greater than mean-vectorizer models.
"""
"""
Now, Lets remove variables for having clean workspace for rest of the kernel.
"""
for name in dir():
    if not name.startswith('_'):
        del globals()[name]
for name in dir():
    if not name.startswith('_'):
        del locals()[name]
import gc; gc.collect()
"""
<a id="47"></a> <br>
#  11-MOVING TO DEEP MODELS
"""
"""
<a id="48"></a> <br>
* **A. INTRODUCTION AND ROADMAP**

Congratulations. Now you have enough information about 
        
*What does text mining means ?!*

*How to convert text to the apprehensible language for MachineLearning algorithms ?!*

*What does countVectorizer and TfIdf vectorizer means ?! *

*What embedding is ?!*

*How to map embeddings to our data ?!*
    

"""
"""
In this section, we will look at how we can do a similar procedure in deep learning world.  We will use Keras as our main deep learning interface. The back engine is also tensorflow.  In the next sections we will look at the embedding methods and how we can apply them in our manner using Keras api's.
    
"""
"""
Keras offers an Embedding layer that can be used for neural networks on text data. It requires that the input data be integer encoded, so that each word is represented by a unique integer. This data preparation step can be performed using the Tokenizer API also provided with Keras. The Embedding layer is initialized with random weights and will learn an embedding for all of the words in the training dataset.
It is a flexible layer that can be used in a variety of ways, such as:

1. It can be used alone to learn a word embedding that can be saved and used in another model later.

2. It can be used as part of a deep learning model where the embedding is learned along with the model itself.

3. It can be used to load a pre-trained word embedding model, a type of transfer learning.

"""
"""
The Embedding layer is defined as the first hidden layer of a network. It must specify 3 arguments:

1. input_dim: This is the size of the vocabulary in the text data. For example, if your data is integer encoded to values between 0-10, then the size of the vocabulary would be 11 words.
2. output_dim: This is the size of the vector space in which words will be embedded. It defines the size of the output vectors from this layer for each word. For example, it could be 32 or 100 or even larger. Test different values for your problem.
3. input_length: This is the length of input sequences, as you would define for any input layer of a Keras model. For example, if all of your input documents are comprised of 1000 words, this would be 1000.


"""
"""
For example, we want have 5000 word and embedding size we want is 300. So, we have Embedding(input_dim=5000, output_dim=300). The output of the Embedding layer is a 2D vector with one embedding for each word in the input sequence of words (input document). If you wish to connect a Dense layer directly to an Embedding layer, you must first flatten the 2D output matrix to a 1D vector using the Flatten layer.
"""
"""
<a id="49"></a> <br>
* **B. PREPARING DATA FOR USING IN KERAS**

We select X_train and X_test as the same way we had done in previous section.

Lets initialize the transformation process.

"""
import pandas as pd
from keras import Sequential
from keras.layers import Bidirectional, GlobalMaxPool1D
from keras.layers import Bidirectional, GlobalMaxPool1D,Dense, Input, Embedding, Dropout, LSTM, CuDNNGRU
from keras import Sequential
from keras.models import Model
from keras.preprocessing.sequence import pad_sequences
from keras.preprocessing.text import Tokenizer
from sklearn.model_selection import train_test_split
train_df = pd.DataFrame.from_csv("../input/train.csv")
X_train, X_test, y_train, y_test = train_test_split(train_df.question_text, train_df.target, test_size=0.33, random_state=42)
"""
Same to the process we had done in countVectorizer, we need to tokenize the text and convert them to list of integeres. But how ?!

Keras provides the tokenizer method for doing it easily.
"""
num_words = 5000
maxlen = 100
tokenizer = Tokenizer(num_words=num_words)
tokenizer.fit_on_texts(list(X_train))
X_train = tokenizer.texts_to_sequences(X_train)
X_test = tokenizer.texts_to_sequences(X_test)
X_train[0]
len(X_train[0]),len(X_train[10])
"""
We need to have a standard dimentions for whole the X_train records. As you can see above, the dimantion of one vector is 14; and it is 22 for anothre record. we need to convert them to standard dimentoin.
Hopefully, like Tokenizer, keras also provide pad_sequence to solving this issue. Lets do it. 

Lets consider the padding size 150. It means we ignore the words 151 and plus in sentences (if exist)
"""
max_len = 150
X_train = pad_sequences(X_train, maxlen=maxlen)
X_test = pad_sequences(X_test, maxlen=maxlen)
len(X_train[0]),len(X_train[10])
"""
Now, you can see the issule has been solved. Now, whole the X_train has the same dimention. Now, it can be passed to the classfieirs. 

We have used CuDNNGRU which is Fast GRU implementation backed by cuDNN. So, first of all turn on GPU for your kernel in the your kernel setting. If you are dealing with a system which is does not contain GPU, replace the CuDNNGRU with LSTM method. more information could be found [here.](https://stackoverflow.com/questions/49183538/simple-example-of-cudnngru-based-rnn-implementation-in-tensorflow)
"""
embedding_size = 300
model = Sequential()
model.add(Embedding(num_words, embedding_size))
## Bidirectional wrapper for RNNs. It involves duplicating the first recurrent
## layer in the network so that there are now two layers side-by-side, then 
## providing the input sequence as-is as input to the first layer and providing
## a reversed copy of the input sequence to the second.
model.add(Bidirectional(CuDNNGRU(64, return_sequences=True))) 
model.add(GlobalMaxPool1D())
model.add(Dense(16, activation="relu"))
model.add(Dropout(0.1))
model.add(Dense(1, activation="sigmoid"))
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
model.summary()
model.fit(x=X_train, y=y_train, batch_size=512, epochs=1, validation_data=(X_test, y_test))
"""
Now we have created our model considering deep learning approach. 

Another point could be touched on is, because of being time consuing procedures, I have limited the epoch numbers on 1. It is clear that the greater number of epochs is proportional to the more accuracy you can get.

I will try to present some concise doctuments  about the difference existed in various types of deep learning models. and represent them in future commits.
"""
"""
The kernel is under development and more analysis in both sides of concept descriptions and code developements have been queued to come in near future.
Be in touch with it ;-)

The rest of tutorial will be completed  as soon as possible ;-).

Any comment, idea or hint will be appreciated.

**Your upvote will be motivation for me for continuing the kernel ;-)**

"""
"""
<a id="100"></a> <br>
#  11-REFERENCES

1- [Datascience for Buisiness.](https://www.amazon.com/Data-Science-Business-Data-Analytic-Thinking/dp/1449361323).

2- [tf-idf website.](http://www.tfidf.com/)

3- [Google Machine Learning Developement website.](https://developers.google.com/machine-learning/crash-course/embeddings/video-lecture)

4- [Text Classification With Word2Vec.](http://nadbordrozd.github.io/blog/2016/05/20/text-classification-with-word2vec/)

5-[How use word embedding layers for DeepLearning with Keras.](https://machinelearningmastery.com/use-word-embedding-layers-deep-learning-keras/)

6-Personal experiences in similar projects.

"""