import pandas as pd
import re

df = pd.read_csv('data_tau_ta.csv')

df.head()

df.shape

import nltk

from nltk.corpus import stopwords

stop = stopwords.words('english')

stop.extend(('.', ',', '"', "'", '?', '!', ':', ';', '(', ')', '[', ']', '{', '}','/','-'))

tokens_list = df['tokens'].tolist()

tokens_list

# Let us get the frequency count
frequency_words = {}
for data in tokens_list:
    data = data.replace("[","")
    data = data.replace("]","")
    data = data.replace("'","")
    data_list = data.split(',')
    print(data_list)
    for token in data_list:
        token = token.rstrip()
        token = token.lstrip()
        if token not in stop:
            if token in frequency_words:
                count = frequency_words[token]
                count = count + 1
                frequency_words[token] = count
            else:
                frequency_words[token] = 1

frequency_words['data']

"""
Explanation: Model the Problem
Preprocessing the data
End of explanation
"""


df_tfidf = pd.DataFrame(data=list(frequency_words.items()),columns=['word','tf'])

df_tfidf.head()

df_tfidf.sort_values(ascending=False, by = "tf", inplace=True)

df_tfidf.head()

"""
Explanation: Term Frequency and Inverse Document Frequency
tf–idf, short for term frequency–inverse document frequency, is a numerical statistic that is intended to reflect how important a word is to a document in a collection or corpus.[1]:8 It is often used as a weighting factor in information retrieval and text mining. The tf-idf value increases proportionally to the number of times a word appears in the document, but is offset by the frequency of the word in the corpus, which helps to adjust for the fact that some words appear more frequently in general.
Variations of the tf–idf weighting scheme are often used by search engines as a central tool in scoring and ranking a document's relevance given a user query. tf–idf can be successfully used for stop-words filtering in various subject fields including text summarization and classification.
Let us start with the "Term Frequency" - TF
End of explanation
"""


def get_documents_count(row):
    document_counter = 0
    word = row['word']
    for document in df.tokens:
        document = document.replace("'",'')
        document = document.replace("[",'')
        document = document.replace("]",'')
        document = document.split(',')
        document = map(str.strip,document)
        if word in document:
            document_counter = document_counter + 1
    return document_counter

df_tfidf['document_count'] = df_tfidf.apply(get_documents_count,axis=1)

df_tfidf.head()

df_tfidf.tail()

# we already have the count of all the documents
total_docs = df.shape[0]

total_docs

"""
Explanation: Let us get in how many documents (each title) does the word occur
End of explanation
"""


import math
from wordcloud import WordCloud
import matplotlib.pyplot as plt
%matplotlib inline

def compute_tfidf(row):
    idf = math.log10(total_docs/row['document_count'])
    return row['tf'] * idf

df_tfidf['tfidf'] = df_tfidf.apply(compute_tfidf,axis=1)

df_tfidf.head()

df_tfidf.tail()

df_tfidf.sort_values(by='tfidf',ascending=True,inplace=True)

df_tfidf.head()

df_tfidf.replace(to_replace=0.0,value=0.1,inplace=True)

df_tfidf.tail()

df_tfidf.set_index('word', inplace=True)

df_tfidf.head()

"""
Explanation: Let us compute the tf-idf

Term Frequency = tf
Inverse Document Frequency = idf

idf = log(total_docs/number of documents that contain the word)
tf-idf = tf . idf
End of explanation
"""


wordcloud = WordCloud()

word_tfidf = df_tfidf['tfidf'].to_dict()

wordcloud.generate_from_frequencies(word_tfidf.items())
plt.figure(figsize=(14,10))
plt.imshow(wordcloud)
plt.axis("off")
plt.show()

"""
Explanation: now let us plot a word cloud to see the prominence of the word
End of explanation
"""


import lda
import numpy as np
import lda.datasets
import sklearn.feature_extraction.text as text

"""
Explanation: Topic modelling (LDA - Latent Dirichlet allocation)
In natural language processing, Latent Dirichlet allocation (LDA) is a generative statistical model that allows sets of observations to be explained by unobserved groups that explain why some parts of the data are similar. For example, if observations are words collected into documents, it posits that each document is a mixture of a small number of topics and that each word's creation is attributable to one of the document's topics.
Original Paper on LDA - http://jmlr.org/papers/v3/blei03a.html
Summary - We describe latent Dirichlet allocation (LDA), a generative probabilistic model for collections of discrete data such as text corpora. LDA is a three-level hierarchical Bayesian model, in which each item of a collection is modeled as a finite mixture over an underlying set of topics. Each topic is, in turn, modeled as an infinite mixture over an underlying set of topic probabilities. In the context of text modeling, the topic probabilities provide an explicit representation of a document. We present efficient approximate inference techniques based on variational methods and an EM algorithm for empirical Bayes parameter estimation. We report results in document modeling, text classification, and collaborative filtering, comparing to a mixture of unigrams model and the probabilistic LSI model.
Here is a graphical approach to build intuition around this topic - http://www.mblondel.org/journal/2010/08/21/latent-dirichlet-allocation-in-python/
Here is a video which explains LDA - https://www.youtube.com/watch?v=ePUAZ8RG-3w
End of explanation
"""


vectorizer = text.CountVectorizer(input='content', stop_words='english', min_df=1)

dtm = vectorizer.fit_transform(df.title).toarray()

dtm

"""
Explanation: Generating the document term matrix
End of explanation
"""


vocab = np.array(vectorizer.get_feature_names())

vocab[:20]

titles = df.title

model = lda.LDA(n_topics=5, n_iter=500, random_state=1)

model.fit(dtm)

model.topic_word_

topic_word = model.topic_word_ 

topic_word

"""
Explanation: Loading the vocabulary
End of explanation
"""


n_top_words = 8

for i, topic_dist in enumerate(topic_word):
    topic_words = np.array(vocab)[np.argsort(topic_dist)][:-n_top_words:-1]
    print('Topic {}: {}'.format(i, ' '.join(topic_words)))

"""
Explanation: Finding the key words that come together for each topic
End of explanation
"""


doc_topic = model.doc_topic_

for n in range(10):
    topic_most_pr = doc_topic[n].argmax()
    print("topic: {} , {}".format(topic_most_pr,titles[n]))

"""
Explanation: Finding the Topic for each Document
End of explanation
"""


from nltk.classify import NaiveBayesClassifier
import math
import collections

pos_features = []
neg_features = []

def make_full_dict(word):
    return dict([(word, True)])

with open('postive_words.txt','r') as posFile:
    lines = posFile.readlines()
    for line in lines:
        pos_features.append([make_full_dict(line.rstrip()),'pos'])
        

pos_features

with open('negative_words.txt','r',encoding='utf-8') as negFile:
    lines = negFile.readlines()
    for line in lines:
        neg_features.append([make_full_dict(line.rstrip()),'neg'])

neg_features

len(pos_features),len(neg_features)

trainFeatures = pos_features + neg_features

trainFeatures

classifier = NaiveBayesClassifier.train(trainFeatures)

referenceSets = collections.defaultdict(set)
testSets = collections.defaultdict(set)

def make_full_dict_sent(words):
    return dict([(word, True) for word in words])

import re

neg_test = 'I hate data science'

title_words = re.findall(r"[\w']+|[.,!?;]",
                         'The Daily Mail stole My Visualization, Twice')

title_words

test=[]

test.append([make_full_dict_sent(title_words),''])

test

for i, (features, label) in enumerate(test):
    predicted = classifier.classify(features)
    print(predicted)

for doc in df.title:
    title_words = re.findall(r"[\w']+|[.,!?;]", doc.lower())
    test = []
    test.append([make_full_dict_sent(title_words),''])
    for i, (features, label) in enumerate(test):
        predicted = classifier.classify(features)
        print(predicted,doc)
    

"""
Explanation: Sentiment Analysis
Sentiment analysis aims to determine the attitude of a speaker or a writer with respect to some topic or the overall contextual polarity of a document. The attitude may be his or her judgment or evaluation (see appraisal theory), affective state (that is to say, the emotional state of the author when writing), or the intended emotional communication (that is to say, the emotional effect the author wishes to have on the reader).
A basic task in sentiment analysis is classifying the polarity of a given text at the document, sentence, or feature/aspect level — whether the expressed opinion in a document, a sentence or an entity feature/aspect is positive, negative, or neutral. We will use knowledge-based techniques classify text by affect categories based on the presence of unambiguous affect words such as happy, sad, afraid, and bored.
Here is a link to the Sentiment Analysis from nltk site - http://www.nltk.org/howto/sentiment.html
Here is an example of Sentiment Analysis on Tweets data - http://www.laurentluce.com/posts/twitter-sentiment-analysis-using-python-and-nltk/
End of explanation
"""


