from gensim.models import ldamodel
from gensim.corpora import Dictionary
import pandas as pd
import re
from gensim.parsing.preprocessing import remove_stopwords, strip_punctuation

import numpy as np

df_fake = pd.read_csv('fake.csv')
df_fake[['title', 'text', 'language']].head()
df_fake = df_fake.loc[(pd.notnull(df_fake.text)) & (df_fake.language == 'english')]

# remove stopwords and punctuations
def preprocess(row):
    return strip_punctuation(remove_stopwords(row.lower()))
    
df_fake['text'] = df_fake['text'].apply(preprocess)

# Convert data to required input format by LDA
texts = []
for line in df_fake.text:
    lowered = line.lower()
    words = re.findall(r'\w+', lowered, flags = re.UNICODE | re.LOCALE)
    texts.append(words)

dictionary = Dictionary(texts)

training_texts = texts[:5000]
holdout_texts = texts[5000:7500]
test_texts = texts[7500:10000]

training_corpus = [dictionary.doc2bow(text) for text in training_texts]
holdout_corpus = [dictionary.doc2bow(text) for text in holdout_texts]
test_corpus = [dictionary.doc2bow(text) for text in test_texts]

from gensim.models.callbacks import CoherenceMetric, DiffMetric, PerplexityMetric, ConvergenceMetric

# define perplexity callback for hold_out and test corpus
pl_holdout = PerplexityMetric(corpus=holdout_corpus, logger="visdom", title="Perplexity (hold_out)")
pl_test = PerplexityMetric(corpus=test_corpus, logger="visdom", title="Perplexity (test)")

# define other remaining metrics available
ch_umass = CoherenceMetric(corpus=training_corpus, coherence="u_mass", logger="visdom", title="Coherence (u_mass)")
ch_cv = CoherenceMetric(corpus=training_corpus, texts=training_texts, coherence="c_v", logger="visdom", title="Coherence (c_v)")
diff_kl = DiffMetric(distance="kullback_leibler", logger="visdom", title="Diff (kullback_leibler)")
convergence_kl = ConvergenceMetric(distance="jaccard", logger="visdom", title="Convergence (jaccard)")

callbacks = [pl_holdout, pl_test, ch_umass, ch_cv, diff_kl, convergence_kl]

# training LDA model
model = ldamodel.LdaModel(corpus=training_corpus, id2word=dictionary, num_topics=35, passes=50, chunksize=1500, iterations=200, alpha='auto', callbacks=callbacks)

"""
Explanation: Setup Visdom
Install it with:
pip install visdom
Start the server:
python -m visdom.server
Visdom now can be accessed at http://localhost:8097 in the browser.
LDA Training Visualization
Knowing about the progress and performance of a model, as we train them, could be very helpful in understanding it’s learning process and makes it easier to debug and optimize them. In this notebook, we will learn how to visualize training statistics for LDA topic model in gensim. To monitor the training, a list of Metrics is passed to the LDA function call for plotting their values live as the training progresses. 
<img src="visdom_graph.png">
Let's plot the training stats for an LDA model being trained on kaggle's fake news dataset. We will use the four evaluation metrics available for topic models in gensim: Coherence, Perplexity, Topic diff and Convergence. (using separate hold_out and test corpus for evaluating the perplexity)
End of explanation
"""


# to get a metric value on a trained model
print(CoherenceMetric(corpus=training_corpus, coherence="u_mass").get_value(model=model))

"""
Explanation: When the model is set for training, you can open http://localhost:8097 to see the training progress.
End of explanation
"""


import logging
from gensim.models.callbacks import CoherenceMetric, DiffMetric, PerplexityMetric, ConvergenceMetric

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)

# define perplexity callback for hold_out and test corpus
pl_holdout = PerplexityMetric(corpus=holdout_corpus, logger="shell", title="Perplexity (hold_out)")
pl_test = PerplexityMetric(corpus=test_corpus, logger="shell", title="Perplexity (test)")

# define other remaining metrics available
ch_umass = CoherenceMetric(corpus=training_corpus, coherence="u_mass", logger="shell", title="Coherence (u_mass)")
diff_kl = DiffMetric(distance="kullback_leibler", logger="shell", title="Diff (kullback_leibler)")
convergence_jc = ConvergenceMetric(distance="jaccard", logger="shell", title="Convergence (jaccard)")

callbacks = [pl_holdout, pl_test, ch_umass, diff_kl, convergence_jc]

# training LDA model
model = ldamodel.LdaModel(corpus=training_corpus, id2word=dictionary, num_topics=35, passes=2, eval_every=None, callbacks=callbacks)

"""
Explanation: The four types of graphs which are plotted for LDA:
Coherence
Coherence measures are generally based on the idea of computing the sum of pairwise scores of top n words w<sub>1</sub>, ...,w<sub>n</sub> used to describe the topic. There are four coherence measures available in gensim: u_mass, c_v, c_uci, c_npmi. A good model will generate coherent topics, i.e., topics with high topic coherence scores. Good topics can be described by a short label based on the topic terms they spit out. 
<img src="Coherence.gif">
Now, this graph along with the others explained below, can be used to decide if it's time to stop the training. We     can see if the value stops changing after some epochs and that we are able to get the highest possible coherence       of our model.  
Perplexity
Perplexity is a measurement of how well a probability distribution or probability model predicts a sample. In LDA, topics are described by a probability distribution over vocabulary words. So, perplexity can be used to evaluate the topic-term distribution output by LDA.
<img src="Perplexity.gif">
For a good model, perplexity should be low.
Topic Difference
Topic Diff calculates the distance between two LDA models. This distance is calculated based on the topics, by either using their probability distribution over vocabulary words (kullback_leibler, hellinger) or by simply using the common vocabulary words between the topics from both model.
<img src="Diff.gif">
In the heatmap, X-axis define the Epoch no. and Y-axis define the distance between identical topics from consecutive epochs. For ex. a particular cell in the heatmap with values (x=3, y=5, z=0.4) represent the distance(=0.4) between the topic 5 from 3rd epoch and topic 5 from 2nd epoch. With increasing epochs, the distance between the identical topics should decrease.
Convergence
Convergence is the sum of the difference between all the identical topics from two consecutive epochs. It is basically the sum of column values in the heatmap above.
<img src="Convergence.gif">
The model is said to be converged when the convergence value stops descending with increasing epochs.
Training Logs
We can also log the metric values after every epoch to the shell apart from visualizing them in Visdom. The only change is to define logger="shell" instead of "visdom" in the input callbacks.
End of explanation
"""


model.metrics

"""
Explanation: The metric values can also be accessed from the model instance for custom uses.
End of explanation
"""


