# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: Python 3
#     language: python
#     name: python3
# ---

# + [markdown] _cell_guid="7f7382af-cf02-47ac-bcea-7f5f407e28e1" _uuid="2ee8e2efa2101c1264fff2b8a72f846f9879646c"
# # Analysis of News Headlines: Topic Modelling with LSA
# In this Project, LSA modelling algorithm is explored. These techniques are applied to the 'A Million News Headlines' dataset, which is a corpus of over one million news article headlines published by the ABC. 

# + _cell_guid="b1f10f39-91d4-45f1-b62f-d1f590e50438" _kg_hide-input=false _kg_hide-output=false _uuid="7edd510ba8ac857514e34d6b38c0466d125cffb9"
## Import required packages and modules

# + _uuid="99df159d8549d11d99e8a349a4a7893812b187f2"
## Import Dataset

# + [markdown] _cell_guid="81e6c728-92d6-46b6-848f-2124df59fc92" _uuid="ea84e7b744d0d5701e0019253aa9905718e55d72"
# ## Exploratory Data Analysis
# As usual, it is prudent to begin with some basic exploratory analysis.

# +
## Check the count of NaN Values

# +
## If the NaN values are less, then drop them or else replace them with suitable values

# +
## Check new data values available

# + [markdown] _cell_guid="1cdbefb7-9e8e-4a11-9cee-d39f8f16f557" _uuid="c31f278963d7bdd4c3553ada1fdb6095d50ca658"
# First develop a list of the top words used across all one million headlines, giving us a glimpse into the core vocabulary of the source data. Stop words should be omitted here to avoid any trivial conjunctions, prepositions, etc.

# +
## Download Stopwords incase you haven't done that before
## you can use nltk.download('stopwords')

# +
## Define certain new stopwords that may have no significance in determining top news headlines

# + _cell_guid="db6ce2f5-1247-4446-90f2-6aa2ba8af168" _kg_hide-input=true _uuid="59f4ae5e8e06786fa3ebdec7ec3011645aad3544"
## Define helper functions to get top n words
## Defined function must return a tuple of the top n words in a sample and their 
## accompanying counts, given a CountVectorizer object and text sample

# + _cell_guid="a2701525-298e-4943-8d3e-adad5d0d629d" _uuid="142e18023411d5c61cbe59a3c08c78c58993cdcc"
## plot top 25 words in headlines dataset and their number of occurances
## Pass the new created set of stopwords to count vectoriser function
## Initially try to work on a batch of data instead of entire dataset (Say on 200000 examples)


# + [markdown] _cell_guid="781d7425-adcb-4642-8685-39722c5f2a4f" _uuid="a595fd6ead91a95c590d027c155fbc6fe3d800f6"
# Next you can generate a histogram of headline word lengths, and use part-of-speech tagging to understand the types of words used across the corpus. This requires first converting all headline strings to TextBlobs and calling the ```pos_tags``` method on each, yielding a list of tagged words for each headline.

# + _uuid="ce8f68db595b0ff3535e2c2b57f527b496a1dc99"
## You can download punkt and averaged perceptron tagger for NLTK if required using
## nltk.download('punkt')
## nltk.download('averaged_perceptron_tagger')

# +
## Identify Tagged Headlines

# + _cell_guid="f8ccacff-5771-4073-ba4a-6ad8bac2785c" _uuid="fc78480a55051d74b1802bc91aeba366b223fcb0"
## For furthur analysis one can try finding average headline word length
## and Part of speech tagging for headline corpus

# + _cell_guid="25556701-ea73-4a62-9456-d89f01c738b6" _uuid="2b3abe64f0e4567e508a5fa45057655f2d2fecd5"
## By plotting the number of headlines published per day, per month and per year,
## one can also get a sense of the sample density.

# + [markdown] _cell_guid="1627d53c-a120-4cdb-b2c2-a616865a24d0" _uuid="c5983d57a8ae630af799b218c196e9e006ddf2f8"
# ## Topic Modelling
# You can now apply a clustering algorithm to the headlines corpus in order to study the topic focus of ABC News, as well as how it has evolved through time. To do so, first experiment with a small subsample of the dataset, then scale up to a larger portion of the available data.

# + [markdown] _cell_guid="19b5a00b-e1c1-433a-aa4c-490bdd40b798" _uuid="0efcc663a7c6a267b0eba42378d902002677d422"
# ### Preprocessing
# The only preprocessing step required in our case is feature construction, where we take the sample of text headlines and represent them in some tractable feature space. In practice, this simply means converting each string to a numerical vector. This can be done using the ```CountVectorizer``` object from SKLearn, which yields an $n×K$ document-term matrix where $K$ is the number of distinct words  across the $n$ headlines in our sample (less stop words and with a limit of ```max_features```).

# + [markdown] _cell_guid="dc19ca30-8586-4a29-a0be-cd46e39ae4b3" _uuid="66eeb9d4c023e095911313862580f90d28f0d87f"
# Thus you have your (very high-rank and sparse) training data,  ```small_document_term_matrix```, and can now actually implement a clustering algorithm. Your choice Latent Semantic Analysis, will take document-term matrix as input and yield an $n \times N$ topic matrix as output, where $N$ is the number of topic categories (which we supply as a parameter). For the moment, we shall take this to be 15.

# + _cell_guid="12f50c7e-c8d4-473d-9918-d476faeed788" _uuid="009f7a3aa962a4e3a74f8bc78c4e14b845acf51f"
## To find top 15 topics we set
## n_topics = 15

# + [markdown] _cell_guid="0f3bc959-5f94-497b-afc2-6799ddd2a689" _uuid="2392087f63ed8827e3fb30ffea17d7c95e31c92f"
# ### Latent Semantic Analysis
# Let's start by experimenting with LSA. This is effectively just a truncated singular value decomposition of a (very high-rank and sparse) document-term matrix, with only the $r=$```n_topics``` largest singular values preserved.

# + _cell_guid="5edba11f-1c8c-45ee-a833-f39a6eed0ffe" _uuid="403f67711d38b3acba59c3725cdf3be86536969a"
## Define LSA Model

# + [markdown] _cell_guid="b4db9892-57a7-48e7-87e3-df1e8f47e948" _uuid="c90ceacbaa7085d28244b6d66b2ce57dcf9007bd"
# Taking the $\arg \max$ of each headline in this topic matrix will give the predicted topics of each headline in the sample. We can then sort these into counts of each topic.

# + _cell_guid="48e9a37b-f6a3-4f8c-87ea-4bbe0026fa10" _kg_hide-input=true _uuid="56ea2060287a4efc2ee17ecd3b2f2caf57d5812e"
## Define helper functions to get keys that returns an integer list of predicted topic 
## categories for a given topic matrix
## and KeysToCount that returns a tuple of topic categories and their 
## accompanying magnitudes for a given list of keys

# + [markdown] _cell_guid="c223207c-25e0-47f2-9475-44bb6b8b92fa" _uuid="f317335665331fa8dbc8aadf3a89750c4c3100db"
# However, these topic categories are in and of themselves a little meaningless. In order to better characterise them, it will be helpful to find the most frequent words in each.

# + _cell_guid="378ec442-fe50-40c1-af07-285c47f06991" _kg_hide-input=true _uuid="c260d3062f3039a58b924428b7013629f99c346d"
## Define helper function get_top_n_words that returns a list of n_topic strings, 
## where each string contains the n most common words in a predicted category, in order

# + [markdown] _cell_guid="c51c0e10-f1f8-4d41-acff-941ecb6af716" _uuid="44c511893a4403d28df7539caa1b13b32dcc4567"
# Thus we have converted our initial small sample of headlines into a list of predicted topic categories, where each category is characterised by its most frequent words. The relative magnitudes of each of these categories can then be easily visualised though use of a bar chart.

# + _cell_guid="5f51277b-37d2-4712-9737-00b50a396e8a" _uuid="a4f5e96898e844072d57e237d307ed589bb6d8cf"
## Visualise each topic vs Number of headlines These will be the most discussed topics 
## In case you want to do furthur analysis you can try dimentionality reduction and 
## analyse and compare it's result to other techniques like LDA that is left as an optional assignment for you

# + [markdown] _cell_guid="0ccf8530-c578-4cad-a705-0a707b426d25" _uuid="26ce1e10578112b422b29c7079c7cd684156abee"
# However, this does not provide a great point of conclusion, you can instead use a dimensionality-reduction technique called $t$-SNE, which will also serve to better illuminate the success of the clustering process.

# + [markdown] _cell_guid="b8435c0e-5c00-4c8e-8650-db70106f67e3" _uuid="615effdf063641b3f6f1da9576f9ea1de11a27e7"
# Now that you have reduced these ```n_topics```-dimensional vectors to two-dimensional representations, you can then plot the clusters using Bokeh. Before doing so however, it will be useful to derive the centroid location of each topic, so as to better contextualise our visualisation.

# + _cell_guid="91916c42-685c-4c3b-bc68-2248ac8efcd0" _kg_hide-input=true _uuid="d37e004a79a6d264ca40c43e9dc847597402e318"
# Define helper functions that returns a list of centroid vectors from each predicted topic category

# + [markdown] _cell_guid="5e600401-a789-422b-a11c-82442c4eb50c" _uuid="98c59c1fbb011e3157212da64ac05d86200c829c"
# All that remains is to plot the clustered headlines. Also included are the top three words in each cluster, which are placed at the centroid for that topic.

StarcoderdataJupyterScriptsDedupFiltered