# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: Python 3
#     language: python
#     name: python3
# ---

# # Exploratory data analysis and working with texts
#
# In this notebook, we learn about:
# 1. descriptive statistics to explore data;
# 2. working with texts

# # Part 1: descriptive statistics
#
# *"The goal of exploratory data analysis is to develop an understanding of your data. EDA is fundamentally a creative process. And like most creative processes, the key to asking quality questions is to generate a large quantity of questions."* 
#
# Key questions:
# * Which kind of variation occurs within variables?
# * Which kind of co-variation occurs between variables?
#
# https://r4ds.had.co.nz/exploratory-data-analysis.html

# +
# imports

import os, codecs
import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
# -

# ## Import the dataset
# Let us import the Venetian apprenticeship contracts dataset in memory.

root_folder = "../data/apprenticeship_venice/"
df_contracts = pd.read_csv(codecs.open(os.path.join(root_folder,"professions_data.csv"), encoding="utf8"), sep=";")
df_professions = pd.read_csv(codecs.open(os.path.join(root_folder,"professions_classification.csv"), encoding="utf8"), sep=",")

# Let's take another look to the dataset.

df_contracts.info()

df_contracts.head(5)

df_contracts.columns

# Every row represents an apprenticeship contract. Contracts were registered both at the guild's and at a public office. This is a sample of contracts from a much larger set of records.
#
# Some of the variables we will work with are:
# * `annual_salary`: the annual salary paid to the apprencice, if any (in Venetian ducats).
# * `a_profession` to `corporation`: increasingly generic classifications for the apprentice's stated profession.
# * `startY` and `enrolmentY`: contract start and registration year respectively.
# * `length`: of the contract, in years.
# * `m_gender` and `a_gender`: of master and apprentice respectively.
# * `a_age`: age of the apprentice at entry, in years.
# * `female_guarantor`: if at least one of the contract's guarantors was female, boolean.

df_professions.head(3)

# The professions data frame contains a classification system for each profession as found in the records (transcription, first column). The last column is the guild (or corporation) which governed the given profession. This work was performed manually by historians. We don't use it here as the classifications we need are already part of the main dataframe.

# ### Questions
#
# * Plot the distribution (histogram) of the apprentices' age, contract length, annual salary and start year.
# * Calculate the proportion of female apprentices and masters, and of contracts with a female guarantor.
# * How likely it is for a female apprentice to have a female master? And for a male apprentice?

df_contracts.annual_salary.hist(bins=100)

df_contracts[df_contracts.annual_salary < 20].annual_salary.hist(bins=25)

df_contracts.a_gender.sum()/df_contracts.shape[0]

df_contracts.m_gender.sum()/df_contracts.shape[0]

df_contracts[(df_contracts.a_gender == 0) & (df_contracts.startY < 1600)].m_gender.sum()
    /df_contracts[(df_contracts.a_gender == 0) & (df_contracts.startY < 1600)].shape[0]

df_contracts.startY.hist(bins=10)

# ## Looking at empirical distributions

df_contracts[df_contracts.annual_salary < 50].annual_salary.hist(bins=40)

df_contracts[df_contracts.a_age < 30].a_age.hist(bins=25)

# ### Two very important distributions

# #### Normal
#
# Also known as Gaussian, is a bell-shaped distribution with mass around the mean and exponentially decaying on the sides. It is fully characterized by the mean (center of mass) and standard deviation (spread).
#
# https://en.wikipedia.org/wiki/Normal_distribution

s1 = np.random.normal(5, 1, 10000)
sns.distplot(s1)

# for boxplots see https://en.wikipedia.org/wiki/Interquartile_range (or ask!)
sns.boxplot(s1)

# #### Heavy-tailed
# Distributions with a small but non-negligible amount of observations with high values. Several probability distributions follow this pattern: https://en.wikipedia.org/wiki/Heavy-tailed_distribution#Common_heavy-tailed_distributions.
#
# We pick the lognormal here: https://en.wikipedia.org/wiki/Log-normal_distribution

s2 = np.random.lognormal(5, 1, 10000)
sns.distplot(s2)

sns.boxplot(s2)

# +
# Why "lognormal"?

sns.distplot(np.log(s2))
# -

# #### Box plots
#
# <img src="figures/eda-boxplot.png" width="800px" heigth="800px">

# ### Outliers, missing values
#
# An *outlier* is an observation far from the center of mass of the distribution. It might be an error or a genuine observation: this distinction requires domain knowledge. Outliers infuence the outcomes of several statistics and machine learning methods: it is important to decide how to deal with them.
#
# A *missing value* is an observation without a value. There can be many reasons for a missing value: the value might not exist (hence its absence is informative and it should be left empty) or might not be known (hence the value is existing but missing in the dataset and it should be marked as NA).
#
# *One way to think about the difference is with this Zen-like koan: An explicit missing value is the presence of an absence; an implicit missing value is the absence of a presence.*

# ## Summary statistics
# A statistic is a measure over a distribution, and it is said to be *robust* if not sensitive to outliers.
#
# * Not robust: min, max, mean, standard deviation.
# * Robust: mode, median, other quartiles.
#
# A closer look at the mean:
#
# $\bar{x} = \frac{1}{n} \sum_{i}x_i$
#
# And variance (the standard deviation is the square root of the variance):
#
# $Var(x) = \frac{1}{n} \sum_{i}(x_i - \bar{x})^2$

# <img src="figures/2560px-Comparison_mean_median_mode.svg.png" width="400px" heigth="400px">

# +
# Not robust: min, max, mean, mode, standard deviation

print(np.mean(s1)) # should be 5
print(np.mean(s2))

# +
# Robust: median, other quartiles

print(np.quantile(s1, 0.5)) # should coincide with mean and mode
print(np.quantile(s2, 0.5))
# -

# #### Questions
#
# * Calculate the min, max, mode and sd. *hint: explore the numpy documentation!*
# * Calculate the 90% quantile values.
# * Consider our normally distributed data in s1. Add an outlier (e.g., value 100). What happens to the mean and mode? Write down your answer and then check.

# Let's explore our dataset
df_contracts[["annual_salary","a_age","length"]].describe()

# ## Relating two variables
#
# #### Covariance
#
# Measure of joint linear variability of two variables:
#
# <img src="figures/covariance.png" width="400px" heigth="400px">
#
# Its normalized version is called the (Pearson's) correlation coefficient:
#
# <img src="figures/pearson.png" width="400px" heigth="400px">
#
# Correlation is helpful to spot possible relations, but is of tricky interpretation and is not exhaustive:
#
# <img src="figures/800px-Correlation_examples2.svg.png" width="700px" heigth="&00px">
#
# See: https://en.wikipedia.org/wiki/Covariance and https://en.wikipedia.org/wiki/Pearson_correlation_coefficient.
#
# *Note: correlation is not causation!*

df_contracts[["annual_salary","a_age","length"]].corr()

sns.scatterplot(df_contracts.length,df_contracts.annual_salary)

# #### Questions
#
# * Try to explore the correlation of other variables in the dataset.
# * Can you think of a possible motivation for the trend we see: older apprentices with a shorter contract getting on average a higher annual salary?

# ## Sampling and uncertainty (mention)
#
# Often, we work with samples and we want the sample to be representative of the population it is taken from, in order to draw conclusions that generalise from the sample to the full population.
#
# Sampling is *tricky*. Samples have *variance* (variation between samples from the same population) and *bias* (systematic variation from the population).

# # Part 2: working with texts
#
# Let's get some basics (or a refresher) of working with texts in Python. Texts are sequences of discrete symbols (words or, more generically, tokens).
#
# Key challenge: representing text for further processing. Two mainstream approaches:
# * *Bag of words*: a text is a collection of tokens occurring with a certain frequence and assumed independently from each other within the text. The mapping from texts to features is determinsitic and straighforward, each text is represented as a vector of the size of the vocabulary.
# * *Embeddings*: a method is used (typically, neural networks), to learn a mapping from each token to a (usually small) vector representing it. A text can be represented in turn as an aggregation of these embeddings.

# ## Import the dataset
# Let us import the Elon Musk's tweets dataset in memory.
#
# <img src="figures/elon_loop.jpeg" width="400px" heigth="400px">

root_folder = "../data/musk_tweets"
df_elon = pd.read_csv(codecs.open(os.path.join(root_folder,"elonmusk_tweets.csv"), encoding="utf8"), sep=",")
df_elon['text'] = df_elon['text'].str[1:]

df_elon.head(5)

df_elon.shape

# ## Natural Language Processing in Python

# import some of the most popular libraries for NLP in Python
import spacy
import nltk
import string
import sklearn

# +
# nltk.download('punkt')
# -

# A typical NLP pipeline might look like the following:
#     
# <img src="figures/spacy_pipeline.png" width="600px" heigth="600px">
#
# ### Tokenization: splitting a text into constituent tokens.

from nltk.tokenize import TweetTokenizer, word_tokenize
tknzr = TweetTokenizer(preserve_case=True, reduce_len=False, strip_handles=False)

example_tweet = df_elon.text[1]
print(example_tweet)

tkz1 = tknzr.tokenize(example_tweet)
print(tkz1)
tkz2 = word_tokenize(example_tweet)
print(tkz2)

# Question: can you spot what the Twitter tokenizer is doing instead of a standard one?

string.punctuation


# +
# some more pre-processing

def filter(tweet):
    
    # remove punctuation and short words and urls
    tweet = [t for t in tweet if t not in string.punctuation and len(t) > 3 and not t.startswith("http")]
    return tweet

def tokenize_and_string(tweet):
    
    tkz = tknzr.tokenize(tweet)
    
    tkz = filter(tkz)
    
    return " ".join(tkz)


# -

print(tkz1)
print(filter(tkz1))

df_elon["clean_text"] = df_elon["text"].apply(tokenize_and_string)

df_elon.head(5)

# +
# save cleaned up version

df_elon.to_csv(os.path.join(root_folder,"df_elon.csv"), index=False)
# -

# ### Building a dictionary

from sklearn.feature_extraction.text import CountVectorizer
count_vect = CountVectorizer(lowercase=False, tokenizer=tknzr.tokenize)
X_count = count_vect.fit_transform(df_elon.clean_text)
X_count.shape

word_list = count_vect.get_feature_names()  
count_list = X_count.toarray().sum(axis=0)
dictionary = dict(zip(word_list,count_list))
count_vect.vocabulary_.get("robots")

X_count[:,count_vect.vocabulary_.get("robots")].toarray().sum()

dictionary["robots"]

# #### Questions
#
# * Find the tokens most used by Elon.
# * Find the twitter users most referred to by Elon (hint: use the @ handler to spot them).

dictionary_list = sorted(dictionary.items(), key=lambda x:x[1], reverse=True)
[d for d in dictionary_list][:10]

dictionary_list_users = sorted(dictionary.items(), key=lambda x:x[1], reverse=True)
[d for d in dictionary_list if d[0].startswith('@')][:10]

# ### Representing tweets as vectors
#
# Texts are of variable length and need to be represented numerically in some way. Most typically, we represent them as *equally-sized vectors*.
#
# Actually, this is what we have already done! Let's take a closer look at `X_count` above..

# +
# This is the first Tweet of the data frame

df_elon.loc[0]

# +
# let's get the vector representation for this Tweet

vector_representation = X_count[0,:]

# +
# there are 3 positions not to zero, as we would expect: the vector contains 1 in the columns related to the 3 words that make up the Tweet. 
# It would contain a number higher than 1 if a given word were occurring multiple times.

np.sum(vector_representation)

# +
# Let's check that indeed the vector contains 1s for the right words
# Remember, the vector has shape (1 x size of the vocabulary)

print(vector_representation[0,count_vect.vocabulary_.get("robots")])
print(vector_representation[0,count_vect.vocabulary_.get("spared")])
print(vector_representation[0,count_vect.vocabulary_.get("humanity")])
# -

# ### Term Frequency - Inverse Document Frequency
# We can use boolean counts (1/0) and raw counts (as we did before) to represent a Tweet over the space of the vocabulary, but there exist improvements on this basic idea. For example, the TF-IDF weighting scheme:
#
# $tfidf(t, d, D) = tf(t, d) \cdot idf(t, D)$
#
# $tf(t, d) = f_{t,d}$
#
# $idf(t, D) = log \Big( \frac{|D|}{|{d \in D: t \in d}|} \Big)$

from sklearn.feature_extraction.text import TfidfVectorizer
count_vect = TfidfVectorizer(lowercase=False, tokenizer=tknzr.tokenize)
X_count_tfidf = count_vect.fit_transform(df_elon.clean_text)
X_count_tfidf.shape

X_count_tfidf[0,:].sum()

X_count[0,:].sum()

# #### Sparse vectors (mention)
# How is Python representing these vectors in memory? Most of their cells are set to zero. 
#
# We call any vector or matrix whose cells are mostly to zero *sparse*.
# There are efficient ways to store them in memory.

X_count_tfidf[0,:]

# ### Spacy pipelines
#
# Useful to construct sequences of pre-processing steps: https://spacy.io/usage/processing-pipelines.

# +
# Install the required model

# #!python -m spacy download en_core_web_sm

# +
# without this line spacy is not able to find the downloaded model

# #!python -m spacy link --force en_core_web_sm en_core_web_sm

# +
# Load a pre-trained pipeline (Web Small): https://spacy.io/usage/models

# #!python -m spacy download en_core_web_sm
nlp = spacy.load('en_core_web_sm')
# -

# *.. the model’s meta.json tells spaCy to use the language "en" and the pipeline ["tagger", "parser", "ner"]. spaCy will then initialize spacy.lang.en.English, and create each pipeline component and add it to the processing pipeline. It’ll then load in the model’s data from its data directory and return the modified Language class for you to use as the nlp object.*
#
# Let's create a simple pipeline that does **lemmatization**, **part of speech tagging** and **named entity recognition** using spaCy models.
#
# *If you don't know what these NLP tasks are, please ask!*

# +
tweet_pos = list()
tweet_ner = list()
tweet_lemmas = list()

for tweet in df_elon.text.values:
    spacy_tweet = nlp(tweet)
    
    local_tweet_pos = list()
    local_tweet_ner = list()
    local_tweet_lemmas = list()
    
    for sentence in list(spacy_tweet.sents):
        # --- lemmatization, remove punctuation and stop wors
        local_tweet_lemmas.extend([token.lemma_ for token in sentence if not token.is_punct | token.is_stop])
        local_tweet_pos.extend([token.pos_ for token in sentence if not token.is_punct | token.is_stop])
    for ent in spacy_tweet.ents:
        local_tweet_ner.append(ent)

    tweet_pos.append(local_tweet_pos)
    tweet_ner.append(local_tweet_ner)
    tweet_lemmas.append(local_tweet_lemmas)
# -

tweet_lemmas[0]

tweet_pos[0]

tweet_ner[0]

# +
# but it actually works!

tweet_ner[3]
# -

# *Note: we are really just scratching the surface of spaCy, but it is worth knowing it's there.*

# ### Searching tweets
#
# Once we have represented Tweets as vectors, we can easily find similar ones using basic operations such as filtering.

target = 0
print(df_elon.clean_text[target])

condition = X_count_tfidf[target,:] > 0

print(condition)

X_filtered = X_count_tfidf[:,np.ravel(condition.toarray())]

X_filtered

print(X_filtered)

# +
from scipy import sparse

sparse.find(X_filtered)
# -

tweet_indices = list(sparse.find(X_filtered)[0])

# +
print("TARGET: " + df_elon.clean_text[target])

for n, tweet_index in enumerate(list(set(tweet_indices))):
    if tweet_index != target:
        print(str(n) +")"+ df_elon.clean_text[tweet_index])
# -

# #### Questions
#
# * Can you rank the matched tweets using their tf-idf weights, so to put higher weighted tweets first?
# * Which limitations do you think a bag of words representation has?
# * Can you spot any limitations of this approach based on similarity measures over bag of words representations?

StarcoderdataJupyterScriptsDedupFiltered