# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: Python 3
#     language: python
#     name: python3
# ---

# + [markdown] colab_type="text" id="Q2FtGCN6bVsg"
# # Finding Fact-checkable Tweets with Machine Learning
#
# -

# ## Overview
#
# Our challenge is to find out which tweets with the hashtag `#txlege` are **checkable statements of fact**.
#
# Take a look at the [kinds of tweets in question](https://twitter.com/search?q=%23txlege&src=typed_query).
#
# Recognizing "fact-checkability" seems like it might be a uniquely human skill. But we can make a machine-learning mmodel that does it pretty well.
#
# ### First, the language model
#
# We'll get into the details below, but here's our two-step process for this project:
#
# First, we need a model trained to recognize the patterns of English. For that, we'd need some huge dataset of English-language text. Fortunately, someone has already done that for us! We'll be using a model trained on thousands of long Wikipedia articles. It's called [wikitext-103](https://einstein.ai/research/blog/the-wikitext-long-term-dependency-language-modeling-dataset).
#
# We'll then use _transfer learning_ (like we did in for the helicopter maps) to further train wikitext-103 on our particular corpus: several thousand #txlege tweets. So we benefit from it's training on both English-language articles _and_ #txlege tweets.
#
# That will give us a **language model** that's good at detecting patterns in #txlege tweets.
#
#
# ### Second, the classification model
#
# Next we need a model that will sort -- aka _classify_ -- tweets into "fact-checkable" and "not fact-checkable." This will combine the patern-recognition embedded in the language model and examples of both kinds of tweets to make a predition on which class _new_ tweets belong to. This is our **classification model**.
#
#
# ## The Plan
#
# Here's what we're going to do:
#
# - Grab files with a bunch of tweets
# - Make a **language model** from a model pretrained on Wikipedia _plus_ all the tweets as we have
# - Make a **classification model** to predict whether a given tweet is checkable or not, using tweets that were hand-labeled by folks at the Austin American-Statesman.
# - Use that classification model to predict the checkability of unseen tweets.

# ## Credits
#
# This notebook was based on one originally created by <NAME> and the other folks at [fast.ai](https://fast.ai) as part of [this fantastic class](https://course.fast.ai/). Specifically, it comes from Lesson 4. You can [see the lession video](https://course.fast.ai/videos/?lesson=4) and [the original class notebook](https://github.com/fastai/course-v3/blob/master/nbs/dl1/lesson3-imdb.ipynb). 
#
# The idea for the project came from Dan Keemahill at the Austin American-Statesman newspaper. Dan, Madlin Mekelburg, and others at the paper hand-coded the tweets used for the classificaiton training.
#
# For more information about this project, and details about how to use this work in the wild, check out our [Quartz AI Studio blog post about the checkable-tweets project](https://qz.ai/?p=89).
#
# -- <NAME>, [Quartz](https://qz.com), October 2019

# ## Setup

# ### For those using Google Colaboratory ...
#
# Be aware that Google Colab instances are ephemeral -- they vanish *Poof* when you close them, or after a period of sitting idle (currently 90 minutes), or if you use one for more than 12 hours.
#
# If you're using Google Colaboratory, be sure to set your runtime to "GPU" which speeds up your notebook for machine learning:
#
# ![change runtime](https://qz-aistudio-public.s3.amazonaws.com/workshops/notebook_images/change_runtime_2.jpg)
# ![pick gpu](https://qz-aistudio-public.s3.amazonaws.com/workshops/notebook_images/pick_gpu_2.jpg)
#
# Then run this cell:

# +
## ALL GOOGLE COLAB USERS RUN THIS CELL

## This runs a script that installs fast.ai
# !curl -s https://course.fast.ai/setup/colab | bash
# -

# ### For those _not_ using Google Colaboratory ...
#
# This section is just for people who decide to use one of the notebooks on a system other than Google Colaboartory. 
#
# Those people should run the cell below.

## NON-COLABORATORY USERS SHOULD RUN THIS CELL
# %reload_ext autoreload
# %autoreload 2
# %matplotlib inline

# ### Everybody do this ...

# Everyone needs to run the next cell, which initializes the Python libraries we'll use in this notebook.

## AND *EVERYBODY* SHOULD RUN THIS CELL
import warnings
warnings.filterwarnings('ignore')
from fastai.text import *
import fastai
print(f'fastai: {fastai.__version__}')
print(f'cuda: {torch.cuda.is_available()}')

# ## The Data

# We're going to be using two sets of tweets for this project:
#
# - A CSV (comma-separated values file) containing a bunch of #txlege tweets
# - A CSV of #txlege tweets that have been hand-coded as "fact-checkable" or "not fact-checkable"
#

# Run this cell to download the data we'll use for this exercise
# !wget -N https://qz-aistudio-public.s3.amazonaws.com/workshops/austin_tweet_data.zip --quiet
# !unzip -q austin_tweet_data.zip
print('Done!')

# We now have a directory called `data` with two files of tweets. Let's take a look.

# %ls data/

# + [markdown] colab_type="text" id="9TPvnzypbVsq"
# ### Take a peek at the tweet data

# + [markdown] colab_type="text" id="I21aOI9xbVsr"
# Working with <NAME> and <NAME> over a couple of weeks during the 2019 Texas state legislative session, I have have a set of 3,797 tweets humans at the Austin American-Statesman have determined are – or are not – statements that can be fact-checked. 

# + colab={"base_uri": "https://localhost:8080/", "height": 204} colab_type="code" id="kWwuVUg9QHo8" outputId="703590d9-71c2-4d1b-96cf-aa0a113be58a"
# Here I read the csv into a data frame I called `austin_tweets`
# and take a look at the first few rows
data_path = './data/'
hand_coded_tweets = pd.read_csv(data_path + 'hand_coded_austin_tweets.csv')
hand_coded_tweets.head()
# -

# To train the language model, we want a bunch more examples of #txlege tweets. It doesn't matter that we didn't classify them as fact-checkable or not. The language model just needs _examples_ of the kinds of tweets it might see. 
#
# So I used the website [IFTTT](https://ifttt.com) and Google Spreadsheets to collect several days worth of #txlege tweets. That's what's in the  `tweet_corpus.txt` file we looked at.

# + colab={"base_uri": "https://localhost:8080/", "height": 204} colab_type="code" id="18kBCGjhbVvQ" outputId="e36434d1-d47d-44c0-c6ab-82a61843ad2a"
# read in the corpus, which has one tweet per row,
# and take a look at the first frew rows
corpus_tweets = pd.read_csv(data_path + 'tweet_corpus.txt')
corpus_tweets.head()

# + [markdown] colab_type="text" id="eym6ulq0bVuz"
# ## Building the language model

# + [markdown] colab_type="text" id="62pxEwWPbVvI"
# First we need a model that 'understands' the rules of English, and ideally also recognizes patterns in our particular corpus, the `#txlege` tweets. This is the language model. 
#
# We'll start with a language model pretrained on a thousands of Wikipedia articles called [wikitext-103](https://einstein.ai/research/blog/the-wikitext-long-term-dependency-language-modeling-dataset). That language model has been trained to guess the next word in a sentence based on all the previous words. It has a recurrent structure with a hidden state that is updated each time it sees a new word. This hidden state thus contains information about the sentence up to that point.
#
# For our project, we want to infuse the Wikitext model with our particular dataset – the #txlege tweets. Because the English of #txlege tweets isn't the same as the English of Wikipedia, we'll adjust the internal parameters of the model by a little bit. That includes adding words that might be extremely common in the tweets but would be barely present in wikipedia–and therefore might not be part of the vocabulary the model was trained on.

# + [markdown] colab_type="text" id="FT0C6-vpbVvK"
# ### Using all of our tweets for the language model
#
# We want as many tweets for the language model as possible to learn the patterns of #txlege tweets.
#
# We'll start with the text of the 3,797 "hand-coded" tweets (though for the language model, we ignore the checkable/not checkable part of that file).
#
# To that, we'll add the additional tweets I collected in `tweet_corpus.txt`. We'll mush those all into one big file of tweets.

# + colab={"base_uri": "https://localhost:8080/", "height": 34} colab_type="code" id="aQyQIh_vNhrH" outputId="9a6afe30-01c4-4445-d7a7-8c835d7d62fc"
# here I concatenate the two tweet sets into one big set
lm_tweets = pd.concat([hand_coded_tweets,corpus_tweets], sort=True)

# as a sanity check, let's look at the size of each set, 
# and then the ontatenated set
print('hand coded tweets:', len(hand_coded_tweets) )
print('corpus of tweets:', len(corpus_tweets) )
print('total tweets:', len(lm_tweets) )

# + [markdown] colab_type="text" id="g55U1v3mbVvm"
# Great: Now we have 7,485 tweets to use for the language model.
#
# One thing to note ... the first set had two columns, `checkable` and `tweet_text`, while the corpus had just one collumn,  `tweet_text`. The combined has the original two columns, though many of the entries will be `NaN` for "not a number." Thats okay, because we're only going to use the `tweet_text` column for the language model.
#
#
#

# + colab={} colab_type="code" id="pFp92p9NbVvs"
# Saving as csv for easier reading in a moment
lm_tweets.to_csv(data_path + 'lm_tweets.csv', index=False)

# + [markdown] colab_type="text" id="3fU_R9lxbVv7"
# Fast.ai uses a concept called a "[data bunch](https://docs.fast.ai/basic_data.html)" to handle machine-learning data, which takes care of a lot of the more fickle machine-learning data preparation.
#
# We have to use a special kind of data bunch for the language model, one that ignores the labels, and will shuffle the texts at each epoch before concatenating them all together (only the training set gets shuffled; we don't shuffle for the validation set). It will also create batches that read the text in order with targets (aka the best guesses) that are the next word in the sentence.
#

# + colab={} colab_type="code" id="C2rJuWVCbVv5"
# Loading in data with the TextLMDataBunch factory class, using all the defaults
data_lm = TextLMDataBunch.from_csv(data_path, 'lm_tweets.csv', text_cols='tweet_text', label_cols='checkable')
data_lm.save('data_lm_tweets')

# + [markdown] colab_type="text" id="rY0LkJCQbVwF"
# We can then put all of our tweets (now stored in `data_lm`) into a learner object along with the pretrained Wikitext model -- here called `AWD_LTSM`, which is downloaded the first time you'll execute the following line.

# + colab={} colab_type="code" id="EiiXrA6zbVwH"
learn = language_model_learner(data_lm, AWD_LSTM, drop_mult=0.3)

# + [markdown] colab_type="text" id="RfAXUyLnOB_J"
# One of the most important settings when we actually _train_ our model is the **learning rate**. I'm not going to dive into it here (though I encourage you to explore it), but will use a fast.ai tool to find the best learning rate to start with:

# + colab={"base_uri": "https://localhost:8080/", "height": 34} colab_type="code" id="W7b-Z94dbVwL" outputId="f6044993-11ab-4889-c0fe-892243e61a2e"
learn.lr_find()

# + colab={"base_uri": "https://localhost:8080/", "height": 283} colab_type="code" id="Un204GpfbVwO" outputId="607fdc89-064b-427b-e92e-f35bdc868923"
learn.recorder.plot(skip_end=15)

# + [markdown] colab_type="text" id="td3D3mK8TtHa"
# This gives us a graph of the optimal learning rate ... which is the point where the graph really dives downward (`1e-02`). Again, there's much more on picking and learning rates in the fast.ai course.

# + [markdown] colab_type="text" id="BQA3dZWPachE"
# Now we can train the Language Model. (Essentailly, we're training it to be good at guessing the *next word* in a sentence, given all of the previous words.)
#
# The variabales we're passing are `1` to just do one cycle of learning, the learning rate of `1e-2`, and some momentum settings we won't get into here -- but these are pretty safe. 

# + colab={"base_uri": "https://localhost:8080/", "height": 80} colab_type="code" id="pveLnA6kbVwQ" outputId="38ef873c-5c2a-461d-9688-d8f4e1edbd5f"
learn.fit_one_cycle(1, 1e-2, moms=(0.8,0.7))

# + colab={"base_uri": "https://localhost:8080/", "height": 80} colab_type="code" id="o-dYIVFcbVwS" outputId="aa2db805-6429-4592-f905-0479b9872820"
learn.fit_one_cycle(1, 1e-1, moms=(0.8,0.7))


# + [markdown] colab_type="text" id="M8kZOKSybVwX"
# To complete the fine-tuning, we "unfreeze" the original wikitext-103 language model and let our new training efforts work their way into the original neural network.

# + colab={"base_uri": "https://localhost:8080/", "height": 359} colab_type="code" id="kTfuNCuhbVwX" outputId="1e99e6bc-8421-4d63-faf3-4c16fb3f07f6"
# This takes a couple of minutes!
learn.unfreeze()
learn.fit_one_cycle(2, 1e-3, moms=(0.8,0.7))

# + [markdown] colab_type="text" id="fv0mZuVhbVwd"
# While our accuracy may _seem_ low ... in this case it means the language model correctly guessed the next word in a sentence more than 1/3 of the time. That's pretty good! And we can see that even when it's wrong, it makes some pretty "logical" guesses. 
#
# Let's give it a starting phrase and see how it does:
#

# + colab={"base_uri": "https://localhost:8080/", "height": 88} colab_type="code" id="_Wws03kmbVwd" outputId="dd545482-c5c8-4e93-a79e-64578e6478a3"
TEXT = "I wonder if this"
N_WORDS = 40
N_SENTENCES = 3

print("\n\n".join(learn.predict(TEXT, N_WORDS, temperature=0.75) for _ in range(N_SENTENCES)))

# + [markdown] colab_type="text" id="hj3aD0UebVwr"
# Remember, these are not real ... they were _generated_ by the model when it tried to guess each of the next words in the sentence! Generating text like this is not why we made the language model (though you can see where text-generation AI starts from!)
#
# Also note that the model is often crafting the response _in the form of a tweet!_
#
# We now save the model's encoder, which is the mathematical representation of what the language model "understands" about English patterns infused by our tweets.

# + colab={} colab_type="code" id="thkQAeQ2bVwr"
learn.save_encoder('fine_tuned_enc')  

# + [markdown] colab_type="text" id="zt6pDPEDbVww"
# ## Building the classifier model
#
# This is the model that will use our langauge model **and** the hand-coded tweets to guess if new tweets are fact-checkable or not.
#
# We'll create a new data bunch that only grabs the hand-coded tweets and keeps track of the labels there (true or false, for fact-checkability). We also pass in the `vocab` -- which is the list of the most useful words from the language model.

# + colab={} colab_type="code" id="ntIAhZbbbVw1"
data_clas = TextClasDataBunch.from_csv(data_path, 'hand_coded_austin_tweets.csv', vocab=data_lm.vocab, text_cols='tweet_text', label_cols='checkable')

data_clas.save('data_clas_tweets')

# + [markdown] colab_type="text" id="WCwf-DyEbVxA"
# We can then create a model to classify tweets. You can see that in the next two lines we include the processed, hand-coded tweets (`data_clas`), the original Wikitext model (`AWD_LSTM`), and the knowledge we saved after infusing the language model with tweets (`fine_tuned_enc`).

# + colab={} colab_type="code" id="xlJKU0g3bVxA"
learn = text_classifier_learner(data_clas, AWD_LSTM, drop_mult=0.5)
learn.load_encoder('fine_tuned_enc');

# + [markdown] colab_type="text" id="Az5OuN6_hJaF"
# With neural networks, there are lots of tweaks you can adjust — known as "hyperparameters" — such as learning rate and momentum. The fast.ai defaults are pretty great, and the tools it has for finding the learning rate are super useful. I'm going to skip those details here for now. There's more to learn at [qz.ai](https://qz.ai) or at the [this great fast.ai course](https://course.fast.ai/).

# + colab={"base_uri": "https://localhost:8080/", "height": 80} colab_type="code" id="S5g2xmtlbVxJ" outputId="79971cc2-dde2-4dfc-a340-7dcae79f2bdd"
learn.fit_one_cycle(2, 1e-2, moms=(0.8,0.7))

learn.freeze_to(-2)
learn.fit_one_cycle(2, slice(1e-2/(2.6**4),1e-2), moms=(0.8,0.7))

learn.freeze_to(-3)
learn.fit_one_cycle(2, slice(5e-3/(2.6**4),5e-3), moms=(0.8,0.7))

# + [markdown] colab_type="text" id="J-TSQ7iDTdIG"
# Let's give it an example ....

# + colab={"base_uri": "https://localhost:8080/", "height": 34} colab_type="code" id="mBAXEa2MbVxa" outputId="4163f4ee-f9e0-4d93-b5a2-c47e38a78205"
example = "Four states have two universities represented in the top 20 highest-paid executives of public colleges. Texas has SIX"
learn.predict(example)

# + [markdown] colab_type="text" id="8u1ptJu1UmT9"
# `True` means checkable! 
# -

# We can open the "black box" a little to see what words the model is keying into.

interp = TextClassificationInterpretation.from_learner(learn) 
interp.show_intrinsic_attention(example)

# Let's save our work.

# + [markdown] colab_type="text" id="XSOgwF1ybVyh"
# ## Saving to Google Drive
#
# At present, your Google Colaboratory Notebook disappears when you close it — along with all of your data. If you'd like to save your model to your Google Drive, run the following cell and grant the permissions it requests.
# -

from google.colab import drive
drive.mount('/content/gdrive', force_remount=True)
root_dir = "/content/gdrive/My Drive/"
base_dir = root_dir + 'ai-workshop/checkable_tweet_models/'
save_path = Path(base_dir)
save_path.mkdir(parents=True, exist_ok=True)

# The next line will save everything we need for predictions to a file to your Google Drive in the `ai-workshops` folder.

learn.export(save_path/"export.pkl")

# Later, to load the model into your code, connect to your Google drive using the same block above that starts `from google.colab import drive ...` and then run this:
#     

# load the model from the 'export.pkl' file on your Google Drive
learn = load_learner(save_path)  

# + [markdown] colab_type="text" id="57uA-44kVh_T"
# We used a model built this way to classify #txlege tweets as they were tweeted. For details about deploying a predictor in the cloud using Render, see our [blog post about building the checkable-tweets project](https://qz.ai/?p=89).

StarcoderdataJupyterScriptsDedupFiltered