texts = [
    "Penny bought bright blue fishes.",
    "Penny bought bright blue and orange fish.",
    "The cat ate a fish at the store.",
    "Penny went to the store. Penny ate a bug. Penny saw a fish.",
    "It meowed once at the bug, it is still meowing at the bug and the fish",
    "The cat is at the store. The cat is orange. The cat is meowing at the fish.",
    "Penny is a fish"
]

"""
Explanation: Analyzing text!
Text analysis has a few parts. We are going to use bag of words analysis, which just treats a sentence like a bag of words - no particular order or anything. It's simple but it usually gets the job done adequately.
Here is our text.
End of explanation
"""


Penny bought bright blue fishes

tokenized - penny bought bright blue fishes

"Penny bought bright blue fishes".split()

"""
Explanation: When you process text, you have a nice long series of steps, but let's say you're interested in three things:

Tokenizing converts all of the sentences/phrases/etc into a series of words, and then it might also include converting it into a series of numbers - math stuff only works with numbers, not words. So maybe 'cat' is 2 and 'rug' is 4 and stuff like that.
Counting takes those words and sees how many there are (obviously) - how many times does meow appear?
Normalizing takes the count and makes new numbers - maybe it's how many times meow appears vs. how many total words there are, or maybe you're seeing how often meow comes up to see whether it's important.
End of explanation
"""


from sklearn.feature_extraction.text import CountVectorizer
count_vectorizer = CountVectorizer()

# .fit_transfer TOKENIZES and COUNTS
X = count_vectorizer.fit_transform(texts)

"""
Explanation: The scikit-learn package does a ton of stuff, some of which includes the above. We're going to start by playing with the CountVectorizer.
End of explanation
"""


X

"""
Explanation: Let's take a look at what it found out!
End of explanation
"""


X.toarray()

"""
Explanation: Okay, that looks like trash and garbage. What's a "sparse array"??????
End of explanation
"""


pd.DataFrame(X.toarray())

"""
Explanation: If we put on our Computer Goggles we see that the first sentence has the first word 3 times, the second word 1 time, the third word 1 time, etc... But we can't read it, really. It would look nicer as a dataframe.
End of explanation
"""


# A fish is Penny
count_vectorizer.get_feature_names()

pd.DataFrame(X.toarray(), columns=count_vectorizer.get_feature_names())

"""
Explanation: What do all of those numbers mean????
End of explanation
"""


# We'll make a new vectorizer
count_vectorizer = CountVectorizer(stop_words='english') 

# .fit_transfer TOKENIZES and COUNTS
X = count_vectorizer.fit_transform(texts)
print(count_vectorizer.get_feature_names())

pd.DataFrame(X.toarray(), columns=count_vectorizer.get_feature_names())

"""
Explanation: So sentence #4 has "at" once, and the first sentence has "bought" once, and the last sentence has "the" three times. But hey, those are garbage words! They're cluttering up our dataframe! We need to add stopwords!
End of explanation
"""


# This is what our normal tokenizer looks like
def boring_tokenizer(str_input):
    words = re.sub(r"[^A-Za-z0-9\-]", " ", str_input).lower().split()
    return words

count_vectorizer = CountVectorizer(stop_words='english', tokenizer=boring_tokenizer)
X = count_vectorizer.fit_transform(texts)
print(count_vectorizer.get_feature_names())

"""
Explanation: I still see meowed and meowing and fish and fishes - they seem the same, so let's lemmatize/stem them.
You can specify a preprocessor or a tokenizer when you're creating your CountVectorizer to do custom stuff on your words. Maybe we want to get rid of punctuation, lowercase things and split them on spaces (this is basically the default). preprocessor is supposed to return a string, so it's a little easier to work with.
End of explanation
"""


from nltk.stem.porter import PorterStemmer #it doesn't know what words are, it just chop off the ends from the words
porter_stemmer = PorterStemmer()
# 
print(porter_stemmer.stem('fishes'))
print(porter_stemmer.stem('meowed'))
print(porter_stemmer.stem('oranges'))
print(porter_stemmer.stem('meowing'))
print(porter_stemmer.stem('organge'))

porter_stemmer = PorterStemmer()

def stemming_tokenizer(str_input):
    words = re.sub(r"[^A-Za-z0-9\-]", " ", str_input).lower().split()
    words = [porter_stemmer.stem(word) for word in words]
    return words

count_vectorizer = CountVectorizer(stop_words='english', tokenizer=boring_tokenizer)
X = count_vectorizer.fit_transform(texts)
print(count_vectorizer.get_feature_names())

"""
Explanation: We're going to use one that features a STEMMER - something that strips the endings off of words (or tries to, at least). This one is from nltk.
End of explanation
"""


pd.DataFrame(X.toarray(), columns=count_vectorizer.get_feature_names())

"""
Explanation: Now lets look at the new version of that dataframe.
End of explanation
"""


pd.DataFrame(X.toarray(), columns=count_vectorizer.get_feature_names())

"""
Explanation: TF-IDF
Part One: Term Frequency
TF-IDF? What? It means term frequency inverse document frequency! It's the most important thing. Let's look at our list of phrases

Penny bought bright blue fishes.
Penny bought bright blue and orange fish.
The cat ate a fish at the store.
Penny went to the store. Penny ate a bug. Penny saw a fish.
It meowed once at the fish, it is still meowing at the fish. It meowed at the bug and the fish.
The cat is fat. The cat is orange. The cat is meowing at the fish.
Penny is a fish

If we're searching for the word fish, which is the most helpful phrase?
End of explanation
"""


from sklearn.feature_extraction.text import TfidfVectorizer

tfidf_vectorizer = TfidfVectorizer(stop_words='english', tokenizer=stemming_tokenizer, use_idf=False, norm='l1')
X = tfidf_vectorizer.fit_transform(texts)
pd.DataFrame(X.toarray(), columns=tfidf_vectorizer.get_feature_names())

"""
Explanation: Probably the one where fish appears three times.
It meowed once at the fish, it is still meowing at the fish. It meowed at the bug and the fish.

But are all the others the same?
Penny is a fish.

Penny went to the store. Penny ate a bug. Penny saw a fish.

In the second one we spend less time talking about the fish. Think about a huge long document where they say your name once, versus a tweet where they say your name once. Which one are you more important in? Probably the tweet, since you take up a larger percentage of the text.
This is term frequency - taking into account how often a term shows up. We're going to take this into account by using the TfidfVectorizer in the same way we used the CountVectorizer.
End of explanation
"""


tfidf_vectorizer = TfidfVectorizer(stop_words='english', tokenizer=stemming_tokenizer, use_idf=False, norm='l1')
X = tfidf_vectorizer.fit_transform(texts)
df = pd.DataFrame(X.toarray(), columns=tfidf_vectorizer.get_feature_names())
df

"""
Explanation: Now our numbers have shifted a little bit. Instead of just being a count, it's the percentage of the words.
value = (number of times word appears in sentence) / (number of words in sentence)

After we remove the stopwords, the term fish is 50% of the words in Penny is a fish vs. 37.5% in It meowed once at the fish, it is still meowing at the fish. It meowed at the bug and the fish..

Note: We made it be the percentage of the words by passing in norm="l1" - by default it's normally an L2 (Euclidean) norm, which is actually better, but I thought it would make more sense using the L1 - a.k.a. terms divided by words -norm.

So now when we search we'll get more relevant results because it takes into account whether half of our words are fish or 1% of millions upon millions of words is fish. But we aren't done yet!
Part Two: Inverse document frequency
Let's say we're searching for "fish meow"
End of explanation
"""


# Just add the columns together
pd.DataFrame([df['fish'], df['meow'], df['fish'] + df['meow']], index=["fish", "meow", "fish + meow"]).T

"""
Explanation: What's the highest combined? for 'fish' and 'meow'?
End of explanation
"""


# use_idf=True is default, but I'll leave it in,idf inverse document frequency
idf_vectorizer = TfidfVectorizer(stop_words='english', tokenizer=stemming_tokenizer, use_idf=True, norm='l1')
X = idf_vectorizer.fit_transform(texts)
idf_df = pd.DataFrame(X.toarray(), columns=idf_vectorizer.get_feature_names())
idf_df

"""
Explanation: Indices 4 and 6 (numbers 5 and 7) are tied - but meow never even appears in one of them!
It meowed once at the bug, it is still meowing at the bug and the fish
Penny is a fish

It seems like since fish shows up again and again it should be weighted a little less - not like it's a stopword, but just... it's kind of cliche to have it show up in the text, so we want to make it less important.
This is inverse term frequency - the more often a term shows up across all documents, the less important it is in our matrix.
End of explanation
"""


# OLD dataframe
pd.DataFrame([df['fish'], df['meow'], df['fish'] + df['meow']], index=["fish", "meow", "fish + meow"]).T

# NEW dataframe
pd.DataFrame([idf_df['fish'], idf_df['meow'], idf_df['fish'] + idf_df['meow']], index=["fish", "meow", "fish + meow"]).T

"""
Explanation: Let's take a look at our OLD values, then our NEW values, just for meow and fish.
End of explanation
"""


# use_idf=True is default, but I'll leave it in
l2_vectorizer = TfidfVectorizer(stop_words='english', tokenizer=stemming_tokenizer, use_idf=True)
X = l2_vectorizer.fit_transform(texts)
l2_df = pd.DataFrame(X.toarray(), columns=l2_vectorizer.get_feature_names())
l2_df

# normal TF-IDF dataframe
pd.DataFrame([idf_df['fish'], idf_df['meow'], idf_df['fish'] + idf_df['meow']], index=["fish", "meow", "fish + meow"]).T

# L2 norm TF-IDF dataframe
pd.DataFrame([l2_df['fish'], l2_df['meow'], l2_df['fish'] + l2_df['meow']], index=["fish", "meow", "fish + meow"]).T

"""
Explanation: Notice how 'meow' increased in value because it's an infrequent term, and fish dropped in value because it's so frequent.
That meowing one (index 4) has gone from 0.50 to 0.43, while Penny is a fish (index 6) has dropped to 0.40. Now hooray, the meowing one is going to show up earlier when searching for "fish meow" because fish shows up all of the time, so we want to ignore it a lil' bit.
But honestly I wasn't very impressed by that drop.
And this is why defaults are important: let's try changing it to norm='l2' (or just removing norm completely).
End of explanation
"""


# Initialize a vectorizer
vectorizer = TfidfVectorizer(use_idf=True, tokenizer=boring_tokenizer, stop_words='english')
X = vectorizer.fit_transform(texts) #fit_transform

# KMeans clustering is a method of clustering.
from sklearn.cluster import KMeans

number_of_clusters = 2
km = KMeans(n_clusters=number_of_clusters)
km.fit(X)

print("Top terms per cluster:")
order_centroids = km.cluster_centers_.argsort()[:, ::-1]
terms = vectorizer.get_feature_names()
for i in range(number_of_clusters):
    top_ten_words = [terms[ind] for ind in order_centroids[i, :5]]
    print("Cluster {}: {}".format(i, ' '.join(top_ten_words)))

results = pd.DataFrame()
results['text'] = texts
results['category'] = km.labels_
results

"""
Explanation: LOOK AT HOW IMPORTANT MEOW IS. Meowing is out of this world important, because no one ever meows.
Who cares? Why do we need to know this?
When someone dumps 100,000 documents on your desk in response to FOIA, you'll start to care! One of the reasons understanding TF-IDF is important is because of document similarity. By knowing what documents are similar you're able to find related documents and automatically group documents into clusters.
For example! Let's cluster these documents using K-Means clustering (check out this gif)
2 categories of documents
End of explanation
"""


from sklearn.cluster import KMeans

number_of_clusters = 4
km = KMeans(n_clusters=number_of_clusters)
km.fit(X)

print("Top terms per cluster:")
order_centroids = km.cluster_centers_.argsort()[:, ::-1]
terms = vectorizer.get_feature_names()
for i in range(number_of_clusters):
    top_ten_words = [terms[ind] for ind in order_centroids[i, :5]]
    print("Cluster {}: {}".format(i, ' '.join(top_ten_words)))

results = pd.DataFrame()
results['text'] = texts
results['category'] = km.labels_
results

"""
Explanation: 4 categories of documents
End of explanation
"""


