# first we want some Python tools to make our lives easier

import pandas as pd
import numpy as np
import networkx as nx
import matplotlib.pyplot as plt

%matplotlib inline

with open("20170718 hashtag_damd uncleaned.csv") as fd:
    for row in fd.readlines()[:3]:
        print(row)

"""
Explanation: DAMD hashtag cooccurrence graph (re)construction
Let's look at the shape of the data about DAMD and see how to computationally construct a graph, and how that compares to doing so with an interactive tool, such as Table 2 Net.

Reading the data from a file into Python
Ok we have received a nice data file. First we can take a look with Excel, or a text editor. Assuming the file 20170718 hashtag_damd uncleaned.csv has been placed in the same directory as this notebook, we can also take a peek in Python.
End of explanation
"""


damd = pd.read_csv("20170718 hashtag_damd uncleaned.csv")

"""
Explanation: That looks like a comma-separated value (CSV) file. There are many other kinds of files for data, but these are quite typical. In a CSV, each line is a data item (a tweet in this case), and columns are variables for each item. We call such a thing a data frame.
End of explanation
"""


damd.columns

"""
Explanation: What variables do we have?
End of explanation
"""


damd = pd.read_csv("20170718 hashtag_damd uncleaned.csv", index_col="tweet_id")
damd.head(3)

"""
Explanation: Let's decide to use the tweet_id as index. It is an unique identifier for the tweets.
End of explanation
"""


damd.hashtags.head()

"""
Explanation: Hashtag co-occurrence graph creation
To find patterns in the data, we might look at #hashtags, and if we can identify some interesting patterns in them. Cooccurrence is a useful thing to look at, and can easily be done in Twitter data.
We might want to bipartite graph ("network") $g = \langle N, V \rangle$, where $N = {{node}_1, {node}_2 \ldots {node}_n}$ is a set of nodes ("spheres"), and $V = {{\langle source, target \rangle_1, \langle source, target \rangle _2 \ldots \langle source, target \rangle _m }}$ set of edges ("lines") of tweets and hashtags, to analyze hashtag co-occurrence.
A bipartite graph has two types of nodes, which are not connected within the type, only across. In our case, hashtags are connected to tweets, but tweets are not directly connected to tweets, and hashtags are not directly connected to hashtags. Makes sense, right?
This data manipulation process can be done with Table 2 Net. But doing so programmatically is a different way to do it. We will use Python library called NetworkX.
Below is a Gephi visualization of a graph made with Table 2 Net, coloured by node type <span style='color: salmon;'>red for tweets</span> and <span style='color: lightgreen;'>green for hashtags</span>, and showing labels for the hashtag nodes with degree of 15 or larger. We have used the algorithm ForceAtlas2 in Gephi for positioning the nodes. The central node, hashtag damd has been hidden, because it carries no information.

First let's take a peek at the shape of the hashtags, how they are stored in the data we have received.
End of explanation
"""


def buildHashtagCooccurrenceGraph(tweets):
    g = nx.Graph(name="Hashtag co-occurrence bipartite")
    for tweet, hashtags in damd.hashtags.astype(str).map(lambda l: l.split(';')).items():
        g.add_node(tweet, Type="tweet_id")
        for hashtag in hashtags:
            g.add_edge(tweet, hashtag.lower())
    return g

g = buildHashtagCooccurrenceGraph(damd)

"""
Explanation: We see that the hashtag column is itself a semicolon separated list, and our data is kind of three dimensional. We need to split it up.
From reading the documentation, we know that nx.Graph.add_edge() requires input as a tuple (source, target), describing one edge. For each tweet, we generate a list of it's hashtags, and then add those edges to the graph one by one. So, from the original data shape
tweet1 hashtag1;hashtag2;hashtag3
tweet2 hashtag9;hashtag4
.
.
.

We create an intermediary data shape for line 5
tweet1 hashtag1
tweet1 hashtag2
tweet1 hashtag3
tweet2 hashtag9
tweet2 hashtag4
.
.
.

This suits what the NetworkX API expects.
Conveniently NetworkX automatically creates the nodes, so we don't have to think about them. How can it automatically know what the nodes are, if it only looks at links?
End of explanation
"""


print(nx.info(g))

"""
Explanation: Now, let's briefly inspect the graph g we created.
End of explanation
"""


nx.write_gexf(g, "hashtag-cooccurrence-bipartite-with-python.gexf")

"""
Explanation: Save to file, for opening in Gephi.
End of explanation
"""


g_table2net = nx.read_gexf("hashtag-cooccurrence-bipartite-with-table2net.gexf")
print(nx.info(g_table2net))

"""
Explanation: Compare the results of graph creation with Table 2 Net and Python
Read in the graph made with Table 2 Net.
End of explanation
"""


# This algoritm is not guaranteed, but it is fast
nx.isomorphism.fast_could_be_isomorphic(g, g_table2net)

"""
Explanation: After poking around in Gephi for half an hour setting colours and filters, positioning with ForceAtlas2 and outputting an image, here is a visualization of the graph. It should be equal to the one above, which was visualized from a graph constructed from the data with Table 2 Net.

In graph theory, "isomorphism" (ἴσος isos "equal", and μορφή morphe "form" or "shape") means that graphs are of the same shape. Why do want to know this? We want to inspect if we successfully reproduced the process that Table 2 Net did.
End of explanation
"""


