import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
import seaborn as sns;

"""
Explanation: HashStream: Tutorial
To start, let's import some packages to help us. If you haven't read README.md, you should go do that now. I highly recommend Continuum Analytics Anaconda distribution of python, and the conda architecture for finding/installing co-dependent packages. Go explore!
End of explanation
"""


from src.preprocess import Preprocess

pre = Preprocess('./tweet_input/tweets.txt')
pre.extract()

"""
Explanation: Preprocessing tweet data
Now, all we need to start collecting tweet data is the preprocess module, which contains the Preprocess class for extracting hashtags and timestamps from raw tweet data. 
Once that's done, all you need to create the preprocess object is to specify the file it will monitor. This object is tied to that file...whether it will be a static or dynamic source of raw twee data
End of explanation
"""


pre.extract(overwrite=False)

"""
Explanation: Let's say that file got updated with fresh tweets at the end, but still had the old tweets inside. If we extract with the overwrite=False flag, the Preprocessing object tracks where it left off, and skips ahead to ONLY read in new tweets. This let's us use the preprocessor even on live-streamed data. HashStream is very scalable!
Since the example tweets.txt file hasn't changed, there shouldn't be anything added:
End of explanation
"""


print len(pre.df.index)
pre.df.head()

"""
Explanation: Another important feature of the preprocessor is it's use of Pandas, which gives us an R style dataframe for manipulation as we please...and its handling of timestamps and slicing is amazing! Access this dataframe with Preprocess.df:
End of explanation
"""


np.mean([len(i) for i in pre.df.hashtags])

"""
Explanation: If we want the average number of hashtags for all tweets in storage, that's easy!
End of explanation
"""


import src.analysis as hs

hs.__all__

"""
Explanation: Analyzing the data
This is where things get cool. use import src.analysis to get access to the main toolbox in HashStream, currently called analysis. Let's import as hs, and check out what the __all__ keyword would give us (i.e. the functions from src.analysis import * would return)
End of explanation
"""


tweet_no = 4

g = hs.get_graphs(pre.df, tweet_no,tweet_no+1, window=10.)
print len(g)
hs.draw_lifted(g[0])

"""
Explanation: Remember that we are building graphs from the shared hashtags among tweets...an edge between hastags means they occured together (in the same tweet) at some point in the time window . 
The get_graphs() function will do just that. Specify a starting tweet and an ending tweet (for 1 tweet, --> start, start+1). None of the tweets before No. 4 returned any hashtags so let's see what tweet 4 gives us:
End of explanation
"""


tweet_no = 4+100

g = hs.get_graphs(pre.df, tweet_no,tweet_no+1, window=10.)
print len(g)
hs.draw_lifted(g[0])

"""
Explanation: Cool, huh? All are connected because they all were in the 4th tweet. Note the draw_lifted() function, which draws the graph with labels 'lifted' above the node. THIS NEEDS MATPLOTLIB!
Now lets add 100 tweets and see what's what.
End of explanation
"""


graph_gen = hs.rolled_graph_gen(pre.df, start=8000, window=10.)

"""
Explanation: Ok, so what's really going on under the hood, is a python generator that calculated the proper time window and graph structure on the fly. the rolled_graph_gen() function is the direct way to make a custom generator. 
Start and end at whatever tweet you like (defaults are beginning and ending of the input dataframe). Then pick your time window to smooth the graphs a bit. I like 10 seconds, so let's check out what happens after the 8000th tweet:
End of explanation
"""


# quick function to extract the timestamp for each graph
get_times = lambda g: g.graph['time'] 

X = hs.g_stats(graph_gen, hs.mean_deg, get_times)

"""
Explanation: Cool. Now, the g_stats() function will calculate any statistics you want, by passing in functions that can take NetworkX graphs as input and output a value. Put as many as you like, and the statistics are saved as columns in the output array. 
Also, using a savename='path/to/file.txt' kwarg will creat a .txt to save your desired statistics! Hashstream has a mean_deg() utility function to calculate the mean degree of the hashtag nodes in a graph, but it's easy to define and use your own.
End of explanation
"""


degrees, times = X[:,0], X[:,1]

plt.plot(degrees)

"""
Explanation: If tqdm is installed, you should get a nice looking progress bar as the generator creates graphs for the stats function on the fly. If not, wait a little bit until it finishes. 
Now, it's easy to see what the last thousand or so tweets are doing.
End of explanation
"""


plt.plot(times, degrees, '.')
plt.xticks(rotation=20)
# X.shape

"""
Explanation: Since this is built on Pandas dataframes, it's also easy to plot vs. time! Since our dataset is only resolved to seconds, there will of course be some overlap, so we'll use dots instead.
End of explanation
"""


sns.distplot(degrees)

"""
Explanation: It's easy to check out the distributions of the data with packages like Seaborn or your plotting package of choice.
End of explanation
"""


graph_list = hs.get_graphs(pre.df, window=10.)

"""
Explanation: Static data crunching
If you have all the tweets you want, it may be better to just save all of the graph objects you'll need in one go. The generator makes this easy, but expect to wait a while the first time around.
End of explanation
"""


hs.draw_lifted(graph_list[0])

"""
Explanation: Now we can directly access the graph objects and all of their attributes at once! For example, the first graph recorded (which we saw before).
End of explanation
"""


X = hs.g_stats(graph_list, hs.mean_deg, get_times)
degrees, times = X[:,0], X[:,1]

g = graph_list[np.argmax([degrees])]
print g.degree().values()

hs.draw_lifted(g)

g = graph_list[np.argmin([degrees])]
print g.degree().values()

hs.draw_lifted(g)

"""
Explanation: Now, we can pass the graph list (instead of a generator) into the stats function. What does the MOST connected (on average) graph look like? The LEAST?
End of explanation
"""


