%matplotlib inline

"""
Explanation: This notebook explores how collaborative relationships form between mailing list participants over time.
The hypothesis, loosely put, is that early exchanges are indicators of growing relationships or trust that should be reflected in information flow at later times.
End of explanation
"""


from bigbang.archive import Archive
import bigbang.parse as parse
import bigbang.graph as graph
import bigbang.mailman as mailman
import bigbang.process as process
import bigbang.twopeople as twoppl
import matplotlib.pyplot as plt
import networkx as nx
import numpy as np
import pandas as pd
from pprint import pprint as pp
import pytz
import math

"""
Explanation: Next we'll import dependencies.
End of explanation
"""


url = "http://mail.python.org/piperpmail/scipy-dev/"
arx= Archive(url,archive_dir="../archives")

arx.data.irow(0).Body

arx.data.shape

"""
Explanation: Let's begin with just one mailing list to simplify.
End of explanation
"""


arx.data[arx.data['In-Reply-To'] > 0][:10]

"""
Explanation: Let's look at the matrix of who replies to whom over the whole history of the list, to get a sense of the overall distribution
End of explanation
"""


messages = arx.data[['From']]

responses = arx.data[arx.data['In-Reply-To'] > 0][['From','Date','In-Reply-To']]

exchanges = pd.merge(messages,responses,how='inner',right_on='In-Reply-To',left_index=True,suffixes=['_original','_response'])

exchanges

exchanges.groupby(['From_original','From_response']).count()

"""
Explanation: In particular we are interested in who replied to who at each time. Recall that this is an open mailing list--everybody potentially reads each message. A response from A to B is an indication that A read B's original message. Therefore, a response indicates not just a single message from A to B, but an exchange from B to A and back again.
Below we modify our data to see who replied to whom.
End of explanation
"""


twoppl.duration(exchanges, "oliphant at ee.byu.edu (Travis Oliphant)", "rossini at blindglobe.net (A.J. Rossini)"  )
twoppl.panda_pair(exchanges, "oliphant at ee.byu.edu (Travis Oliphant)", "rossini at blindglobe.net (A.J. Rossini)"  )

"""
Explanation: The next step is to create a DataFrame that for each pair A and B:
* The duration of time between the first reply between that pair and the last.
* The total number of replies from A to B, $r_{AB}$, and from B to A, $r_{BA}$.
* The reciprocity of the conversation $min(r_{AB},r_{BA})/max(r_{AB},r_{BA})$
Using the exchanges data frame we can use the functions in twopeople.py to calculate the above quantities. The cell below gives a sample output for calculating duration, number of replies, and reciprocity for two specific contributers.
End of explanation
"""


twoppl.duration(exchanges, "oliphant at ee.byu.edu (Travis Oliphant)", "rossini at blindglobe.net (A.J. Rossini)"  )

"""
Explanation: The "duration" column gives us a datetime data structure so we must be careful later on when extracting information from this column.
End of explanation
"""


pairs = twoppl.unique_pairs(exchanges)
pairs

"""
Explanation: Now let's create a dataframe that consists of these three quantities (duration, number of replies, and reciprocity) for each pair of contributers who interacted. 
To get all the unique pairs we can use the unique_pairs function in twopeople.py. This willl give us all pairs between any two individuals who had some degree of interation.
Note: The unique pairs we get back do not include reversed pairs. For example, if one of the pairs was ("Bob", "Mary"), we would not have ("Mary", "Bob") in our output.
End of explanation
"""


allpairs = twoppl.panda_allpairs(exchanges, pairs)
allpairs

"""
Explanation: Using panda_allpairs, we can create the desired data frame
End of explanation
"""


print "corresponding counts for number of replies up to 19"
print ("number of replies", "frequency of occurence")
for i in range(20):
    print (i, len(allpairs[allpairs['num_replies'] <= i]) - len(allpairs[allpairs['num_replies'] <= i - 1]))

plt.hist(allpairs['num_replies'])
plt.title("Number of replies")

"""
Explanation: The printed values below display the counts associated for a given number of replies.
We can see  that many of the interactions have a relatively small amount of replies. One possible explanation for this large amount may be having many interactions that are merely a follow-up or question followed by a thank you response (we can explore this further by parsing the message bodies in arx.data but for now we will just speculate).
End of explanation
"""


greaterThanFive = allpairs[allpairs['num_replies'] > 5]['num_replies']
counts = greaterThanFive.value_counts()
counts.plot()

"""
Explanation: To get a better idea of what's going on for larger values, let's look at the rows that have number of replies greater than 5.
End of explanation
"""


#Completely arbitrary bins

#Group A reciprocity between (0, .25]
#Group B reciprocity between (.25, .5] 
#Group C reciprocity between (.5, .75]  
#Group D reciprocity between (.75, 1.00]

#"low" number of replies less than or equal to 10
#"moderate" number of replies between 10 and 20
#"high" replies greater than 20 replies 

def genId(num):
    if num <= .25:
        return 'A.(0, .25]'
    if num <= .5:
        return "B.(.25, .5]"
    if num <= .75:
        return "C.(.5, .75]"
    return "D.(.75, 1.00]"

def genNumReplies(num):
    if num <= 10:
        return 'a.low'
    if num <= 20:
        return "b.moderate"
    return "c.high"

"""
Explanation: The graph seems to follow a power law which is expected for this type of data. 
Now let's see if we can find any patters between the number of replies and reciprocity. Intuitively, we would expect that the number of replies be positively associated with reciprocity but let's see... 
We will first look at the data for which the number of replies is greater than 5 to possibly get rid of some noise (later on we will explore the data without removing these entries). 
Below we divided reciprocity and number of replies into completely arbitrary bins as shown below. Hopefully, this will make it easier to see patters between these two variables as they have quite a bit of noise. 
(The genId and genNumReplies functions just give each entry a corresponding label for graphing purposes later on. These labels are based on which bin a given entry falls under)
End of explanation
"""


moreThanFive = allpairs[allpairs['num_replies'] > 5]
recipVec = moreThanFive['reciprocity']
numReplies = moreThanFive['num_replies']
ids = recipVec.apply(lambda val: genId(val))
groupedRep = numReplies.apply(lambda val: genNumReplies(val))
groupsdf = pd.DataFrame({"num_replies": numReplies, "ids": ids, "grouped_num_replies": groupedRep})
groupsdf


"""
Explanation: The following lines generate a data frame that contains three columns:
1) Number of replies
2) Id corresponding to replies bin
3) Id corresponding to reciprocity bin
(The extra letters such as the a in "a.low" are just used so that pandas orders the columns in the way we want)
End of explanation
"""



grpA = groupsdf[groupsdf["ids"] == "A.(0, .25]"]['num_replies']
grpB = groupsdf[groupsdf["ids"] == "B.(.25, .5]"]['num_replies']
grpC = groupsdf[groupsdf["ids"] == "C.(.5, .75]"]['num_replies']
grpD = groupsdf[groupsdf["ids"] == "D.(.75, 1.00]"]['num_replies']
grpA.value_counts().hist()
plt.title("Number of Replies for Reciprocity between 0-.25")

grpB.value_counts().hist()
plt.title("Number of Replies for Reciprocity between .25-.5")

grpC.value_counts().hist()
plt.title("Number of Replies for Reciprocity between .5-.75")

grpD.value_counts().hist()
plt.title("Number of Replies for Reciprocity between .75-1.0")

"""
Explanation: Now that we have this data frame lets look at the corresponding histograms for each "level" of reciprocity.
End of explanation
"""


crossed = pd.crosstab(groupsdf["grouped_num_replies"], groupsdf["ids"])
crossed

crossed.plot()

"""
Explanation: It's pretty hard to compare the four histograms so let's create a contingency table for the groupsdf data frame.
End of explanation
"""


crossed.apply(lambda r: r/sum(r), axis=0)

"""
Explanation: Since each reciprocity group has a different amount of counts, let's normalize the counts to get a better picture of what's going on. 
We will first normzalize column-wise, that is for say column A.[0,.25] we will sum the total number of responses and get the relative proportions for the replies bins.
End of explanation
"""


crossed.apply(lambda r: r/sum(r), axis=0).plot()
plt.title("normalized (columnwise) plot")

"""
Explanation: We see that at the very extremes, namely reciprocity between 0-.25 and reciprocity between .75-1.0, there are some clear differences; reciprocity seems to be positively associated with the number of replies as we had initially expected.
On the other hand, the bin for reciprocity between .25-.5 weakens this association as this bin seems as if it should swap positions with bin A. However, since the bin widths we chose were completely arbitrary it may explain this paradox. 
The fact that the extremes seem to follow our expectations is quite interesting; it provides some evidence that if we choose our bin sizes appropriately, we can perhaps get a nice positive association.
End of explanation
"""


crossed.apply(lambda r: r/sum(r), axis=1)

crossed.apply(lambda r: r/sum(r), axis=1).plot()
plt.title("normalized (row-wise) plot")

"""
Explanation: Now will do the normalization by row. This will give us the relative proportion of some bin for number of replies is distributed across the bins for reciprocity.
End of explanation
"""


recipVec2 = allpairs['reciprocity']
numReplies2 = allpairs['num_replies']
ids = recipVec2.apply(lambda val: genId(val))
groupedRep2 = numReplies2.apply(lambda val: genNumReplies(val))
groupsdf2 = pd.DataFrame({"num_replies": numReplies2, "ids": ids, "grouped_num_replies": groupedRep2})

crossed2 = pd.crosstab(groupsdf2["grouped_num_replies"], groupsdf2["ids"])
crossed2

crossed2.plot()

crossed2.apply(lambda r: r/sum(r), axis=0)

crossed2.apply(lambda r: r/sum(r), axis=0).plot()

"""
Explanation: Now let's go back and do the exact same thing but not removing entries with a very low amount of replies.
End of explanation
"""


plt.scatter(allpairs.num_replies, allpairs.reciprocity)
plt.title("number of replies vs. reciprocity")

allpairs['duration'] = allpairs['duration'].apply(lambda x: x.item() / pow(10,9))
plt.scatter(allpairs.duration, allpairs.num_replies)
plt.title("duration vs. number of replies")

"""
Explanation: Now we will look at various scatterplots for different variables to get a rough sense of how our data is spread.
End of explanation
"""


df_filt = allpairs[allpairs['num_replies'] > 5]
plt.scatter(df_filt.reciprocity, df_filt.duration)
plt.title("reciprocity vs. duration")

plt.scatter(df_filt.reciprocity, df_filt.duration.apply(lambda x: math.log(x)))
plt.title("reciprocity vs. log of duration")

plt.scatter(df_filt.duration.apply(lambda x: math.log(x+1)), df_filt.num_replies.apply(lambda x: math.log(x+1)))
plt.title("log of duration vs. log of number of replies")

"""
Explanation: Now let's look at some scatterplots for the entries with number of replies greater than 5.
End of explanation
"""


def calMessageLen(message):
    if message == None:
        return 0
    return len(message)

arx.data['length'] = arx.data['Body'].apply(lambda x: calMessageLen(x))
arx.data

"""
Explanation: Since we actually have the bodies of each message, we will now procceed by seeing if there are any patterns between the type of messages sent and reciprocity, duration, and the number of replies. 
As a very rough measure, we have created a function calMessageLen that calculates the length of a given message.
End of explanation
"""


