df = pd.read_csv('../data/date_fixed_running_data.csv')

df.head()

"""
Explanation: Alright now we’re going to be working with the running data set. so let’s go ahead and import it. We’ll see again that it’s not converting the dates so we’ve got to do that manually. However it doesn’t work exactly like it worked last time.
End of explanation
"""


df['Unnamed: 0'].head()

df = pd.read_csv('../data/date_fixed_running_data.csv', parse_dates=['Date'])

"""
Explanation: This is because when you save a data frame to a csv it doesn’t label the index column. So now our column is actually the ‘zero’ column. When you use parse_dates you can use either the column name (if available) or the 0-based column index number. This happened because the index didn’t have a name when we saved the last csv.
End of explanation
"""


df = pd.read_csv('../data/date_fixed_running_data.csv', parse_dates=[0])

df.head()

"""
Explanation: We can import the date correctly by specifying the zero based look up or by specifying the name like we did in the last video.
End of explanation
"""


cols = ['Date', 'Miles', 'Time']
df.columns = cols
df.head()

"""
Explanation: Let’s go ahead and rename the columns and set our index to the dates again.
End of explanation
"""


df.rename(columns={df.columns[0]:'Date'}, inplace=True)

df.head()

"""
Explanation: We can also rename the column using the rename method.
End of explanation
"""


df.set_index('Date', inplace=True)

df.plot()

"""
Explanation: Now we can plot it again once we set the index.
End of explanation
"""


raw_time_fmt = df.Time[0]

def get_total_seconds(raw_time):
    if raw_time is np.nan:
        return np.nan # if it's blank, keep it blank
    hrs, mins, seconds = str(raw_time).split(':')
    seconds = int(seconds) + 60 * int(mins) + 60 * 60 * int(hrs)
    return seconds    
print(get_total_seconds(raw_time_fmt))

df['Seconds'] = df.Time.map(get_total_seconds)

df.head(10)

"""
Explanation: Now one thing we’re going to want to do is get a breakdown of times for each run. Stats like minutes per mile and that sort of thing.
To do that we’re going to use map. In order to convert to seconds we’re going to need to convert our times represented as hour/min/seconds into just pure seconds. From that we can do minutes etc.
Let’s go ahead and write our map function. Now there is an edge case that we need to handle. Remember we don’t have time data for every single run so we’ve got to handle the NaN values correctly. 
we'll do this by just keeping them in there.
End of explanation
"""


df.describe()

"""
Explanation: Now we can describe and see I recorded times for 52 of the runs.
End of explanation
"""


df.fillna(0).describe()

"""
Explanation: This is a great example of what  np.NaN is so great in pandas. When we do this describe we get answers unlike in numpy and it just disregards the empty values. If we were to fill it in with zeros that would drag down our statistics.
We can see that here. See how much it changes the data when it is filled with 0 which would be incorrect analysis.
End of explanation
"""


df['Minutes'] = df['Seconds'].map(lambda x: x / 60)

df.describe()

"""
Explanation: Now that we have seconds it would be useful to see minutes too. I find it hard to think in seconds.
So let’s add a new column.
End of explanation
"""


df.plot(x='Miles', y='Minutes', kind='scatter')

"""
Explanation: Now let’s do a describe and plot it again.
End of explanation
"""


df.corr()

df.corr(method='kendall')

df.corr(method='spearman')

"""
Explanation: Let’s plot Miles and Minutes together in a scatter plot. Wow that’s linear. Let’s see how correlated they are. We do this with the cor method. We can see that Miles to time are very tightly correlated (using pearson standard correlation coefficients) there are two other correlation methods that you can use, kendall Tau, and Spearman rank correlation.
End of explanation
"""


df.boxplot('Minutes', return_type='axes')

"""
Explanation: Now let’s see a box plot. With these two we get a much better idea of the data. We can see that most of my runs are below an hour except for a couple that are much longer.-
End of explanation
"""


df['Minutes'] / df['Miles']

df['Min_per_mile'] = df['Minutes'] / df['Miles']

df.describe()

"""
Explanation: Now let’s add minutes per mile, we can just divide our two series to get those numbers.
End of explanation
"""


df.plot(x='Miles', y='Min_per_mile', kind='scatter')
plt.ylabel("Minutes / Mile")

"""
Explanation: We can see that along more shorter distances, my speed can vary a lot.
End of explanation
"""


df.hist('Min_per_mile')

"""
Explanation: Let’s see a histogram of my speeds. 
Histograms are a great way of representing frequency data or how much certain things are occuring.
End of explanation
"""


df.hist('Min_per_mile',bins=20)

"""
Explanation: seems pretty center in that 7 minutes to 7.5 minute range. Let’s see if we can get more information with more bins which we specify with the bin argument.
End of explanation
"""


df.hist('Min_per_mile',bins=20, figsize=(10,8))
plt.xlim((5, 11))
plt.ylim((0, 12))
plt.title("Minutes Per Mile Histogram")
plt.grid(False)
plt.savefig('../assets/minutes_per_mile_histogram.png')

df['Miles']

"""
Explanation: That’s interesting. Under 7 and then at 7.5 are the most popular. I bet that has something to do with my running distances too or the courses I choose to run.
End of explanation
"""


df['Miles'].plot()

"""
Explanation: Now another cool thing you can do with time series is see the rolling mean or rolling sum or even rolling correlations. There’s a lot of different “rolling” type things you can do.
End of explanation
"""


df['Miles'].plot()
pd.rolling_mean(df['Miles'], 7).plot()

"""
Explanation: So here’s a standard plot of our Miles again, just a line over time. To add another line to the same plot we just add more details to the box. As I was touching on the rolling values. Let’s talk about the rolling average. Now to do that I pass it a series or a data frame.
End of explanation
"""


df['Miles'].plot()
pd.rolling_std(df['Miles'], 7).plot()

df['Miles'].plot()

pd.rolling_sum(df['Miles'], 7).plot()

"""
Explanation: I can do the same with the rolling standard deviation or sum.
End of explanation
"""


df.index

"""
Explanation: Now on the last note one thing that’s cool about date time indexes is that you can query them very naturally. If I want to get all my runs in october of 2014, I just enter that as a string.
End of explanation
"""


df['2014-11':'2014-12']

"""
Explanation: If I want to get from November to December, I can do that as a Series.
End of explanation
"""


df['2014-11':'2015-1-1']['Miles'].plot()

"""
Explanation: How do you think we might go from october to January 1 2015?
Go ahead and give it a try and see if you can figure it out.
End of explanation
"""


df['2014-8-12']

"""
Explanation: Now we can specify a series this way but we can’t specific a specific date. To get a specific date’s run.
End of explanation
"""


df.loc['2014-8-12']

"""
Explanation: To do that we need to use loc.
End of explanation
"""


df.head()

df.to_csv('../data/date_fixed_running_data_with_time.csv')

df.to_html('../data/date_fixed_running_data_with_time.html')

"""
Explanation: now that we’ve done all this work. We should save it so that we don’t have to remember what our operations were or what stage we did them at. Now we could save it to csv like we did our other one but I wanted to illustrate all the different ways you can save this file.
Let’s save our csv, but we can also save it as an html page(which will give us a table view) or a json file.
End of explanation
"""


df.to_json('../data/date_fixed_running_data_with_time.json')

df.reset_index()

df['Date'] = df.index

df.index = range(df.shape[0])

df.head()
df.to_json('../data/date_fixed_running_data_with_time.json')

"""
Explanation: One thing to note with JSON files is that they want unique indexes (because they're going to be come the keys), so we've got to give it a new index. We can do this by resetting our index or setting our index to a column.
End of explanation
"""


df.Date[0]

"""
Explanation: Now there’s a LOT more you can do with date time indexing but this is about all that I wanted to cover in this video. We will get into more specifics later. By now you should be getting a lot more familiar with pandas and what the ipython + pandas workflow is.
End of explanation
"""


