import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

import pandas as pd
pd.set_option('max_rows', 6)  # max number of rows to show in this notebook — to save space!

import seaborn as sns  # for better style in plots

"""
Explanation: 1D analysis: pandas!
End of explanation
"""


df = pd.read_csv('../data/yellow_tripdata_2016-05-01_decimated.csv', parse_dates=[0, 2], index_col=[0])

"""
Explanation: Reading in data to a dataframe
For 1D analysis, we are generally thinking about data that varies in time, so time series analysis. The pandas package is particularly suited to deal with this type of data, having very convenient methods for interpreting, searching through, and using time representations.
Let's start with the example we started the class with: taxi rides in New York City.
End of explanation
"""


df.index

"""
Explanation: What do all these (and other) input keyword arguments do?

header: tells which row of the data file is the header, from which it will extract column names
parse_dates: try to interpret the values in [col] or [[col1, col2]] as dates, to convert them into datetime objects.
index_col: if no index column is given, an index counting from 0 is given to the rows. By inputting index_col=[column integer], that column will be used as the index instead. This is usually done with the time information for the dataset.
skiprows: can skip specific rows, skiprows=[list of rows to skip numbered from start of file with 0], or number of rows to skip, skiprows=N.

We can check to make sure the date/time information has been read in as the index, which allows us to reference the other columns using this time information really easily:
End of explanation
"""


df['trip_distance']

"""
Explanation: From this we see that the index is indeed using the timing information in the file, and we can see that the dtype is datetime.
Selecting rows and columns of data
In particular, we will select rows based on the index. Since in this example we are indexing by time, we can use human-readable notation to select based on date/times themselves instead of index. Columns can be selected by name.
We can now access the columns of the file using dictionary-like keyword arguments, like so:
End of explanation
"""


df.trip_distance

"""
Explanation: We can equivalently access the columns of data as if they are methods. This means that we can use tab autocomplete to see methods and data available in a dataframe.
End of explanation
"""


df['trip_distance'].plot(figsize=(14,6))

"""
Explanation: We can plot in this way, too:
End of explanation
"""


df['2016-05-01 00']

"""
Explanation: Simple data selection
One of the biggest benefits of using pandas is being able to easily reference the data in intuitive ways. For example, because we set up the index of the dataframe to be the date and time, we can pull out data using dates. In the following, we pull out all data from the first hour of the day:
End of explanation
"""


df['passenger_count']['2016-05-01 00']

"""
Explanation: Here we further subdivide to examine the passenger count during that time period:
End of explanation
"""


df['2016-05-01 00':'2016-05-01 11']

"""
Explanation: We can also access a range of data, for example any data rows from midnight until noon:
End of explanation
"""


df['2016-05-01 00:30']

"""
Explanation: If you want more choice in your selection
The following, adding on minutes, does not work:
End of explanation
"""


df.loc['2016-05-01 00:30']

"""
Explanation: However, we can use another approach to have more control, with .loc to access combinations of specific columns and/or rows, or subsets of columns and/or rows.
End of explanation
"""


df.loc['2016-05-01 00:30', 'passenger_count']

"""
Explanation: You can also select data for more specific time periods.
df.loc[row_label, col_label]
End of explanation
"""


df.loc['2016-05-01 00:30', ['passenger_count','trip_distance']]

"""
Explanation: You can select more than one column:
End of explanation
"""


df.loc['2016-05-01 00:30':'2016-05-01 01:30', ['passenger_count','trip_distance']]

"""
Explanation: You can select a range of data:
End of explanation
"""


df.iloc[0:5, :]

"""
Explanation: You can alternatively select data by index instead of by label, using iloc instead of loc. Here we select the first 5 rows of data for all columns:
End of explanation
"""


df = pd.read_csv('../data/yellow_tripdata_2016-05-01_decimated.csv', parse_dates=[0, 2], index_col=[0])
df.index

df.index.strftime('%b %d, %Y %H:%m')

"""
Explanation: Exercise

Access the data from dataframe df for the last three hours of the day at once. Plot the tip amount (tip_amount) for this time period.
After you can make a line plot, try making a histogram of the data. Play around with the data range and the number of bins. A number of plot types are available built-in to a pandas dataframe inside the plot method under the keyword argument kind.



Exercise

Using pandas, read in the CTD data we've used in class several times. What variable would make sense to use for your index column?


Notes about datetimes
You can change the format of datetimes using strftime(). 
Compare the datetimes in our dataframe index in the first cell below with the second cell, in which we format the look of the datetimes differently. We can choose how it looks using formatting codes. You can find a comprehensive list of the formatting directives at http://strftime.org/. Note that inside the parentheses, you can write other characters that will be passed through (like the comma in the example below).
End of explanation
"""


now = pd.Timestamp('October 22, 2019 1:19PM')
now

tomorrow = pd.Timedelta('1 day')
now + tomorrow

"""
Explanation: You can create and use datetimes using pandas. It will interpret the information you put into a string as best it can. Year-month-day is a good way to put in dates instead of using either American or European-specific ordering. 
After defining a pandas Timestamp, you can also change time using Timedelta.
End of explanation
"""


pd.date_range(start='Jan 1 2019', end='May 1 2019', freq='15T')

"""
Explanation: You can set up a range of datetimes to make your own data frame indices with the following. Codes for frequency are available.
End of explanation
"""


df.index.minute

df.index.dayofweek

"""
Explanation: Note that you can get many different measures of your time index.
End of explanation
"""


df['tip squared'] = df.tip_amount**2  # making up some numbers to save to a new column
df['tip squared'].plot()

"""
Explanation: Exercise

How would you change the call to strftime above to format all of the indices such that the first index, for example, would be "the 1st of May, 2016 at the hour of 00 and the minute of 00 and the seconds of 00, which is the following day of the week: Sunday." Use the format codes for as many of the values as possible.


Adding column to dataframe
We can add data to our dataframe very easily. Below we add an index that gives the minute in the hour throughout the day.
End of explanation
"""


df2 = pd.read_table('../data/burl1h2010.txt', header=0, skiprows=[1], delim_whitespace=True, 
                    parse_dates={'dates': ['#YY', 'MM', 'DD', 'hh']},  index_col=0)
df2

df2.index

"""
Explanation: Another example: Wind data
Let's read in the wind data file that we have used before to have another data set to use. Note the parameters used to read it in properly.
End of explanation
"""


df.plot?

"""
Explanation: Plotting with pandas
You can plot with matplotlib and control many things directly from pandas. Get more info about plotting from pandas dataframes directly from:
End of explanation
"""


import matplotlib.pyplot as plt

fig, axes = plt.subplots(1, 2, figsize=(14,4))

df2['WSPD']['2010-5'].plot(ax=axes[0])
df2.loc['2010-5'].plot(y='WSPD', ax=axes[1])

"""
Explanation: You can mix and match plotting with matplotlib by either setting up a figure and axes you want to use with calls to plot from your dataframe (which you input to the plot call), or you can start with a pandas plot and save an axes from that call. Each will be demonstrated next. Or, you can bring the pandas data to matplotlib fully.
Start from matplotlib, then input axes to pandas
To demonstrate plotting starting from matplotlib, we will also demonstrate a note about column selection for plotting. You can select which data columns to plot either by selecting in the line before the plot call, or you can choose the columns within the plot call. 
The key part here is that you input to your pandas plot call the axes you wanted plotted into (here: ax=axes[0]).
End of explanation
"""


ax = df2['WSPD']['2010 11 1'].plot()
ax.set_ylabel('Wind speed')

"""
Explanation: Start with pandas, then use matplotlib commands
The important part here is that the call to pandas dataframe plotting returns an axes handle which you can save; here, it is saved as "ax".
End of explanation
"""


plt.plot(df2['WSPD'])

"""
Explanation: Bring pandas dataframe data to matplotlib fully
You can also use matplotlib directly by pulling the data you want to plot out of your dataframe.
End of explanation
"""


# all
df2.plot()

"""
Explanation: Plot all or multiple columns at once
End of explanation
"""


# multiple
fig, axes = plt.subplots(1, 2, figsize=(14,4))
df2[['WSPD', 'GST']].plot(ax=axes[0])
df2.plot(y=['WSPD', 'GST'], ax=axes[1])

"""
Explanation: To plot more than one but less than all columns, give a list of column names. Here are two ways to do the same thing:
End of explanation
"""


ax = df2['WSPD'].plot(figsize=(14,4))

from matplotlib.dates import DateFormatter

ax = df2['WSPD'].plot(figsize=(14,4))
ax.set_xlabel('2010')

date_form = DateFormatter("%b %d")
ax.xaxis.set_major_formatter(date_form)

# import matplotlib.dates as mdates
# # You can also control where the ticks are located, by date with Locators
# ticklocations = mdates.MonthLocator()
# ax.xaxis.set_major_locator(ticklocations)


"""
Explanation: Formatting dates
You can control how datetimes look on the x axis in these plots as demonstrated in this section. The formatting codes used in the call to DateFormatter are the same as those used above in this notebook for strftime.
Note that you can also control all of this with minor ticks additionally.
End of explanation
"""


axleft = df2['WSPD']['2010-10'].plot(figsize=(14,4))
axright = df2['WDIR']['2010-10'].plot(secondary_y=True, alpha=0.5)
axleft.set_ylabel('Speed [m/s]', color='blue');
axright.set_ylabel('Dir [degrees]', color='orange');

"""
Explanation: Plotting with twin axis
You can very easily plot two variables with different y axis limits with the secondary_y keyword argument to df.plot.
End of explanation
"""


df2.resample('1d').max()    #['DEWP']  # now the data is daily

"""
Explanation: Resampling
Sometimes we want our data to be at a different sampling frequency that we have, that is, we want to change the time between rows or observations. Changing this is called resampling. We can upsample to increase the number of data points in a given dataset (or decrease the period between points) or we can downsample to decrease the number of data points.
The wind data is given every hour. Here we downsample it to be once a day instead. After the resample function, a method needs to be used for how to combine the data over the downsampling period since the existing data needs to be combined in some way. We could use the max value over the 1-day period to represent each day:
End of explanation
"""


df2['GST']['2010-4-1':'2010-4-5'].plot()
df2.resample('1d').max()['GST']['2010-4-1':'2010-4-5'].plot()

"""
Explanation: It's always important to check our results to make sure they look reasonable. Let's plot our resampled data with the original data to make sure they align well. We'll choose one variable for this check.
We can see that the daily max wind gust does indeed look like the max value for each day, though note that it is plotted at the start of the day.
End of explanation
"""


df2.resample('30min').max()  # max doesn't say what to do with data in new rows

"""
Explanation: We can also upsample our data or add more rows of data. Note that like before, after we resample our data we still need a method on the end telling pandas how to process the data. However, since in this case we are not combining data (downsampling) but are adding more rows (upsampling), using a function like max doesn't change the existing observations (taking the max of a single row). For the new rows, we haven't said how to fill them so they are nan's by default.
Here we are changing from having data every hour to having it every 30 minutes.
End of explanation
"""


df2.resample('30min').ffill()

"""
Explanation: When upsampling, a reasonable option is to fill the new rows with data from the previous existing row:
End of explanation
"""


df2.resample('15 T').interpolate()

"""
Explanation: Here we upsample to have data every 15 minutes, but we interpolate to fill in the data between. This is a very useful thing to be able to do.
End of explanation
"""


df3 = pd.read_table('http://pong.tamu.edu/tabswebsite/daily/tabs_V_salt_all', index_col=0, parse_dates=True)
df3

ax = df3.groupby(df3.index.month).aggregate(np.mean)['Salinity'].plot(color='k', grid=True, figsize=(14, 4), marker='o')

# the x axis is now showing month of the year, which is what we aggregated over
ax.set_xlabel('Month of year')

ax.set_ylabel('Average salinity')

"""
Explanation: The codes for time period/frequency are available and are presented here for convenience:
Alias   Description
B   business day frequency
C   custom business day frequency (experimental)
D   calendar day frequency
W   weekly frequency
M   month end frequency
SM  semi-month end frequency (15th and end of month)
BM  business month end frequency
CBM custom business month end frequency
MS  month start frequency
SMS semi-month start frequency (1st and 15th)
BMS business month start frequency
CBMS    custom business month start frequency
Q   quarter end frequency
BQ  business quarter endfrequency
QS  quarter start frequency
BQS business quarter start frequency
A   year end frequency
BA  business year end frequency
AS  year start frequency
BAS business year start frequency
BH  business hour frequency
H   hourly frequency
T, min  minutely frequency
S   secondly frequency
L, ms   milliseconds
U, us   microseconds
N   nanoseconds


Exercise

We looked at NYC taxi trip distance earlier, but it was hard to tell what was going on with so much data. Resample this high resolution data to be lower resolution so that any trends in the information are easier to see. By what method do you want to do this downsampling? Plot your results.


groupby and difference between groupby and resampling
groupby allows us to aggregate data across a category or value. We'll use the example of grouping across a measure of time.
Let's examine this further using a dataset of some water properties near the Flower Garden Banks in Texas. We want to find the average salinity by month across the years of data available, that is, we want to know the average salinity value for each month of the year, calculated for each month from all of the years of data available. We will end up with 12 data points in this case. 
This is distinct from resampling for which if you calculate the average salinity by month, you will get a data point for each month in the time series. If there are 5 years of data in your dataset, you will end up with 12*5=60 data points total.
In the groupby example below, we first read the data into dataframe 'df3', then we group it by month (across years, since there are many years of data). From this grouping, we decide what function we want to apply to all of the numbers we've aggregated across the months of the year. We'll use mean for this example.
End of explanation
"""


