import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

plt.style.use('ggplot')

"""
Explanation: <p><font size="6"><b>04 - Pandas: Working with time series data</b></font></p>


© 2021, Joris Van den Bossche and Stijn Van Hoey  (&#106;&#111;&#114;&#105;&#115;&#118;&#97;&#110;&#100;&#101;&#110;&#98;&#111;&#115;&#115;&#99;&#104;&#101;&#64;&#103;&#109;&#97;&#105;&#108;&#46;&#99;&#111;&#109;, &#115;&#116;&#105;&#106;&#110;&#118;&#97;&#110;&#104;&#111;&#101;&#121;&#64;&#103;&#109;&#97;&#105;&#108;&#46;&#99;&#111;&#109;). Licensed under CC BY 4.0 Creative Commons
End of explanation
"""


import datetime

dt = datetime.datetime(year=2016, month=12, day=19, hour=13, minute=30)
dt

print(dt) # .day,...

print(dt.strftime("%d %B %Y"))

"""
Explanation: Introduction: datetime module
Standard Python contains the datetime module to handle date and time data:
End of explanation
"""


ts = pd.Timestamp('2016-12-19')
ts

"""
Explanation: Dates and times in pandas
The Timestamp object
Pandas has its own date and time objects, which are compatible with the standard datetime objects, but provide some more functionality to work with.  
The Timestamp object can also be constructed from a string:
End of explanation
"""


ts.month

"""
Explanation: Like with datetime.datetime objects, there are several useful attributes available on the Timestamp. For example, we can get the month (experiment with tab completion!):
End of explanation
"""


ts + pd.Timedelta('5 days')

"""
Explanation: There is also a Timedelta type, which can e.g. be used to add intervals of time:
End of explanation
"""


pd.to_datetime("2016-12-09")

pd.to_datetime("09/12/2016")

pd.to_datetime("09/12/2016", format="%d/%m/%Y")

"""
Explanation: Parsing datetime strings

Unfortunately, when working with real world data, you encounter many different datetime formats. Most of the time when you have to deal with them, they come in text format, e.g. from a CSV file. To work with those data in Pandas, we first have to parse the strings to actual Timestamp objects.
<div class="alert alert-info">
<b>REMEMBER</b>: <br><br>

To convert string formatted dates to Timestamp objects: use the `pandas.to_datetime` function

</div>
End of explanation
"""


s = pd.Series(['2016-12-09 10:00:00', '2016-12-09 11:00:00', '2016-12-09 12:00:00'])

s

"""
Explanation: A detailed overview of how to specify the format string, see the table in the python documentation: https://docs.python.org/3/library/datetime.html#strftime-and-strptime-behavior
Timestamp data in a Series or DataFrame column
End of explanation
"""


ts = pd.to_datetime(s)

ts

"""
Explanation: The to_datetime function can also be used to convert a full series of strings:
End of explanation
"""


ts.dt.hour

ts.dt.dayofweek

"""
Explanation: Notice the data type of this series has changed: the datetime64[ns] dtype. This indicates that we have a series of actual datetime values.
The same attributes as on single Timestamps are also available on a Series with datetime data, using the .dt accessor:
End of explanation
"""


pd.Series(pd.date_range(start="2016-01-01", periods=10, freq='3H'))

"""
Explanation: To quickly construct some regular time series data, the pd.date_range function comes in handy:
End of explanation
"""


data = pd.read_csv("data/vmm_flowdata.csv")

data.head()

"""
Explanation: Time series data: Timestamp in the index
River discharge example data
For the following demonstration of the time series functionality, we use a sample of discharge data of the Maarkebeek (Flanders) with 3 hour averaged values, derived from the Waterinfo website.
End of explanation
"""


data['Time'] = pd.to_datetime(data['Time'])

"""
Explanation: We already know how to parse a date column with Pandas:
End of explanation
"""


data = data.set_index("Time")

data

"""
Explanation: With set_index('datetime'), we set the column with datetime values as the index, which can be done by both Series and DataFrame.
End of explanation
"""


data = pd.read_csv("data/vmm_flowdata.csv", index_col=0, parse_dates=True)

"""
Explanation: The steps above are provided as built-in functionality of read_csv:
End of explanation
"""


data.index

"""
Explanation: <div class="alert alert-info">
<b>REMEMBER</b>: <br><br>

`pd.read_csv` provides a lot of built-in functionality to support this kind of transactions when reading in a file! Check the help of the read_csv function...

</div>

The DatetimeIndex
When we ensure the DataFrame has a DatetimeIndex, time-series related functionality becomes available:
End of explanation
"""


data.index.day

data.index.dayofyear

data.index.year

"""
Explanation: Similar to a Series with datetime data, there are some attributes of the timestamp values available:
End of explanation
"""


%matplotlib widget

data.plot()

# switching back to static inline plots (the default)
%matplotlib inline

"""
Explanation: The plot method will also adapt its labels (when you zoom in, you can see the different levels of detail of the datetime labels):
End of explanation
"""


data[pd.Timestamp("2012-01-01 09:00"):pd.Timestamp("2012-01-01 19:00")]

"""
Explanation: We have too much data to sensibly plot on one figure. Let's see how we can easily select part of the data or aggregate the data to other time resolutions in the next sections.
Selecting data from a time series
We can use label based indexing on a timeseries as expected:
End of explanation
"""


data["2012-01-01 09:00":"2012-01-01 19:00"]

"""
Explanation: But, for convenience, indexing a time series also works with strings:
End of explanation
"""


data['2013':]

"""
Explanation: A nice feature is "partial string" indexing, where we can do implicit slicing by providing a partial datetime string.
E.g. all data of 2013:
End of explanation
"""


data['2012-01':'2012-03']

"""
Explanation: Or all data of January up to March 2012:
End of explanation
"""


data = pd.read_csv("data/vmm_flowdata.csv", index_col=0, parse_dates=True)

"""
Explanation: Exercises
Let's practice this yourself using the same dataset:
End of explanation
"""


# %load _solutions/pandas_04_time_series_data1.py

"""
Explanation: <div class="alert alert-success">

<b>EXERCISE 1</b>:

 <ul>
  <li>select all data starting from 2012</li>
</ul>
</div>
End of explanation
"""


# %load _solutions/pandas_04_time_series_data2.py

"""
Explanation: <div class="alert alert-success">

<b>EXERCISE 2</b>:

 <ul>
  <li>select all data in January for all different years</li>
</ul>
</div>
End of explanation
"""


# %load _solutions/pandas_04_time_series_data3.py

"""
Explanation: <div class="alert alert-success">

<b>EXERCISE 3</b>:

 <ul>
  <li>select all data in April, May and June for all different years</li>
</ul>
</div>
End of explanation
"""


# %load _solutions/pandas_04_time_series_data4.py

"""
Explanation: <div class="alert alert-success">

<b>EXERCISE 4</b>:

 <ul>
  <li>select all 'daytime' data (between 8h and 20h) for all days</li>
</ul>
</div>
End of explanation
"""


data.resample('D').mean().head()

"""
Explanation: The power of pandas: resample
A very powerfull method is resample: converting the frequency of the time series (e.g. from hourly to daily data).
The time series has a frequency of 1 hour. I want to change this to daily:
End of explanation
"""


data.resample('D').max().head()

"""
Explanation: Other mathematical methods can also be specified:
End of explanation
"""


data.resample('M').mean().plot() # 10D

"""
Explanation: <div class="alert alert-info">
<b>REMEMBER</b>: <br><br>

The string to specify the new time frequency: http://pandas.pydata.org/pandas-docs/stable/user_guide/timeseries.html#offset-aliases <br>

These strings can also be combined with numbers, eg `'10D'`...

</div>
End of explanation
"""


# %load _solutions/pandas_04_time_series_data5.py

"""
Explanation: <div class="alert alert-success">

<b>EXERCISE 5</b>:

 <ul>
  <li>Plot the monthly standard deviation of the columns</li>
</ul>
</div>
End of explanation
"""


# %load _solutions/pandas_04_time_series_data6.py

"""
Explanation: <div class="alert alert-success">

<b>EXERCISE 6</b>:

 <ul>
  <li>Plot the monthly mean and median values for the years 2011-2012 for 'L06_347'<br><br></li>
</ul>

__Note__ Did you know <a href="https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.agg.html"><code>agg</code></a> to derive multiple statistics at the same time?

</div>
End of explanation
"""


# %load _solutions/pandas_04_time_series_data7.py

# %load _solutions/pandas_04_time_series_data8.py

"""
Explanation: <div class="alert alert-success">

<b>EXERCISE 7</b>:

 <ul>
  <li>plot the monthly mininum and maximum daily average value of the 'LS06_348' column</li>
</ul>
</div>
End of explanation
"""


# %load _solutions/pandas_04_time_series_data9.py

"""
Explanation: <div class="alert alert-success">
<b>EXERCISE 8</b>:

 <ul>
  <li>Make a bar plot of the mean of the stations in year of 2013</li>
</ul>

</div>
End of explanation
"""


