import pandas as pd

"""
Explanation: Introduction to pandas


pandas is a Python package providing convenient data structures to work with labelled data.


pandas is perfectly suited for observational / statistical data sets, having many similarities with Excel spreadsheets.


Key features:

easy handling of missing data
size mutability: columns can be inserted and deleted from DataFrame
automatic and explicit data alignment: objects can be explicitly aligned to a set of labels, or the user can simply ignore the labels and let Series, DataFrame, etc. automatically align the data for you in computations
powerful, flexible group by functionality to perform split-apply-combine operations on data sets
make it easy to convert ragged, differently-indexed data in other Python and NumPy data structures into DataFrame objects
intelligent label-based slicing, fancy indexing, and subsetting of large data sets
intuitive merging and joining data sets
flexible reshaping and pivoting of data sets
hierarchical labeling of axes (possible to have multiple labels per tick)
robust IO tools for loading data from flat files (CSV and delimited), Excel files, databases, and saving / loading data from the ultrafast HDF5 format
time series-specific functionality



Primary data structures of pandas

Series (1-dimensional)
DataFrame (2-dimensional)

pandas is built on top of NumPy and is intended to integrate well within a scientific computing environment with many other 3rd party libraries.
Loading data

In this part, we will use observations of the air quality as an example
The data have been collected hourly from one of the monitoring sites in London (Marleybone Road) over the year 2015

Measured are: ozone ($O_3$), nitrogen oxides (NOx), carbon monoxide (CO) and PM10 particulate matter


Data source: https://uk-air.defra.gov.uk/data/

Site description: https://uk-air.defra.gov.uk/networks/site-info?uka_id=UKA00315

First, we import pandas module. We use an alias "pd" to write code quicker.
End of explanation
"""


import matplotlib.pyplot as plt
import numpy as np
import os
%matplotlib inline

import warnings
warnings.filterwarnings('ignore')

fname = '../data/air_quality_hourly_london_marylebone.csv'

"""
Explanation: We also import the os module that is useful for building paths to files (among many other things). And numpy with matplotlib just in case too.
End of explanation
"""


# minimal setup to read the given file
air_quality = pd.read_csv(fname, header=4, skipfooter=4, na_values='No data', engine='python')

"""
Explanation: Let's try to read the data using pandas.read_csv() function.
End of explanation
"""


type(air_quality)

# Internal nature of the object
print(air_quality.shape)
print()
print(air_quality.dtypes)

# View just the tip of data
air_quality.head(5)

"""
Explanation: Q: What happens if you remove the header? skipfooter? engine?
Data structures: DataFrame and Series
Let's interrogate the DataFrame object!
End of explanation
"""


# View the last rows of data
air_quality.tail(n=2)  # Note the optional argument (available for head() too)

"""
Explanation: Q: What did you notice about "Status" columns? Compare them to the original text file.
End of explanation
"""


air_quality.index

"""
Explanation: Get descriptors for the vertical axis (axis=0):
End of explanation
"""


air_quality.columns

"""
Explanation: Get descriptors for the horizontal axis (axis=1):
End of explanation
"""


air_quality.info()

"""
Explanation: A lot of information at once including memory usage:
End of explanation
"""


carbon_monoxide = air_quality['Carbon monoxide']

"""
Explanation: Series, pandas' 1D data containter
A series can be constructed with the pd.Series constructor (passing an array of values) or from a DataFrame, by extracting one of its columns.
End of explanation
"""


print(type(carbon_monoxide))
print(carbon_monoxide.dtype)
print(carbon_monoxide.shape)
print(carbon_monoxide.nbytes)

"""
Explanation: Some of its attributes:
End of explanation
"""


carbon_monoxide

"""
Explanation: Show me what you got! <img src=https://staticdelivery.nexusmods.com/mods/1151/images/543-1-1447533110.png style="width:40px;">
End of explanation
"""


air_quality['Nitrogen oxides as nitrogen dioxide'].values

type(air_quality['Nitrogen oxides as nitrogen dioxide'].values)

"""
Explanation: Numpy as pandas's backend
It is always possible to fall back to a good old NumPy array to pass on to scientific libraries that need them: SciPy, scikit-learn, etc
End of explanation
"""


# def dateparse(date_str, time_str):
#     diff = pd.to_timedelta((df['End Date'] == '24:00:00').astype(int), unit='d')
#     pd.datetime.strptime(x+y, '%Y-%m-%d%H:%M:%S') 

air_quality = pd.read_csv(fname, header=4, skipfooter=4, na_values='No data', engine='python',
                          parse_dates={'Time': [0, 1]})

"""
Explanation: Cleaning data
<blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">The truth about data science: cleaning your data is 90% of the work. Fitting the model is easy. Interpreting the results is the other 90%.</p>&mdash; Jake VanderPlas (@jakevdp) <a href="https://twitter.com/jakevdp/status/742406386525446144">June 13, 2016</a></blockquote>
<script async src="//platform.twitter.com/widgets.js" charset="utf-8"></script>

Dealing with dates and times
End of explanation
"""


air_quality.columns = ['Time', 'O3', 'O3_status', 'NOx', 'NOx_status',
                       'CO', 'CO_status', 'PM10', 'PM10_status', 'Co', 'Co_status']
air_quality.columns

"""
Explanation: Renaming columns
End of explanation
"""


air_quality = air_quality.drop('Co', 1)
air_quality = air_quality.drop('Co_status', 1)

air_quality.head()

"""
Explanation: Deleting columns
Let us concentrate our attention on the first 4 chemical species, and remove cobalt data from our DataFrame:
End of explanation
"""


# air_quality.plot()

"""
Explanation: Basic visualisation
Exercise
Try calling plot() method of the air_quality object:
End of explanation
"""


# air_quality.plot( ... )

"""
Explanation: What happens if put subplots=True as an argument of the plot() method?
End of explanation
"""


fig, (ax0, ax1) = plt.subplots(ncols=2, figsize=(10, 4))
air_quality.boxplot(ax=ax0, column=['O3', 'PM10'])
air_quality.O3.plot(ax=ax1, kind="kde")

"""
Explanation: It is easy to create other useful plots using DataFrame:
End of explanation
"""


air_quality.O3.plot(grid=True, figsize=(12, 2))

"""
Explanation: As well as just a simple line plot:
End of explanation
"""


air_quality[air_quality.O3.values < 0]

"""
Explanation: Setting missing values
As you may notice, we have negative values of ozone concentration, which does not make sense. So, let us replace those negative values with NaN:
End of explanation
"""


# Replace negative ozone values with NaN
air_quality.O3[air_quality.O3.values < 0] = np.nan

"""
Explanation: We can mask them out in the same way as you do with numpy arrays:
End of explanation
"""


# with pd.ExcelWriter("test.xls") as our_writer:
#     air_quality.to_excel(writer, sheet_name='Blah-blah')

"""
Explanation: Saving data
For each read_** function to load data, there is a to_** method attached to Series and DataFrames.
Excel spreadsheets
Uncomment the following code cell and run to save the whole DataFrame to an Excel file.
End of explanation
"""


# Your code here

"""
Explanation: Exercise: writing to CSV text files
Find a method to save DataFrames to a text file (or whatever format you like more).
End of explanation
"""


air_quality.describe()

"""
Explanation: Some statistics
End of explanation
"""


from pandas.tools.plotting import scatter_matrix

with plt.style.context('ggplot'):
    scatter_matrix(air_quality, figsize=(7, 7))

"""
Explanation: Correlations and regressions

Is there correlations between the timeseries we loaded?
First, let's take a glance at the whole DataFrame using a fancy scatter_matrix function.
End of explanation
"""


air_quality.NOx.corr(air_quality['CO'])

"""
Explanation: Computing correlations
Both Series and DataFrames have a corr() method to compute the correlation coefficient.
End of explanation
"""


air_quality.corr()

"""
Explanation: If series are already grouped into a DataFrame, computing all correlation coefficients is trivial:
End of explanation
"""


# fig, ax = plt.subplots()
# p = ax.imshow(air_quality.corr(), interpolation="nearest", cmap='RdBu_r', vmin=-1, vmax=1)
# ax.set_xticks(np.arange(len(air_quality.corr().columns)))
# ax.set_yticks(np.arange(len(air_quality.corr().index)))
# ax.set_xticklabels(air_quality.corr().columns)
# ax.set_yticklabels(air_quality.corr().index)
# fig.colorbar(p)

"""
Explanation: If you want to visualise this correlation matrix, uncomment the following code cell.
End of explanation
"""


soi_df = pd.read_csv('../data/soi.csv', skiprows=1, parse_dates=[0], index_col=0, na_values=-999.9,
                     date_parser=lambda x: pd.datetime.strptime(x, '%Y%m'))

olr_df = pd.read_csv('../data/olr.csv', skiprows=1, parse_dates=[0], index_col=0, na_values=-999.9,
                     date_parser=lambda x: pd.datetime.strptime(x, '%Y%m'))

df = pd.DataFrame({'OLR': olr_df.Value,
                   'SOI': soi_df.Value})

df.describe()

df.plot()

"""
Explanation: Creating DataFrames

DataFrame can also be created manually, by grouping several Series together.
Now just for fun we switch to another dataset:
create 2 Series objects from 2 CSV files
create a DataFrame by combining the two Series
End of explanation
"""


import statsmodels.formula.api as sm

sm_model = sm.ols(formula="SOI ~ OLR", data=df).fit()

# df['SOI'].plot()
# df['OLR'].plot()
# ax = sm_model.fittedvalues.plot(label="model prediction")
# ax.legend(loc="lower center", ncol=3)

"""
Explanation: Ordinary Least Square (OLS) regressions
<img src=http://statsmodels.sourceforge.net/devel/_static/statsmodels_hybi_banner.png>
The recommeded way to build ordinaty least square regressions is by using statsmodels.
End of explanation
"""


# your code here

"""
Explanation: Exercise: create a scatter plot

You can use df.plot function with the appropriate keywords
What happens if you use "c=" keyword?
Pass another parameter: edgecolors='none'
End of explanation
"""


import matplotlib.dates as mdates

"""
Explanation: Advanced scatter plot
Using the power of matplotlib, we can create a scatter plot with points coloured by the date index. To do this we need to import one additional submodule:
End of explanation
"""


mdt = mdates.date2num(df.index.astype(pd.datetime))

"""
Explanation: Convert numpy.datetime64 objects (which are the indices of our DataFrame) to matplotlib floating point numbers. These numbers represent the number of days (fraction part represents hours, minutes, seconds) since 0001-01-01 00:00:00 UTC (assuming Gregorian calendar).
End of explanation
"""


df['mpl_date'] = mdt

"""
Explanation: Append the new data to the original DataFrame:
End of explanation
"""


ax = df.plot(kind='scatter', x='OLR', y='SOI', c='mpl_date',
             colormap='viridis', colorbar=False, edgecolors='none')
plt.colorbar(ax.collections[0], ticks=mdates.YearLocator(5), 
             format=mdates.DateFormatter('%Y'))

"""
Explanation: Create a scatter plot
End of explanation
"""


sub_soi_df = soi_df['1992':'2015']
sub_soi_df.head()

"""
Explanation: Exercise: rolling functions
1. Subset data

Start by subsetting the SOI DataFrame
Use either numerical indices, or, even better, datetime indices
End of explanation
"""


sub_soi_df.plot(lw=0.5, marker='d', ms=3, linestyle='-', color='k', figsize=(8, 3), grid=True)

"""
Explanation: 2. Plot the subset data

You can create figure and axis using matplotlib.pyplot
Or just use the plot() method of pandas DataFrame
End of explanation
"""


# df.rolling?

roll = sub_soi_df.rolling(window=10, center=False)

"""
Explanation: 3. Explore what rolling() method is

What does this method return?
End of explanation
"""


fig, ax = plt.subplots()
ax.plot(sub_soi_df, label='SOI')
ax.plot(roll.mean(), label='mean')
leg = ax.legend()

"""
Explanation: 4. Plot the original series and the smoothed series
End of explanation
"""


