import pandas as pd
temperatures = pd.Series([23, 20, 25, 18])
temperatures

"""
Explanation: <div style="width:1000 px">

<div style="float:right; width:98 px; height:98px;">
<img src="https://raw.githubusercontent.com/Unidata/MetPy/master/metpy/plots/_static/unidata_150x150.png" alt="Unidata Logo" style="height: 98px;">
</div>

<h1>Introduction to Pandas</h1>
<h3>Unidata Python Workshop</h3>

<div style="clear:both"></div>
</div>

<hr style="height:2px;">

Overview:

Teaching: 35 minutes
Exercises: 40 minutes

Questions

What is Pandas?
What are the basic Pandas data structures?
How can I read data into Pandas?
What are some of the data operations available in Pandas?

Objectives

<a href="#series">Data Series</a>
<a href="#frames">Data Frames</a>
<a href="#loading">Loading Data in Pandas</a>
<a href="#missing">Missing Data</a>
<a href="#manipulating">Manipulating Data</a>

<a name="series"></a>
Data Series
Data series are one of the fundamental data structures in Pandas. You can think of them like a dictionary; they have a key (index) and value (data/values) like a dictionary, but also have some handy functionality attached to them.
To start out, let's create a series from scratch. We'll imagine these are temperature observations.
End of explanation
"""


temperatures = pd.Series([23, 20, 25, 18], index=['TOP', 'OUN', 'DAL', 'DEN'])
temperatures

"""
Explanation: The values on the left are the index (zero based integers by default) and on the right are the values. Notice that the data type is an integer. Any NumPy datatype is acceptable in a series.
That's great, but it'd be more useful if the station were associated with those values. In fact you could say we want the values indexed by station name.
End of explanation
"""


temperatures['DAL']

temperatures[['DAL', 'OUN']]

"""
Explanation: Now, very similar to a dictionary, we can use the index to access and modify elements.
End of explanation
"""


temperatures[temperatures > 20]

temperatures + 2

"""
Explanation: We can also do basic filtering, math, etc.
End of explanation
"""


dps = {'TOP': 14,
       'OUN': 18,
       'DEN': 9,
       'PHX': 11,
       'DAL': 23}

dewpoints = pd.Series(dps)
dewpoints

"""
Explanation: Remember how I said that series are like dictionaries? We can create a series straight from a dictionary.
End of explanation
"""


'PHX' in dewpoints

'PHX' in temperatures

"""
Explanation: It's also easy to check and see if an index exists in a given series:
End of explanation
"""


temperatures.name = 'temperature'
temperatures.index.name = 'station'

temperatures

"""
Explanation: Series have a name attribute and their index has a name attribute.
End of explanation
"""


# Your code goes here


"""
Explanation: Exercise

Create a series of pressures for stations TOP, OUN, DEN, and DAL (assign any values you like).
Set the series name and series index name.
Print the pressures for all stations which have a dewpoint below 15.
End of explanation
"""


# %load solutions/make_series.py


"""
Explanation: Solution
End of explanation
"""


data = {'station': ['TOP', 'OUN', 'DEN', 'DAL'],
        'temperature': [23, 20, 25, 18],
        'dewpoint': [14, 18, 9, 23]}

df = pd.DataFrame(data)
df

"""
Explanation: <a href="#top">Top</a>
<hr style="height:2px;">

<a name="frames"></a>
Data Frames
Series are great, but what about a bunch of related series? Something like a table or a spreadsheet? Enter the data frame. A data frame can be thought of as a dictionary of data series. They have indexes for their rows and their columns. Each data series can be of a different type, but they will all share a common index.
The easiest way to create a data frame by hand is to use a dictionary.
End of explanation
"""


df['temperature']

df.dewpoint

"""
Explanation: You can access columns (data series) using dictionary type notation or attribute type notation.
End of explanation
"""


df['wspeed'] = 0.
df

"""
Explanation: Notice the index is shared and that the name of the column is attached as the series name.
You can also create a new column and assign values. If I only pass a scalar it is duplicated.
End of explanation
"""


df.index = df.station
df

"""
Explanation: Let's set the index to be the station.
End of explanation
"""


df = df.drop('station', axis='columns')
df

"""
Explanation: Well, that's close, but we now have a redundant column, so let's get rid of it.
End of explanation
"""


df['pressure'] = pd.Series([1010,1000,998,1018], index=['DEN','TOP','DAL','OUN'])
df

"""
Explanation: We can also add data and order it by providing index values. Note that the next cell contains data that's "out of order" compared to the dataframe shown above. However, by providing the index that corresponds to each value, the data is organized correctly into the dataframe.
End of explanation
"""


df.loc['DEN']

"""
Explanation: Now let's get a row from the dataframe instead of a column.
End of explanation
"""


df.T

"""
Explanation: We can even transpose the data easily if we needed that do make things easier to merge/munge later.
End of explanation
"""


df.values

df.temperature.values

"""
Explanation: Look at the values attribute to access the data as a 1D or 2D array for series and data frames recpectively.
End of explanation
"""


# Your code goes here


"""
Explanation: Exercise

Add a series of rain observations to the existing data frame.
Apply an instrument correction of -2 to the dewpoint observations.
End of explanation
"""


# %load solutions/rain_obs.py


"""
Explanation: Solution
End of explanation
"""


import pandas as pd

df = pd.read_csv('Jan17_CO_ASOS.txt', sep='\t')

df.head()

df = pd.read_csv('Jan17_CO_ASOS.txt', sep='\t', parse_dates=['valid'])

df.head()

df = pd.read_csv('Jan17_CO_ASOS.txt', sep='\t', parse_dates=['valid'], na_values='M')

df.head()

"""
Explanation: <a href="#top">Top</a>
<hr style="height:2px;">

<a name="loading"></a>
Loading Data in Pandas
The real power of pandas is in manupulating and summarizing large sets of tabular data. To do that, we'll need a large set of tabular data. We've included a file in this directory called JAN17_CO_ASOS.txt that has all of the ASOS observations for several stations in Colorado for January of 2017. It's a few hundred thousand rows of data in a tab delimited format. Let's load it into Pandas.
End of explanation
"""


df.columns

df.columns = ['station', 'time', 'temperature', 'dewpoint', 'pressure']

df.head()

"""
Explanation: Let's look in detail at those column names. Turns out we need to do some cleaning of this file. Welcome to real world data analysis.
End of explanation
"""


len(df)

df = df.dropna(axis='rows', how='all', subset=['temperature', 'dewpoint', 'pressure'])

len(df)

df.head()

"""
Explanation: For other formats of data CSV, fixed width, etc. that are tools to read it as well. You can even read excel files straight into Pandas.
<a href="#top">Top</a>
<hr style="height:2px;">

<a name="missing"></a>
Missing Data
We've already dealt with some missing data by turning the 'M' string into actual NaN's while reading the file in. We can do one better though and delete any rows that have all values missing. There are similar operations that could be performed for columns. You can even drop if any values are missing, all are missing, or just those you specify are missing.
End of explanation
"""


# Your code goes here
# df2 = 

"""
Explanation: Exercise
Our dataframe df has data in which we dropped any entries that were missing all of the temperature, dewpoint and pressure observations. Let's modify our command some and create a new dataframe df2 that only keeps observations that have all three variables (i.e. if a pressure is missing, the whole entry is dropped). This is useful if you were doing some computation that requires a complete observation to work.
End of explanation
"""


# %load solutions/drop_obs.py


"""
Explanation: Solution
End of explanation
"""


df.reset_index(drop=True)

df.head()

"""
Explanation: Lastly, we still have the original index values. Let's reindex to a new zero-based index for only the rows that have valid data in them.
End of explanation
"""


print(f'Min: {df.temperature.min()}\nMax: {df.temperature.max()}')

"""
Explanation: <a href="#top">Top</a>
<hr style="height:2px;">

<a name="manipulating"></a>
Manipulating Data
We can now take our data and do some intersting things with it. Let's start with a simple min/max.
End of explanation
"""


df.temperature.corr(df.dewpoint)

"""
Explanation: You can also do some useful statistics on data with attached methods like corr for correlation coefficient.
End of explanation
"""


df.groupby('station').mean()

"""
Explanation: We can also call a groupby on the data frame to start getting some summary information for each station.
End of explanation
"""


# Calculate min


# Calculate max


# Calculate standard deviation


"""
Explanation: Exercise
Calculate the min, max, and standard deviation of the temperature field grouped by each station.
End of explanation
"""


# %load solutions/calc_stats.py


"""
Explanation: Solution
End of explanation
"""


df.groupby('station').describe()

"""
Explanation: Now, let me show you how to do all of that and more in a single call.
End of explanation
"""


df.groupby('station').get_group('0CO').head().reset_index(drop=True)

"""
Explanation: Now let's suppose we're going to make a meteogram or similar and want to get all of the data for a single station.
End of explanation
"""


# Your code goes here


"""
Explanation: Exercise

Round the temperature column to whole degrees.
Group the observations by temperature and use the count method to see how many instances of the rounded temperatures there are in the dataset.
End of explanation
"""


# %load solutions/temperature_count.py


"""
Explanation: Solution
End of explanation
"""


