# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: Python [conda root]
#     language: python
#     name: conda-root-py
# ---

# + [markdown] slideshow={"slide_type": "slide"}
# <div class="page-wrap">
#     <h1 class="title-slide">Data Analysis with Pandas </h1>
#     <br>
#     <center><img src="img/pyladies.png" width="25%"></center>
# </div><!-- close .page-wrap -->
#
#
# <footer>
#     <p class="footer-left">September 8, 2018</p>
#     <p class="footer-right">Instructor: <NAME></p>
#
# </footer>

# + [markdown] slideshow={"slide_type": "slide"}
# # Agenda
#
# - Lesson 1: Getting Started with Data
# - Lesson 2: Sorting, Filtering, and Aggregation
# - Lesson 3: Subsets and Indexing

# + [markdown] slideshow={"slide_type": "slide"}
# # Lesson 1: Getting Started with Data

# + [markdown] slideshow={"slide_type": "slide"}
# #  Python Data Analysis Ecosystem
#
# ![ecosystem](img/ecosystem.png)

# + [markdown] slideshow={"slide_type": "slide"}
# # Intro to Pandas
#
# - `pandas` = Python Data Analysis Library (https://pandas.pydata.org/)
# - With `pandas` you can do pretty much everything you would in a spreadsheet, plus a whole lot more!

# + [markdown] slideshow={"slide_type": "fragment"}
# ### Why Pandas?
# - Working with large data files and complex calculations
# - Dealing with messy and missing data
# - Merging data from multiple files
# - Timeseries analysis
# - Automate repetitive tasks
# - Combine with other Python libraries to create beautiful and fully customized visualizations

# + [markdown] slideshow={"slide_type": "slide"}
# # Reading a CSV file
#
# Let's look at the file `data/weather_hourly_YVR_2017.csv`
# - One full year of Environment Canada hourly weather measurements at Vancouver Airport
# - First, let's check it out in the JupyterLab CSV viewer

# + [markdown] slideshow={"slide_type": "fragment"}
# Now let's start working with the data in `pandas`
# - First, import the `pandas` library and give it the nickname `pd`

# + slideshow={"slide_type": "-"}
import pandas as pd

# + [markdown] slideshow={"slide_type": "fragment"}
# Use the function `pd.read_csv` to read the data into Python and store as a variable:

# + slideshow={"slide_type": "fragment"}
weather_yvr = pd.read_csv('data/weather_hourly_YVR_2017.csv')

# + slideshow={"slide_type": "fragment"}
weather_yvr

# + [markdown] slideshow={"slide_type": "fragment"}
# - Only the first 30 and last 30 rows are displayed (but the data is all there in our `weather_yvr` variable)
# - You may notice some weird `NaN` values&mdash;these represent missing data (`NaN` = "not a number")

# + [markdown] slideshow={"slide_type": "slide"}
# What type of object is `weather_yvr`?

# + slideshow={"slide_type": "-"}
type(weather_yvr)

# + [markdown] slideshow={"slide_type": "fragment"}
# - `weather_yvr` is a **DataFrame**, a data structure from the `pandas` library
#   - A DataFrame is a 2-dimensional array (organized into rows and columns, like a table in a spreadsheet)

# + [markdown] slideshow={"slide_type": "fragment"}
# - When we display `weather_yvr`, the integer numbers in bold on the left are the DataFrame's **index**
#   - In this case, the index is simply a range of integers corresponding with the row numbers

# + slideshow={"slide_type": "-"}
weather_yvr

# + [markdown] slideshow={"slide_type": "slide"}
# For large DataFrames, it's often useful to display just the first few or last few rows:

# + slideshow={"slide_type": "fragment"}
weather_yvr.head()

# + [markdown] slideshow={"slide_type": "fragment"}
# - The `head` method returns a new DataFrame consisting of the first `n` rows (default 5)
# - To display the documentation for this method, you can run the command `weather_yvr.head?` in your Jupyter notebook

# + [markdown] slideshow={"slide_type": "fragment"}
# First two rows:

# + slideshow={"slide_type": "-"}
weather_yvr.head(2)

# + [markdown] slideshow={"slide_type": "fragment"}
# Last four rows:

# + slideshow={"slide_type": "-"}
weather_yvr.tail(4)

# + [markdown] slideshow={"slide_type": "slide"}
# # Data at a Glance
#
# `pandas` provides many ways to quickly and easily summarize your data:
# - How many rows and columns are there?
# - What are all the column names and what type of data is in each column?

# + [markdown] slideshow={"slide_type": "fragment"}
# - Numerical data: What is the average and range of the values?
# - Text data: What are the unique values and how often does each occur?
# - How many values are missing in each column?

# + [markdown] slideshow={"slide_type": "slide"}
# Number of rows and columns:

# + slideshow={"slide_type": "fragment"}
weather_yvr.shape

# + [markdown] slideshow={"slide_type": "fragment"}
# - The DataFrame `weather_yvr` has 8760 rows and 11 columns
# - The index does not count as a column
# - `shape` is a **data attribute** of the variable `weather_yvr`

# + [markdown] slideshow={"slide_type": "slide"}
# - Within a column of a DataFrame, the data must all be of the same type
# - We can find out the names and data types of each column from the `dtypes` attribute:

# + slideshow={"slide_type": "fragment"}
weather_yvr.dtypes

# + [markdown] slideshow={"slide_type": "fragment"}
# - In a `pandas` DataFrame, a column containing text data (or containing a mix of text and numbers) is assigned a `dtype` of `object` and is treated as a column of strings

# + [markdown] slideshow={"slide_type": "slide"}
# If we just want a list of the column names, we can use the `columns` attribute:

# + slideshow={"slide_type": "fragment"}
weather_yvr.columns

# + [markdown] slideshow={"slide_type": "slide"}
# # Exercise 1.1 
#
# Let's explore `'data/weather_airports_24hr_snapshot.csv'`, which contains a 24 hour snapshot of Environment Canada weather measurements at major airport stations around Canada.
#
# a) Read the CSV file into a new DataFrame `weather_all` and display the first 10 rows.
#
# b) How many rows and columns does `weather_all` have?
#
# c) Display the names and data types of each column.

# + [markdown] slideshow={"slide_type": "slide"}
# a) Read the file `'data/weather_airports_24hr_snapshot.csv'` into a new DataFrame `weather_all` and display the first 10 rows.
# -



# + [markdown] slideshow={"slide_type": "slide"}
# b) How many rows and columns does `weather_all` have?
# -



# + [markdown] slideshow={"slide_type": "fragment"}
# c) Display the names and data types of each column.

# + slideshow={"slide_type": "-"}


# + [markdown] slideshow={"slide_type": "slide"}
# # Simple Summary Statistics
#
# Returning to our `weather_yvr` data:

# + slideshow={"slide_type": "-"}
print(weather_yvr.shape)
weather_yvr.head(3)

# + [markdown] slideshow={"slide_type": "fragment"}
# The `describe` method computes simple summary statistics and returns them as a DataFrame:

# + slideshow={"slide_type": "-"}
weather_yvr.describe()

# + [markdown] slideshow={"slide_type": "fragment"}
# - The `describe` method is a way to quickly summarize the averages, extremes, and variability of each numerical data column
# - You can look at each statistic individually with methods such as `mean`, `median`, `min`, `max`,`std`, and `count`

# + slideshow={"slide_type": "fragment"}
weather_yvr.mean()

# + [markdown] slideshow={"slide_type": "slide"}
# # Working with DataFrame Columns

# + [markdown] slideshow={"slide_type": "-"}
# Similar to a dictionary, we can index a specific column of a DataFrame using the column name inside square brackets:
# - *Pro tip: In Jupyter notebooks, auto-complete works for DataFrame column names!*

# + slideshow={"slide_type": "-"}
weather_yvr['Temp (deg C)']

# + [markdown] slideshow={"slide_type": "fragment"}
# The numbers on the left are the **index**

# + [markdown] slideshow={"slide_type": "slide"}
# What type of object is this?

# + slideshow={"slide_type": "fragment"}
temperature = weather_yvr['Temp (deg C)']
type(temperature)

# + [markdown] slideshow={"slide_type": "fragment"}
# It's a Series, another data structure from the `pandas` library.
#
# - **DataFrame:** 2-dimensional array, like a table in a spreadsheet
#   - The rows are axis 0
#   - The columns are axis 1
# - **Series:** 1-dimensional array, like a single column or row in a spreadsheet
#   - Each individual column or row of a DataFrame is represented as a Series

# + [markdown] slideshow={"slide_type": "slide"}
# Let's look at the Series again:

# + slideshow={"slide_type": "-"}
temperature

# + [markdown] slideshow={"slide_type": "fragment"}
# The last line of the output above tells us that our Series `temperature` is named `'Temp (deg C)'`, it is 8760 rows long, and its data type is `float64`.

# + [markdown] slideshow={"slide_type": "slide"}
# Many of the methods we use on a DataFrame can also be used on a Series, and vice versa

# + slideshow={"slide_type": "fragment"}
temperature.head()

# + slideshow={"slide_type": "fragment"}
weather_yvr['Rel Hum (%)'].describe()

# + slideshow={"slide_type": "fragment"}
weather_yvr['Wind Spd (km/h)'].max()

# + [markdown] slideshow={"slide_type": "slide"}
# # Simple Calculations
#
# We can perform calculations on Series.
# - Let's convert temperature from Celsius to Fahrenheit (multiply by 1.8 and add 32)

# + slideshow={"slide_type": "fragment"}
temperature_F = 1.8 * temperature + 32
temperature_F.head()

# + [markdown] slideshow={"slide_type": "skip"}
# Side note: the new Series `temperature_F` still has the same name (`'Temp (deg C)'`) as the original Series
# - If we want to change the name, we can update the `name` attribute of the Series:

# + slideshow={"slide_type": "skip"}
temperature_F.name = 'Temp (deg F)'
temperature_F.head()

# + [markdown] slideshow={"slide_type": "fragment"}
# Similar to a dictionary, we can add a new column to a DataFrame by simply assigning a value to a new column name:

# + slideshow={"slide_type": "fragment"}
weather_yvr['Temp (deg F)'] = 1.8 * weather_yvr['Temp (deg C)'] + 32
weather_yvr.head(3)

# + [markdown] slideshow={"slide_type": "skip"}
# By creating a new column labelled `'Temp (deg F)'`, the Series in this column is automatically given the same name as its column label:

# + slideshow={"slide_type": "skip"}
weather_yvr['Temp (deg F)'].head()

# + [markdown] slideshow={"slide_type": "slide"}
# # Multiple Columns of a DataFrame
#
# We can also select several columns, in whatever order we like, using a list of column names:

# + slideshow={"slide_type": "fragment"}
subset_columns = ['Datetime', 'Wind Spd (km/h)', 
                  'Wind Dir (deg)', 'Rel Hum (%)', 
                  'Temp (deg C)']
weather_subset = weather_yvr[subset_columns]
weather_subset.head()

# + [markdown] slideshow={"slide_type": "fragment"}
# `weather_subset` is a DataFrame containing only the specified columns of `weather_yvr`.
# -

# We can use subsets  to compute summary statistics just for specific columns of interest:

stats_columns = ['Rel Hum (%)', 'Temp (deg C)']
weather_yvr[stats_columns].mean()

# + [markdown] slideshow={"slide_type": "slide"}
# If our list of columns is pretty short, it's often convenient to skip a step and define the list directly within the indexing operator: 

# + slideshow={"slide_type": "fragment"}
weather_yvr[['Datetime', 'Temp (deg C)']].head()

# + [markdown] slideshow={"slide_type": "fragment"}
# Note the double square brackets!
# - These are required because we need both:
#   - A pair of square brackets to extract the subset, AND
#   - A pair of square brackets to define the list of columns to select.

# + [markdown] slideshow={"slide_type": "fragment"}
# What happens if we forget and just use a single pair of square brackets?
# - It doesn't work!
# - This is a very easy mistake to make, especially when you're coding on the fly in Jupyter
# - For more details on why forgetting the second square brackets causes an error, check out the section "Interlude: Forgetting the Double Square Brackets" in Lesson 3

# + [markdown] slideshow={"slide_type": "slide"}
# # Saving to CSV
#
# We can save a DataFrame to a CSV file using the `to_csv` method.
#
# - When you first start working with `to_csv`, it's easy to accidentally overwrite existing files, so you might want to first make a copy of your data folder, as a backup.

# + [markdown] slideshow={"slide_type": "fragment"}
# Let's save the data from the columns `'Datetime'`, `'Temp (deg C)'`, and the new `'Temp (deg F)'` to a CSV file:

# + slideshow={"slide_type": "fragment"}
cols_save = ['Datetime', 'Temp (deg C)', 'Temp (deg F)'] 
temp_df = weather_yvr[cols_save]
temp_df.head()

# + slideshow={"slide_type": "fragment"}
temp_df.to_csv('data/temperatures_YVR.csv', index=False)

# + [markdown] slideshow={"slide_type": "fragment"}
# - By default, the `to_csv` method will save the DataFrame's index as an additional column in the CSV file. To turn this off, we use the keyword argument `index=False`.

# + [markdown] slideshow={"slide_type": "fragment"}
# Check out your new file in the JupyterLab CSV viewer!

# + [markdown] slideshow={"slide_type": "slide"}
# # Categorical Data
#
# What about text data like the `'Conditions'` column?

# + slideshow={"slide_type": "-"}
weather_yvr['Conditions']

# + [markdown] slideshow={"slide_type": "fragment"}
# This column consists of categories, with many repeated values.
# - What are the unique values in the Series?
# - How often does each value occur?
# - What are the most common values?

# + [markdown] slideshow={"slide_type": "fragment"}
# We can answer these questions with the `unique`, `nunique`, and `value_counts` methods.
# - These methods are only applicable to Series, not DataFrames.

# + [markdown] slideshow={"slide_type": "slide"}
# `value_counts` is a very handy method to quickly summarize a Series of text data and find the most common values:

# + slideshow={"slide_type": "fragment"}
weather_yvr['Conditions'].value_counts()

# + [markdown] slideshow={"slide_type": "skip"}
# We can use the `unique` method to list the unique values:

# + slideshow={"slide_type": "skip"}
weather_yvr['Conditions'].unique()

# + [markdown] slideshow={"slide_type": "skip"}
# We can use the `nunique` method to find the number of unique values:

# + slideshow={"slide_type": "skip"}
weather_yvr['Conditions'].nunique()

# + [markdown] slideshow={"slide_type": "slide"}
# # Lesson 1 Recap
#
# ### Importing `pandas` Library
# ```
# import pandas as pd
# ```
#
# ### DataFrames and Series
#
# Data in `pandas` is organized into DataFrames and Series.
#
# - **DataFrame:** 2-dimensional array, like a table in a spreadsheet
#   - The rows are axis 0
#   - The columns are axis 1
# - **Series:** 1-dimensional array, like a single column or row in a spreadsheet
#   - Each individual column or row of a DataFrame is represented as a Series
#
# ### Reading a CSV File
#
# To read a CSV file and store it as a DataFrame variable:
# ```
# df = pd.read_csv('some_cool_data.csv')
# ```
#
# Missing data in a DataFrame or Series is represented as `NaN` ("not a number").
#
# ### Saving to a CSV File
#
# To save a DataFrame to a CSV file: 
# ```
# df.to_csv('cool_output.csv', index=False)
# ```
# - To include the DataFrame's index as a column in the CSV file, omit the `index=False` keyword argument.
#
# ### Quick and Easy Summaries of a DataFrame
#
# |||
# ---|----
# **Useful Attributes** |
# Number of rows and columns (rows first, columns second) | `df.shape` 
# Names and data types of each column |  `df.dtypes` 
# Just the names of each column | `df.columns` 
# **Rows at a Glance** |
# First `n` rows (default 5) |`df.head(n)`
# Last `n` rows (default 5) | `df.tail(n)`
# A random sampling of `n` rows (default 1) | `df.sample(n)`
#
#
# #### Summary Statistics
#
# Full set of summary statistics (min, max, mean, standard deviation, etc.) for each numerical column of a DataFrame:
# ```
# df.describe()
# ```
#
# Mean value of each column:
# ```
# df.mean()
# ```
#
# And similarly for other summary statistics: `df.min()`, `df.max()`, `df.median()`, `df.std()`
#
# ### Working with DataFrame Columns
#
# #### Single Columns
#
# Each column of a DataFrame is a Series.
# ```
# series_X = df['X']
# ```
#
# Most DataFrame methods can be applied to a Series, for example:
# - `df['X'].head()`
# - `df['X'].max()`
#
# Basic calculations with a Series and adding a new column to a DataFrame: 
# ```
# df['Double X'] = 2 * df['X']
# ```
#
# #### Multiple Columns
#
# Use a list of column names to select several columns of a DataFrame, in a specified order:
# ```
# df_subset = df[['E', 'A', 'C']]
# ```
#
# ### Categorical Data
#
# For a column `df['Category']` of categorical data, some useful summary methods are:
#
# |||
# ---|---
# Unique values | `df['Category'].unique()`
# Number of unique values | `df['Category'].nunique()`
# Counts of each unique value | `df['Category'].value_counts()`
#
# *Note: These methods can only be applied to a Series (not a DataFrame).*

# + [markdown] slideshow={"slide_type": "slide"}
# # Exercise 1.2
#
# If you haven't already, read the file `'data/weather_airports_24hr_snapshot.csv'` into a new DataFrame `weather_all`.
#
# a) What are the warmest and coldest temperatures in this data?
#
# b) How many unique station names are in this data? Display a list of the unique names.
#
# c) What is the most common weather category in the `'Conditions'` column? How many unique categories are there?
#
# d) Add a column with the wind speed in miles per hour (multiply the wind speed in km/h by 0.62137). Save the data from columns `'Station Name'`, `'Datetime (Local Standard)'`, `'Wind Spd (km/h)'`, and your new column of wind speed in miles per hour, to a CSV file.

# + [markdown] slideshow={"slide_type": "-"}
# ##### Bonus exercises
#
# Create a variable `conditions` corresponding to the `'Conditions'` column of `weather_all`. We'll use this variable in each of the following exercises.
#
# e) What type of object is returned by `conditions.value_counts()`? Can you think of a method that could be applied to this output so that it returns only the counts for the top `n` values? How about the bottom `n` values?
# - Display only the counts for the 5 most common weather categories in `conditions`
# - Display only the counts for the 5 least common weather categories in `conditions`

# + [markdown] slideshow={"slide_type": "-"}
# f) Use `conditions.value_counts?` to check out the documentation for the `value_counts` method. Experiment with the `normalize`, `sort` and `dropna` keyword arguments. How does the output change when you change these arguments?

# + [markdown] slideshow={"slide_type": "-"}
# g) `pandas` Series have a few *accessors*, which are attributes that [act like an interface to additional methods](https://realpython.com/python-pandas-tricks/#3-take-advantage-of-accessor-methods). With a Series of text data, like `conditions`, the `str` accessor allows you to apply string methods such as `upper`, `lower`, `strip`, `replace`, etc. to all the items in the Series.
#
# - Check out some of the documentation with `conditions.str?` and `conditions.str.upper?`. 
# - Create a new Series with the weather categories converted to upper case.
# - Create a new Series with any instance of the string `'Snow'` in a weather category replaced with the string `'SNOW!!!'`.
# - For both of these new Series, use `value_counts` or `unique` methods to verify that the output is what you were expecting.

# + [markdown] slideshow={"slide_type": "slide"}
# ---
# If you haven't already, read the file `'data/weather_airports_24hr_snapshot.csv'` into a new DataFrame `weather_all`.
#
# a) What are the warmest and coldest temperatures in this data?

# + slideshow={"slide_type": "-"}

# -



# + slideshow={"slide_type": "-"}


# + slideshow={"slide_type": "-"}


# + [markdown] slideshow={"slide_type": "slide"}
# b) How many unique station names are in this data? Display a list of the unique names.

# + slideshow={"slide_type": "-"}


# + slideshow={"slide_type": "-"}


# + slideshow={"slide_type": "-"}


# + [markdown] slideshow={"slide_type": "slide"}
# c) What is the most common weather category in the `'Conditions'` column? How many unique categories are there?

# + slideshow={"slide_type": "-"}


# + slideshow={"slide_type": "-"}


# + slideshow={"slide_type": "-"}


# + [markdown] slideshow={"slide_type": "slide"}
# d) Add a column with the wind speed in miles per hour (multiply the wind speed in km/h by 0.62137). Save the data from columns `'Station Name'`, `'Datetime (Local Standard)'`, `'Wind Spd (km/h)'`, and your new column of wind speed in miles per hour, to a CSV file.

# + slideshow={"slide_type": "-"}


# + slideshow={"slide_type": "-"}


# + slideshow={"slide_type": "-"}


StarcoderdataJupyterScriptsDedupFiltered